Note: This is test shard 1 of 8.
[==========] Running 8 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from AdminCliTest
[ RUN ] AdminCliTest.RebalancerReportOnly
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 08:11:06.486224 21401 test_util.cc:268] Using random seed: 1123231142
I20240503 08:11:06.496834 21401 ts_itest-base.cc:112] Starting cluster with:
I20240503 08:11:06.496904 21401 ts_itest-base.cc:113] --------------
I20240503 08:11:06.496941 21401 ts_itest-base.cc:114] 5 tablet servers
I20240503 08:11:06.496978 21401 ts_itest-base.cc:115] 3 replicas per TS
I20240503 08:11:06.497018 21401 ts_itest-base.cc:116] --------------
2024-05-03T08:11:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:11:06Z Disabled control of system clock
I20240503 08:11:06.518895 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.230.126:37715
--webserver_interface=127.20.230.126
--webserver_port=0
--builtin_ntp_servers=127.20.230.84:41643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.230.126:37715 with env {}
W20240503 08:11:06.829627 21413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:06.829926 21413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:06.830008 21413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:06.837363 21413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:11:06.837453 21413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:06.837493 21413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:11:06.837544 21413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:11:06.846284 21413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:41643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.230.126:37715
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.230.126:37715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.230.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:06.847465 21413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:06.849037 21413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:06.859823 21418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:06.860476 21419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:06.860781 21421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:06.861455 21413 server_base.cc:1034] running on GCE node
I20240503 08:11:06.862186 21413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:06.863529 21413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:06.864732 21413 hybrid_clock.cc:648] HybridClock initialized: now 1714723866864669 us; error 38 us; skew 500 ppm
I20240503 08:11:06.865010 21413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:06.870779 21413 webserver.cc:460] Webserver started at http://127.20.230.126:39671/ using document root <none> and password file <none>
I20240503 08:11:06.871500 21413 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:06.871608 21413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:06.871951 21413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:06.874497 21413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "99da26087f614a4eb98e3d1d45164e59"
format_stamp: "Formatted at 2024-05-03 08:11:06 on dist-test-slave-6vp8"
I20240503 08:11:06.875141 21413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "99da26087f614a4eb98e3d1d45164e59"
format_stamp: "Formatted at 2024-05-03 08:11:06 on dist-test-slave-6vp8"
I20240503 08:11:06.879895 21413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20240503 08:11:06.883031 21427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:06.884232 21413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 08:11:06.884379 21413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/wal
uuid: "99da26087f614a4eb98e3d1d45164e59"
format_stamp: "Formatted at 2024-05-03 08:11:06 on dist-test-slave-6vp8"
I20240503 08:11:06.884546 21413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:06.907745 21413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:06.908504 21413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:06.908776 21413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:06.926463 21413 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.126:37715
I20240503 08:11:06.926483 21478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.126:37715 every 8 connection(s)
I20240503 08:11:06.927994 21413 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 08:11:06.931535 21479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:06.932902 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 21413
I20240503 08:11:06.933183 21401 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 08:11:06.942925 21479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99da26087f614a4eb98e3d1d45164e59: Bootstrap starting.
I20240503 08:11:06.945535 21479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99da26087f614a4eb98e3d1d45164e59: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:06.946485 21479 log.cc:824] T 00000000000000000000000000000000 P 99da26087f614a4eb98e3d1d45164e59: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:06.948925 21479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99da26087f614a4eb98e3d1d45164e59: No bootstrap required, opened a new log
I20240503 08:11:06.956996 21479 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99da26087f614a4eb98e3d1d45164e59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99da26087f614a4eb98e3d1d45164e59" member_type: VOTER last_known_addr { host: "127.20.230.126" port: 37715 } }
I20240503 08:11:06.957345 21479 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99da26087f614a4eb98e3d1d45164e59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:06.957455 21479 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99da26087f614a4eb98e3d1d45164e59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99da26087f614a4eb98e3d1d45164e59, State: Initialized, Role: FOLLOWER
I20240503 08:11:06.958132 21479 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99da26087f614a4eb98e3d1d45164e59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99da26087f614a4eb98e3d1d45164e59" member_type: VOTER last_known_addr { host: "127.20.230.126" port: 37715 } }
I20240503 08:11:06.958343 21479 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99da26087f614a4eb98e3d1d45164e59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:06.958470 21479 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99da26087f614a4eb98e3d1d45164e59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:06.958629 21479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99da26087f614a4eb98e3d1d45164e59 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:06.961086 21479 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99da26087f614a4eb98e3d1d45164e59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99da26087f614a4eb98e3d1d45164e59" member_type: VOTER last_known_addr { host: "127.20.230.126" port: 37715 } }
I20240503 08:11:06.961516 21479 leader_election.cc:302] T 00000000000000000000000000000000 P 99da26087f614a4eb98e3d1d45164e59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99da26087f614a4eb98e3d1d45164e59; no voters:
I20240503 08:11:06.962092 21479 leader_election.cc:288] T 00000000000000000000000000000000 P 99da26087f614a4eb98e3d1d45164e59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:06.962210 21485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99da26087f614a4eb98e3d1d45164e59 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:06.962716 21485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99da26087f614a4eb98e3d1d45164e59 [term 1 LEADER]: Becoming Leader. State: Replica: 99da26087f614a4eb98e3d1d45164e59, State: Running, Role: LEADER
I20240503 08:11:06.963222 21485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99da26087f614a4eb98e3d1d45164e59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99da26087f614a4eb98e3d1d45164e59" member_type: VOTER last_known_addr { host: "127.20.230.126" port: 37715 } }
I20240503 08:11:06.964058 21479 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99da26087f614a4eb98e3d1d45164e59 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:11:06.966603 21487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99da26087f614a4eb98e3d1d45164e59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99da26087f614a4eb98e3d1d45164e59. Latest consensus state: current_term: 1 leader_uuid: "99da26087f614a4eb98e3d1d45164e59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99da26087f614a4eb98e3d1d45164e59" member_type: VOTER last_known_addr { host: "127.20.230.126" port: 37715 } } }
I20240503 08:11:06.966956 21487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99da26087f614a4eb98e3d1d45164e59 [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:06.967386 21486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99da26087f614a4eb98e3d1d45164e59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99da26087f614a4eb98e3d1d45164e59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99da26087f614a4eb98e3d1d45164e59" member_type: VOTER last_known_addr { host: "127.20.230.126" port: 37715 } } }
I20240503 08:11:06.967686 21486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99da26087f614a4eb98e3d1d45164e59 [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:06.968619 21494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:11:06.973899 21494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:11:06.982595 21494 catalog_manager.cc:1347] Generated new cluster ID: a70f79bf40114b019e2f03042c0df177
I20240503 08:11:06.982682 21494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:11:07.002916 21494 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:11:07.003994 21494 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:11:07.013067 21494 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 99da26087f614a4eb98e3d1d45164e59: Generated new TSK 0
I20240503 08:11:07.013810 21494 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:11:07.025597 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.65:0
--local_ip_for_outbound_sockets=127.20.230.65
--webserver_interface=127.20.230.65
--webserver_port=0
--tserver_master_addrs=127.20.230.126:37715
--builtin_ntp_servers=127.20.230.84:41643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20240503 08:11:07.344897 21504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:07.345157 21504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:07.345255 21504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:07.352764 21504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:07.352921 21504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.65
I20240503 08:11:07.361266 21504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:41643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.230.65
--webserver_port=0
--tserver_master_addrs=127.20.230.126:37715
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.65
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:07.362463 21504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:07.364030 21504 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:11:07.376382 21504 server_base.cc:1034] running on GCE node
W20240503 08:11:07.377022 21510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:07.377378 21512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:07.377452 21509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:07.378057 21504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:07.378955 21504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:07.380159 21504 hybrid_clock.cc:648] HybridClock initialized: now 1714723867380092 us; error 45 us; skew 500 ppm
I20240503 08:11:07.380450 21504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:07.382933 21504 webserver.cc:460] Webserver started at http://127.20.230.65:35299/ using document root <none> and password file <none>
I20240503 08:11:07.383579 21504 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:07.383694 21504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:07.383965 21504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:07.386426 21504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "d8fde80124544c7fabc5658e228b3cdc"
format_stamp: "Formatted at 2024-05-03 08:11:07 on dist-test-slave-6vp8"
I20240503 08:11:07.387123 21504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "d8fde80124544c7fabc5658e228b3cdc"
format_stamp: "Formatted at 2024-05-03 08:11:07 on dist-test-slave-6vp8"
I20240503 08:11:07.391834 21504 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 08:11:07.394999 21518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:07.396241 21504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 08:11:07.396391 21504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "d8fde80124544c7fabc5658e228b3cdc"
format_stamp: "Formatted at 2024-05-03 08:11:07 on dist-test-slave-6vp8"
I20240503 08:11:07.396555 21504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:07.422083 21504 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:07.422847 21504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:07.423120 21504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:07.423986 21504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:07.425376 21504 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:07.425455 21504 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:07.425580 21504 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:07.425632 21504 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:07.456280 21504 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.65:34133
I20240503 08:11:07.456321 21631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.65:34133 every 8 connection(s)
I20240503 08:11:07.457454 21504 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 08:11:07.460096 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 21504
I20240503 08:11:07.460340 21401 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 08:11:07.463449 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.66:0
--local_ip_for_outbound_sockets=127.20.230.66
--webserver_interface=127.20.230.66
--webserver_port=0
--tserver_master_addrs=127.20.230.126:37715
--builtin_ntp_servers=127.20.230.84:41643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 08:11:07.470237 21632 heartbeater.cc:347] Connected to a master server at 127.20.230.126:37715
I20240503 08:11:07.470489 21632 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:07.471184 21632 heartbeater.cc:511] Master 127.20.230.126:37715 requested a full tablet report, sending...
I20240503 08:11:07.473130 21444 ts_manager.cc:194] Registered new tserver with Master: d8fde80124544c7fabc5658e228b3cdc (127.20.230.65:34133)
I20240503 08:11:07.474720 21444 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.65:51539
W20240503 08:11:07.786495 21636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:07.786782 21636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:07.786901 21636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:07.794040 21636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:07.794231 21636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.66
I20240503 08:11:07.803025 21636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:41643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.230.66
--webserver_port=0
--tserver_master_addrs=127.20.230.126:37715
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.66
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:07.804255 21636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:07.805908 21636 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:07.817207 21641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:07.817903 21644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:07.818219 21642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:07.818816 21636 server_base.cc:1034] running on GCE node
I20240503 08:11:07.819484 21636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:07.820501 21636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:07.821731 21636 hybrid_clock.cc:648] HybridClock initialized: now 1714723867821668 us; error 36 us; skew 500 ppm
I20240503 08:11:07.822046 21636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:07.824054 21636 webserver.cc:460] Webserver started at http://127.20.230.66:45769/ using document root <none> and password file <none>
I20240503 08:11:07.825155 21636 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:07.825277 21636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:07.825553 21636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:07.827988 21636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "45dfaca03ad641b2910fe2a3bad71327"
format_stamp: "Formatted at 2024-05-03 08:11:07 on dist-test-slave-6vp8"
I20240503 08:11:07.828675 21636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "45dfaca03ad641b2910fe2a3bad71327"
format_stamp: "Formatted at 2024-05-03 08:11:07 on dist-test-slave-6vp8"
I20240503 08:11:07.833235 21636 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20240503 08:11:07.836207 21650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:07.837378 21636 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 08:11:07.837517 21636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "45dfaca03ad641b2910fe2a3bad71327"
format_stamp: "Formatted at 2024-05-03 08:11:07 on dist-test-slave-6vp8"
I20240503 08:11:07.837667 21636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:07.865794 21636 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:07.866597 21636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:07.866851 21636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:07.867743 21636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:07.869163 21636 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:07.869243 21636 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:07.869355 21636 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:07.869405 21636 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:07.899963 21636 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.66:41181
I20240503 08:11:07.900017 21762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.66:41181 every 8 connection(s)
I20240503 08:11:07.901221 21636 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 08:11:07.908737 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 21636
I20240503 08:11:07.908895 21401 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 08:11:07.911996 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.67:0
--local_ip_for_outbound_sockets=127.20.230.67
--webserver_interface=127.20.230.67
--webserver_port=0
--tserver_master_addrs=127.20.230.126:37715
--builtin_ntp_servers=127.20.230.84:41643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 08:11:07.914175 21763 heartbeater.cc:347] Connected to a master server at 127.20.230.126:37715
I20240503 08:11:07.914453 21763 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:07.915174 21763 heartbeater.cc:511] Master 127.20.230.126:37715 requested a full tablet report, sending...
I20240503 08:11:07.916452 21444 ts_manager.cc:194] Registered new tserver with Master: 45dfaca03ad641b2910fe2a3bad71327 (127.20.230.66:41181)
I20240503 08:11:07.917443 21444 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.66:52209
W20240503 08:11:08.229085 21766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:08.229346 21766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:08.229447 21766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:08.236912 21766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:08.237071 21766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.67
I20240503 08:11:08.246474 21766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:41643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.230.67
--webserver_port=0
--tserver_master_addrs=127.20.230.126:37715
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.67
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:08.247627 21766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:08.249222 21766 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:08.260665 21771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:08.261586 21772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:08.261807 21774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:08.262493 21766 server_base.cc:1034] running on GCE node
I20240503 08:11:08.263022 21766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:08.263906 21766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:08.265111 21766 hybrid_clock.cc:648] HybridClock initialized: now 1714723868265052 us; error 40 us; skew 500 ppm
I20240503 08:11:08.265381 21766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:08.267379 21766 webserver.cc:460] Webserver started at http://127.20.230.67:35073/ using document root <none> and password file <none>
I20240503 08:11:08.268065 21766 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:08.268186 21766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:08.268458 21766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:08.270869 21766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "cfbc5d18b1ec480dba4a4d6f9f18b352"
format_stamp: "Formatted at 2024-05-03 08:11:08 on dist-test-slave-6vp8"
I20240503 08:11:08.271557 21766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "cfbc5d18b1ec480dba4a4d6f9f18b352"
format_stamp: "Formatted at 2024-05-03 08:11:08 on dist-test-slave-6vp8"
I20240503 08:11:08.276357 21766 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20240503 08:11:08.279506 21780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:08.280718 21766 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 08:11:08.280858 21766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "cfbc5d18b1ec480dba4a4d6f9f18b352"
format_stamp: "Formatted at 2024-05-03 08:11:08 on dist-test-slave-6vp8"
I20240503 08:11:08.281015 21766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:08.308769 21766 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:08.309641 21766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:08.309957 21766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:08.310868 21766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:08.312341 21766 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:08.312417 21766 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:08.312536 21766 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:08.312589 21766 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:08.347059 21766 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.67:39371
I20240503 08:11:08.347102 21892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.67:39371 every 8 connection(s)
I20240503 08:11:08.348274 21766 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:11:08.356482 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 21766
I20240503 08:11:08.356644 21401 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 08:11:08.359612 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.68:0
--local_ip_for_outbound_sockets=127.20.230.68
--webserver_interface=127.20.230.68
--webserver_port=0
--tserver_master_addrs=127.20.230.126:37715
--builtin_ntp_servers=127.20.230.84:41643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 08:11:08.360455 21893 heartbeater.cc:347] Connected to a master server at 127.20.230.126:37715
I20240503 08:11:08.360729 21893 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:08.361948 21893 heartbeater.cc:511] Master 127.20.230.126:37715 requested a full tablet report, sending...
I20240503 08:11:08.363252 21444 ts_manager.cc:194] Registered new tserver with Master: cfbc5d18b1ec480dba4a4d6f9f18b352 (127.20.230.67:39371)
I20240503 08:11:08.364009 21444 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.67:59391
I20240503 08:11:08.477262 21632 heartbeater.cc:503] Master 127.20.230.126:37715 was elected leader, sending a full tablet report...
W20240503 08:11:08.682260 21896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:08.682533 21896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:08.682659 21896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:08.689793 21896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:08.689970 21896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.68
I20240503 08:11:08.698830 21896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:41643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.230.68
--webserver_port=0
--tserver_master_addrs=127.20.230.126:37715
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.68
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:08.700026 21896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:08.701750 21896 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:08.713330 21901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:08.713971 21902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:08.714315 21904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:08.715006 21896 server_base.cc:1034] running on GCE node
I20240503 08:11:08.715552 21896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:08.716369 21896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:08.717568 21896 hybrid_clock.cc:648] HybridClock initialized: now 1714723868717511 us; error 33 us; skew 500 ppm
I20240503 08:11:08.717862 21896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:08.719857 21896 webserver.cc:460] Webserver started at http://127.20.230.68:45319/ using document root <none> and password file <none>
I20240503 08:11:08.720494 21896 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:08.720608 21896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:08.720876 21896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:08.723275 21896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "e67da8a05b724737b2493b490d78e683"
format_stamp: "Formatted at 2024-05-03 08:11:08 on dist-test-slave-6vp8"
I20240503 08:11:08.723973 21896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "e67da8a05b724737b2493b490d78e683"
format_stamp: "Formatted at 2024-05-03 08:11:08 on dist-test-slave-6vp8"
I20240503 08:11:08.728564 21896 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20240503 08:11:08.731621 21910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:08.732785 21896 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 08:11:08.732921 21896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "e67da8a05b724737b2493b490d78e683"
format_stamp: "Formatted at 2024-05-03 08:11:08 on dist-test-slave-6vp8"
I20240503 08:11:08.733103 21896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:08.752203 21896 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:08.752919 21896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:08.753181 21896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:08.754037 21896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:08.755395 21896 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:08.755468 21896 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:08.755584 21896 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:08.755633 21896 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:08.785740 21896 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.68:41351
I20240503 08:11:08.785805 22022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.68:41351 every 8 connection(s)
I20240503 08:11:08.787026 21896 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:11:08.794725 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 21896
I20240503 08:11:08.794895 21401 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 08:11:08.798020 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.69:0
--local_ip_for_outbound_sockets=127.20.230.69
--webserver_interface=127.20.230.69
--webserver_port=0
--tserver_master_addrs=127.20.230.126:37715
--builtin_ntp_servers=127.20.230.84:41643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 08:11:08.799031 22023 heartbeater.cc:347] Connected to a master server at 127.20.230.126:37715
I20240503 08:11:08.799367 22023 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:08.800302 22023 heartbeater.cc:511] Master 127.20.230.126:37715 requested a full tablet report, sending...
I20240503 08:11:08.801915 21444 ts_manager.cc:194] Registered new tserver with Master: e67da8a05b724737b2493b490d78e683 (127.20.230.68:41351)
I20240503 08:11:08.802876 21444 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.68:51213
I20240503 08:11:08.920135 21763 heartbeater.cc:503] Master 127.20.230.126:37715 was elected leader, sending a full tablet report...
W20240503 08:11:09.112504 22026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:09.112757 22026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:09.112872 22026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:09.119767 22026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:09.119939 22026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.69
I20240503 08:11:09.128587 22026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:41643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.230.69
--webserver_port=0
--tserver_master_addrs=127.20.230.126:37715
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.69
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:09.129798 22026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:09.131433 22026 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:09.143463 22032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:09.143767 22034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:09.144510 22031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:09.144968 22026 server_base.cc:1034] running on GCE node
I20240503 08:11:09.145542 22026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:09.146404 22026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:09.147599 22026 hybrid_clock.cc:648] HybridClock initialized: now 1714723869147536 us; error 41 us; skew 500 ppm
I20240503 08:11:09.147873 22026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:09.149920 22026 webserver.cc:460] Webserver started at http://127.20.230.69:44853/ using document root <none> and password file <none>
I20240503 08:11:09.150574 22026 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:09.150686 22026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:09.150961 22026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:09.153337 22026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "455ce0a25fc241718b48a64ce283886d"
format_stamp: "Formatted at 2024-05-03 08:11:09 on dist-test-slave-6vp8"
I20240503 08:11:09.154495 22026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "455ce0a25fc241718b48a64ce283886d"
format_stamp: "Formatted at 2024-05-03 08:11:09 on dist-test-slave-6vp8"
I20240503 08:11:09.158929 22026 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20240503 08:11:09.162415 22040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:09.163642 22026 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20240503 08:11:09.163772 22026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "455ce0a25fc241718b48a64ce283886d"
format_stamp: "Formatted at 2024-05-03 08:11:09 on dist-test-slave-6vp8"
I20240503 08:11:09.163956 22026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:09.190393 22026 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:09.191637 22026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:09.191908 22026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:09.192822 22026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:09.194314 22026 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:09.194393 22026 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:09.194482 22026 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:09.194521 22026 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:09.224416 22026 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.69:33107
I20240503 08:11:09.224448 22154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.69:33107 every 8 connection(s)
I20240503 08:11:09.225656 22026 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:11:09.234145 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 22026
I20240503 08:11:09.234349 21401 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 08:11:09.238157 22155 heartbeater.cc:347] Connected to a master server at 127.20.230.126:37715
I20240503 08:11:09.238463 22155 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:09.239138 22155 heartbeater.cc:511] Master 127.20.230.126:37715 requested a full tablet report, sending...
I20240503 08:11:09.240576 21443 ts_manager.cc:194] Registered new tserver with Master: 455ce0a25fc241718b48a64ce283886d (127.20.230.69:33107)
I20240503 08:11:09.241663 21443 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.69:41371
I20240503 08:11:09.250304 21401 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 08:11:09.270581 21443 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44736:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 08:11:09.304368 22089 tablet_service.cc:1467] Processing CreateTablet for tablet 430d7962edf24fad9875a8147454972c (DEFAULT_TABLE table=TestTable [id=22279f774b5e4a028d17077d19078625]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:09.305753 22089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 430d7962edf24fad9875a8147454972c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:09.312520 21698 tablet_service.cc:1467] Processing CreateTablet for tablet 430d7962edf24fad9875a8147454972c (DEFAULT_TABLE table=TestTable [id=22279f774b5e4a028d17077d19078625]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:09.314007 21698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 430d7962edf24fad9875a8147454972c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:09.315510 21566 tablet_service.cc:1467] Processing CreateTablet for tablet 430d7962edf24fad9875a8147454972c (DEFAULT_TABLE table=TestTable [id=22279f774b5e4a028d17077d19078625]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:09.316965 21566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 430d7962edf24fad9875a8147454972c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:09.318279 22173 tablet_bootstrap.cc:492] T 430d7962edf24fad9875a8147454972c P 455ce0a25fc241718b48a64ce283886d: Bootstrap starting.
I20240503 08:11:09.321607 22173 tablet_bootstrap.cc:654] T 430d7962edf24fad9875a8147454972c P 455ce0a25fc241718b48a64ce283886d: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:09.323246 22173 log.cc:824] T 430d7962edf24fad9875a8147454972c P 455ce0a25fc241718b48a64ce283886d: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:09.324544 22175 tablet_bootstrap.cc:492] T 430d7962edf24fad9875a8147454972c P 45dfaca03ad641b2910fe2a3bad71327: Bootstrap starting.
I20240503 08:11:09.325870 22173 tablet_bootstrap.cc:492] T 430d7962edf24fad9875a8147454972c P 455ce0a25fc241718b48a64ce283886d: No bootstrap required, opened a new log
I20240503 08:11:09.326095 22173 ts_tablet_manager.cc:1367] T 430d7962edf24fad9875a8147454972c P 455ce0a25fc241718b48a64ce283886d: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240503 08:11:09.327726 22175 tablet_bootstrap.cc:654] T 430d7962edf24fad9875a8147454972c P 45dfaca03ad641b2910fe2a3bad71327: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:09.328792 22175 log.cc:824] T 430d7962edf24fad9875a8147454972c P 45dfaca03ad641b2910fe2a3bad71327: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:09.331010 22177 tablet_bootstrap.cc:492] T 430d7962edf24fad9875a8147454972c P d8fde80124544c7fabc5658e228b3cdc: Bootstrap starting.
I20240503 08:11:09.331707 22175 tablet_bootstrap.cc:492] T 430d7962edf24fad9875a8147454972c P 45dfaca03ad641b2910fe2a3bad71327: No bootstrap required, opened a new log
I20240503 08:11:09.331926 22175 ts_tablet_manager.cc:1367] T 430d7962edf24fad9875a8147454972c P 45dfaca03ad641b2910fe2a3bad71327: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20240503 08:11:09.333432 22173 raft_consensus.cc:348] T 430d7962edf24fad9875a8147454972c P 455ce0a25fc241718b48a64ce283886d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455ce0a25fc241718b48a64ce283886d" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 33107 } } peers { permanent_uuid: "45dfaca03ad641b2910fe2a3bad71327" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 41181 } } peers { permanent_uuid: "d8fde80124544c7fabc5658e228b3cdc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 34133 } }
I20240503 08:11:09.333838 22173 raft_consensus.cc:374] T 430d7962edf24fad9875a8147454972c P 455ce0a25fc241718b48a64ce283886d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:09.333946 22173 raft_consensus.cc:729] T 430d7962edf24fad9875a8147454972c P 455ce0a25fc241718b48a64ce283886d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 455ce0a25fc241718b48a64ce283886d, State: Initialized, Role: FOLLOWER
I20240503 08:11:09.334158 22177 tablet_bootstrap.cc:654] T 430d7962edf24fad9875a8147454972c P d8fde80124544c7fabc5658e228b3cdc: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:09.334609 22173 consensus_queue.cc:260] T 430d7962edf24fad9875a8147454972c P 455ce0a25fc241718b48a64ce283886d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455ce0a25fc241718b48a64ce283886d" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 33107 } } peers { permanent_uuid: "45dfaca03ad641b2910fe2a3bad71327" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 41181 } } peers { permanent_uuid: "d8fde80124544c7fabc5658e228b3cdc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 34133 } }
I20240503 08:11:09.335613 22155 heartbeater.cc:503] Master 127.20.230.126:37715 was elected leader, sending a full tablet report...
I20240503 08:11:09.335783 22177 log.cc:824] T 430d7962edf24fad9875a8147454972c P d8fde80124544c7fabc5658e228b3cdc: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:09.336345 22173 ts_tablet_manager.cc:1398] T 430d7962edf24fad9875a8147454972c P 455ce0a25fc241718b48a64ce283886d: Time spent starting tablet: real 0.010s user 0.006s sys 0.003s
I20240503 08:11:09.338816 22177 tablet_bootstrap.cc:492] T 430d7962edf24fad9875a8147454972c P d8fde80124544c7fabc5658e228b3cdc: No bootstrap required, opened a new log
I20240503 08:11:09.338619 22175 raft_consensus.cc:348] T 430d7962edf24fad9875a8147454972c P 45dfaca03ad641b2910fe2a3bad71327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455ce0a25fc241718b48a64ce283886d" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 33107 } } peers { permanent_uuid: "45dfaca03ad641b2910fe2a3bad71327" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 41181 } } peers { permanent_uuid: "d8fde80124544c7fabc5658e228b3cdc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 34133 } }
I20240503 08:11:09.338981 22175 raft_consensus.cc:374] T 430d7962edf24fad9875a8147454972c P 45dfaca03ad641b2910fe2a3bad71327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:09.339077 22175 raft_consensus.cc:729] T 430d7962edf24fad9875a8147454972c P 45dfaca03ad641b2910fe2a3bad71327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45dfaca03ad641b2910fe2a3bad71327, State: Initialized, Role: FOLLOWER
I20240503 08:11:09.339113 22177 ts_tablet_manager.cc:1367] T 430d7962edf24fad9875a8147454972c P d8fde80124544c7fabc5658e228b3cdc: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20240503 08:11:09.339648 22175 consensus_queue.cc:260] T 430d7962edf24fad9875a8147454972c P 45dfaca03ad641b2910fe2a3bad71327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455ce0a25fc241718b48a64ce283886d" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 33107 } } peers { permanent_uuid: "45dfaca03ad641b2910fe2a3bad71327" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 41181 } } peers { permanent_uuid: "d8fde80124544c7fabc5658e228b3cdc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 34133 } }
I20240503 08:11:09.341159 22175 ts_tablet_manager.cc:1398] T 430d7962edf24fad9875a8147454972c P 45dfaca03ad641b2910fe2a3bad71327: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20240503 08:11:09.345551 22177 raft_consensus.cc:348] T 430d7962edf24fad9875a8147454972c P d8fde80124544c7fabc5658e228b3cdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455ce0a25fc241718b48a64ce283886d" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 33107 } } peers { permanent_uuid: "45dfaca03ad641b2910fe2a3bad71327" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 41181 } } peers { permanent_uuid: "d8fde80124544c7fabc5658e228b3cdc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 34133 } }
I20240503 08:11:09.345847 22177 raft_consensus.cc:374] T 430d7962edf24fad9875a8147454972c P d8fde80124544c7fabc5658e228b3cdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:09.345937 22177 raft_consensus.cc:729] T 430d7962edf24fad9875a8147454972c P d8fde80124544c7fabc5658e228b3cdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8fde80124544c7fabc5658e228b3cdc, State: Initialized, Role: FOLLOWER
I20240503 08:11:09.346465 22177 consensus_queue.cc:260] T 430d7962edf24fad9875a8147454972c P d8fde80124544c7fabc5658e228b3cdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455ce0a25fc241718b48a64ce283886d" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 33107 } } peers { permanent_uuid: "45dfaca03ad641b2910fe2a3bad71327" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 41181 } } peers { permanent_uuid: "d8fde80124544c7fabc5658e228b3cdc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 34133 } }
I20240503 08:11:09.348321 22177 ts_tablet_manager.cc:1398] T 430d7962edf24fad9875a8147454972c P d8fde80124544c7fabc5658e228b3cdc: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20240503 08:11:09.366261 21893 heartbeater.cc:503] Master 127.20.230.126:37715 was elected leader, sending a full tablet report...
I20240503 08:11:09.398763 22181 raft_consensus.cc:482] T 430d7962edf24fad9875a8147454972c P d8fde80124544c7fabc5658e228b3cdc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:09.399019 22181 raft_consensus.cc:504] T 430d7962edf24fad9875a8147454972c P d8fde80124544c7fabc5658e228b3cdc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455ce0a25fc241718b48a64ce283886d" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 33107 } } peers { permanent_uuid: "45dfaca03ad641b2910fe2a3bad71327" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 41181 } } peers { permanent_uuid: "d8fde80124544c7fabc5658e228b3cdc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 34133 } }
I20240503 08:11:09.400646 22181 leader_election.cc:288] T 430d7962edf24fad9875a8147454972c P d8fde80124544c7fabc5658e228b3cdc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 455ce0a25fc241718b48a64ce283886d (127.20.230.69:33107), 45dfaca03ad641b2910fe2a3bad71327 (127.20.230.66:41181)
W20240503 08:11:09.403752 21764 tablet.cc:2369] T 430d7962edf24fad9875a8147454972c P 45dfaca03ad641b2910fe2a3bad71327: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:11:09.407505 21718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "430d7962edf24fad9875a8147454972c" candidate_uuid: "d8fde80124544c7fabc5658e228b3cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45dfaca03ad641b2910fe2a3bad71327" is_pre_election: true
I20240503 08:11:09.407562 22109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "430d7962edf24fad9875a8147454972c" candidate_uuid: "d8fde80124544c7fabc5658e228b3cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "455ce0a25fc241718b48a64ce283886d" is_pre_election: true
I20240503 08:11:09.407935 22109 raft_consensus.cc:2454] T 430d7962edf24fad9875a8147454972c P 455ce0a25fc241718b48a64ce283886d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8fde80124544c7fabc5658e228b3cdc in term 0.
I20240503 08:11:09.407938 21718 raft_consensus.cc:2454] T 430d7962edf24fad9875a8147454972c P 45dfaca03ad641b2910fe2a3bad71327 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8fde80124544c7fabc5658e228b3cdc in term 0.
I20240503 08:11:09.408473 21522 leader_election.cc:302] T 430d7962edf24fad9875a8147454972c P d8fde80124544c7fabc5658e228b3cdc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 455ce0a25fc241718b48a64ce283886d, d8fde80124544c7fabc5658e228b3cdc; no voters:
I20240503 08:11:09.408816 22181 raft_consensus.cc:2781] T 430d7962edf24fad9875a8147454972c P d8fde80124544c7fabc5658e228b3cdc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:09.408958 22181 raft_consensus.cc:482] T 430d7962edf24fad9875a8147454972c P d8fde80124544c7fabc5658e228b3cdc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:09.409054 22181 raft_consensus.cc:3037] T 430d7962edf24fad9875a8147454972c P d8fde80124544c7fabc5658e228b3cdc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:09.411281 22181 raft_consensus.cc:504] T 430d7962edf24fad9875a8147454972c P d8fde80124544c7fabc5658e228b3cdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455ce0a25fc241718b48a64ce283886d" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 33107 } } peers { permanent_uuid: "45dfaca03ad641b2910fe2a3bad71327" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 41181 } } peers { permanent_uuid: "d8fde80124544c7fabc5658e228b3cdc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 34133 } }
I20240503 08:11:09.411804 22181 leader_election.cc:288] T 430d7962edf24fad9875a8147454972c P d8fde80124544c7fabc5658e228b3cdc [CANDIDATE]: Term 1 election: Requested vote from peers 455ce0a25fc241718b48a64ce283886d (127.20.230.69:33107), 45dfaca03ad641b2910fe2a3bad71327 (127.20.230.66:41181)
I20240503 08:11:09.412142 22109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "430d7962edf24fad9875a8147454972c" candidate_uuid: "d8fde80124544c7fabc5658e228b3cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "455ce0a25fc241718b48a64ce283886d"
I20240503 08:11:09.412250 21718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "430d7962edf24fad9875a8147454972c" candidate_uuid: "d8fde80124544c7fabc5658e228b3cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45dfaca03ad641b2910fe2a3bad71327"
I20240503 08:11:09.412324 22109 raft_consensus.cc:3037] T 430d7962edf24fad9875a8147454972c P 455ce0a25fc241718b48a64ce283886d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:09.412417 21718 raft_consensus.cc:3037] T 430d7962edf24fad9875a8147454972c P 45dfaca03ad641b2910fe2a3bad71327 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:09.415238 22109 raft_consensus.cc:2454] T 430d7962edf24fad9875a8147454972c P 455ce0a25fc241718b48a64ce283886d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8fde80124544c7fabc5658e228b3cdc in term 1.
I20240503 08:11:09.415597 21522 leader_election.cc:302] T 430d7962edf24fad9875a8147454972c P d8fde80124544c7fabc5658e228b3cdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 455ce0a25fc241718b48a64ce283886d, d8fde80124544c7fabc5658e228b3cdc; no voters:
I20240503 08:11:09.415686 21718 raft_consensus.cc:2454] T 430d7962edf24fad9875a8147454972c P 45dfaca03ad641b2910fe2a3bad71327 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8fde80124544c7fabc5658e228b3cdc in term 1.
I20240503 08:11:09.415868 22181 raft_consensus.cc:2781] T 430d7962edf24fad9875a8147454972c P d8fde80124544c7fabc5658e228b3cdc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:09.416278 22181 raft_consensus.cc:686] T 430d7962edf24fad9875a8147454972c P d8fde80124544c7fabc5658e228b3cdc [term 1 LEADER]: Becoming Leader. State: Replica: d8fde80124544c7fabc5658e228b3cdc, State: Running, Role: LEADER
I20240503 08:11:09.416751 22181 consensus_queue.cc:237] T 430d7962edf24fad9875a8147454972c P d8fde80124544c7fabc5658e228b3cdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455ce0a25fc241718b48a64ce283886d" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 33107 } } peers { permanent_uuid: "45dfaca03ad641b2910fe2a3bad71327" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 41181 } } peers { permanent_uuid: "d8fde80124544c7fabc5658e228b3cdc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 34133 } }
I20240503 08:11:09.420851 21443 catalog_manager.cc:5484] T 430d7962edf24fad9875a8147454972c P d8fde80124544c7fabc5658e228b3cdc reported cstate change: term changed from 0 to 1, leader changed from <none> to d8fde80124544c7fabc5658e228b3cdc (127.20.230.65). New cstate: current_term: 1 leader_uuid: "d8fde80124544c7fabc5658e228b3cdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455ce0a25fc241718b48a64ce283886d" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 33107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45dfaca03ad641b2910fe2a3bad71327" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8fde80124544c7fabc5658e228b3cdc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 34133 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:09.447304 21401 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 08:11:09.449960 21401 ts_itest-base.cc:233] Waiting for 1 tablets on tserver d8fde80124544c7fabc5658e228b3cdc to finish bootstrapping
I20240503 08:11:09.458631 21401 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 45dfaca03ad641b2910fe2a3bad71327 to finish bootstrapping
W20240503 08:11:09.460126 21633 tablet.cc:2369] T 430d7962edf24fad9875a8147454972c P d8fde80124544c7fabc5658e228b3cdc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:11:09.465905 21401 ts_itest-base.cc:233] Waiting for 0 tablets on tserver cfbc5d18b1ec480dba4a4d6f9f18b352 to finish bootstrapping
I20240503 08:11:09.473807 21401 ts_itest-base.cc:233] Waiting for 0 tablets on tserver e67da8a05b724737b2493b490d78e683 to finish bootstrapping
W20240503 08:11:09.477233 22156 tablet.cc:2369] T 430d7962edf24fad9875a8147454972c P 455ce0a25fc241718b48a64ce283886d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:11:09.481549 21401 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 455ce0a25fc241718b48a64ce283886d to finish bootstrapping
I20240503 08:11:09.805438 22023 heartbeater.cc:503] Master 127.20.230.126:37715 was elected leader, sending a full tablet report...
I20240503 08:11:09.860534 22181 consensus_queue.cc:1035] T 430d7962edf24fad9875a8147454972c P d8fde80124544c7fabc5658e228b3cdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "455ce0a25fc241718b48a64ce283886d" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 33107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:09.869997 22181 consensus_queue.cc:1035] T 430d7962edf24fad9875a8147454972c P d8fde80124544c7fabc5658e228b3cdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "45dfaca03ad641b2910fe2a3bad71327" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:09.926610 22089 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:09.950085 21958 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:09.955972 21566 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:09.957250 21698 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:09.958632 21828 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:10.108595 22089 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:10.115165 21828 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:10.115785 21698 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:10.120908 21958 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:10.144289 21566 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:10.266721 21401 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8bBFGA/build/asan/bin/kudu pid 21504
I20240503 08:11:10.268185 21627 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:10.359041 21401 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8bBFGA/build/asan/bin/kudu with pid 21504
I20240503 08:11:10.378298 21401 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8bBFGA/build/asan/bin/kudu pid 21636
I20240503 08:11:10.379158 21757 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:10.467402 21401 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8bBFGA/build/asan/bin/kudu with pid 21636
I20240503 08:11:10.481575 21401 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8bBFGA/build/asan/bin/kudu pid 21766
I20240503 08:11:10.482555 21887 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:10.563892 21401 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8bBFGA/build/asan/bin/kudu with pid 21766
I20240503 08:11:10.576702 21401 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8bBFGA/build/asan/bin/kudu pid 21896
I20240503 08:11:10.577690 22017 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:10.659725 21401 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8bBFGA/build/asan/bin/kudu with pid 21896
I20240503 08:11:10.672292 21401 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8bBFGA/build/asan/bin/kudu pid 22026
I20240503 08:11:10.673204 22149 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:10.758700 21401 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8bBFGA/build/asan/bin/kudu with pid 22026
I20240503 08:11:10.771615 21401 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8bBFGA/build/asan/bin/kudu pid 21413
I20240503 08:11:10.772661 21474 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:10.875319 21401 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8bBFGA/build/asan/bin/kudu with pid 21413
2024-05-03T08:11:10Z chronyd exiting
[ OK ] AdminCliTest.RebalancerReportOnly (4427 ms)
[----------] 1 test from AdminCliTest (4427 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.TabletServerIsDown/0
I20240503 08:11:10.911425 21401 test_util.cc:268] Using random seed: 1127656348
I20240503 08:11:10.913089 21401 ts_itest-base.cc:112] Starting cluster with:
I20240503 08:11:10.913177 21401 ts_itest-base.cc:113] --------------
I20240503 08:11:10.913229 21401 ts_itest-base.cc:114] 5 tablet servers
I20240503 08:11:10.913300 21401 ts_itest-base.cc:115] 3 replicas per TS
I20240503 08:11:10.913348 21401 ts_itest-base.cc:116] --------------
2024-05-03T08:11:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:11:10Z Disabled control of system clock
I20240503 08:11:10.932476 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.230.126:40121
--webserver_interface=127.20.230.126
--webserver_port=0
--builtin_ntp_servers=127.20.230.84:33777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.230.126:40121
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240503 08:11:11.228215 22276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:11.228498 22276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:11.228629 22276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:11.235683 22276 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:11:11.235793 22276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:11:11.235860 22276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:11.235934 22276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:11:11.235992 22276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:11:11.244633 22276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:33777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.20.230.126:40121
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.230.126:40121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.230.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:11.245834 22276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:11.247680 22276 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:11.258693 22281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:11.259356 22282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:11.259773 22284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:11.260483 22276 server_base.cc:1034] running on GCE node
I20240503 08:11:11.261161 22276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:11.262495 22276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:11.263703 22276 hybrid_clock.cc:648] HybridClock initialized: now 1714723871263658 us; error 33 us; skew 500 ppm
I20240503 08:11:11.264032 22276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:11.266134 22276 webserver.cc:460] Webserver started at http://127.20.230.126:36693/ using document root <none> and password file <none>
I20240503 08:11:11.266801 22276 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:11.266906 22276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:11.267202 22276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:11.269629 22276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "ab3e717f935f45d8871dfc88bca05f56"
format_stamp: "Formatted at 2024-05-03 08:11:11 on dist-test-slave-6vp8"
I20240503 08:11:11.270371 22276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "ab3e717f935f45d8871dfc88bca05f56"
format_stamp: "Formatted at 2024-05-03 08:11:11 on dist-test-slave-6vp8"
I20240503 08:11:11.275089 22276 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20240503 08:11:11.278120 22290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:11.279353 22276 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.003s
I20240503 08:11:11.279512 22276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ab3e717f935f45d8871dfc88bca05f56"
format_stamp: "Formatted at 2024-05-03 08:11:11 on dist-test-slave-6vp8"
I20240503 08:11:11.279678 22276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:11.307029 22276 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:11.307806 22276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:11.308053 22276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:11.324062 22276 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.126:40121
I20240503 08:11:11.324083 22341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.126:40121 every 8 connection(s)
I20240503 08:11:11.325318 22276 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 08:11:11.328776 22342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.334347 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 22276
I20240503 08:11:11.334543 21401 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 08:11:11.337797 22342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab3e717f935f45d8871dfc88bca05f56: Bootstrap starting.
I20240503 08:11:11.341511 22342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab3e717f935f45d8871dfc88bca05f56: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:11.342797 22342 log.cc:824] T 00000000000000000000000000000000 P ab3e717f935f45d8871dfc88bca05f56: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:11.345590 22342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab3e717f935f45d8871dfc88bca05f56: No bootstrap required, opened a new log
I20240503 08:11:11.351783 22342 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab3e717f935f45d8871dfc88bca05f56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab3e717f935f45d8871dfc88bca05f56" member_type: VOTER last_known_addr { host: "127.20.230.126" port: 40121 } }
I20240503 08:11:11.352072 22342 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab3e717f935f45d8871dfc88bca05f56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:11.352147 22342 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab3e717f935f45d8871dfc88bca05f56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab3e717f935f45d8871dfc88bca05f56, State: Initialized, Role: FOLLOWER
I20240503 08:11:11.352618 22342 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab3e717f935f45d8871dfc88bca05f56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab3e717f935f45d8871dfc88bca05f56" member_type: VOTER last_known_addr { host: "127.20.230.126" port: 40121 } }
I20240503 08:11:11.352783 22342 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab3e717f935f45d8871dfc88bca05f56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:11.352866 22342 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab3e717f935f45d8871dfc88bca05f56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:11.352975 22342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab3e717f935f45d8871dfc88bca05f56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:11.354843 22342 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab3e717f935f45d8871dfc88bca05f56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab3e717f935f45d8871dfc88bca05f56" member_type: VOTER last_known_addr { host: "127.20.230.126" port: 40121 } }
I20240503 08:11:11.355232 22342 leader_election.cc:302] T 00000000000000000000000000000000 P ab3e717f935f45d8871dfc88bca05f56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab3e717f935f45d8871dfc88bca05f56; no voters:
I20240503 08:11:11.355788 22342 leader_election.cc:288] T 00000000000000000000000000000000 P ab3e717f935f45d8871dfc88bca05f56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:11.355918 22347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab3e717f935f45d8871dfc88bca05f56 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:11.356447 22347 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab3e717f935f45d8871dfc88bca05f56 [term 1 LEADER]: Becoming Leader. State: Replica: ab3e717f935f45d8871dfc88bca05f56, State: Running, Role: LEADER
I20240503 08:11:11.356982 22347 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab3e717f935f45d8871dfc88bca05f56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab3e717f935f45d8871dfc88bca05f56" member_type: VOTER last_known_addr { host: "127.20.230.126" port: 40121 } }
I20240503 08:11:11.357848 22342 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab3e717f935f45d8871dfc88bca05f56 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:11:11.359781 22348 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab3e717f935f45d8871dfc88bca05f56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab3e717f935f45d8871dfc88bca05f56. Latest consensus state: current_term: 1 leader_uuid: "ab3e717f935f45d8871dfc88bca05f56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab3e717f935f45d8871dfc88bca05f56" member_type: VOTER last_known_addr { host: "127.20.230.126" port: 40121 } } }
I20240503 08:11:11.359907 22349 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab3e717f935f45d8871dfc88bca05f56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab3e717f935f45d8871dfc88bca05f56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab3e717f935f45d8871dfc88bca05f56" member_type: VOTER last_known_addr { host: "127.20.230.126" port: 40121 } } }
I20240503 08:11:11.360144 22348 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab3e717f935f45d8871dfc88bca05f56 [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:11.360200 22349 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab3e717f935f45d8871dfc88bca05f56 [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:11.366640 22353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:11:11.370615 22353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:11:11.379892 22353 catalog_manager.cc:1347] Generated new cluster ID: 4958e24bd99a43029c599cd26a19d2f4
I20240503 08:11:11.379994 22353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:11:11.393990 22353 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:11:11.395023 22353 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:11:11.403527 22353 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ab3e717f935f45d8871dfc88bca05f56: Generated new TSK 0
I20240503 08:11:11.404397 22353 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:11:11.418556 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.65:0
--local_ip_for_outbound_sockets=127.20.230.65
--webserver_interface=127.20.230.65
--webserver_port=0
--tserver_master_addrs=127.20.230.126:40121
--builtin_ntp_servers=127.20.230.84:33777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 08:11:11.719811 22367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:11.720064 22367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:11.720183 22367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:11.727213 22367 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:11:11.727311 22367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:11.727464 22367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.65
I20240503 08:11:11.735837 22367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:33777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.230.65
--webserver_port=0
--tserver_master_addrs=127.20.230.126:40121
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.65
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:11.737011 22367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:11.738646 22367 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:11.749536 22372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:11.750005 22373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:11.750496 22375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:11.751189 22367 server_base.cc:1034] running on GCE node
I20240503 08:11:11.751704 22367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:11.752542 22367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:11.753720 22367 hybrid_clock.cc:648] HybridClock initialized: now 1714723871753677 us; error 35 us; skew 500 ppm
I20240503 08:11:11.754021 22367 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:11.756011 22367 webserver.cc:460] Webserver started at http://127.20.230.65:45753/ using document root <none> and password file <none>
I20240503 08:11:11.756646 22367 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:11.756747 22367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:11.757035 22367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:11.759467 22367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "47fc50e3aaad476c9b9fb7b385093f23"
format_stamp: "Formatted at 2024-05-03 08:11:11 on dist-test-slave-6vp8"
I20240503 08:11:11.760151 22367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "47fc50e3aaad476c9b9fb7b385093f23"
format_stamp: "Formatted at 2024-05-03 08:11:11 on dist-test-slave-6vp8"
I20240503 08:11:11.764698 22367 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 08:11:11.767724 22381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:11.768873 22367 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:11.769009 22367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "47fc50e3aaad476c9b9fb7b385093f23"
format_stamp: "Formatted at 2024-05-03 08:11:11 on dist-test-slave-6vp8"
I20240503 08:11:11.769173 22367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:11.797726 22367 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:11.798530 22367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:11.798774 22367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:11.799672 22367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:11.801050 22367 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:11.801136 22367 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:11.801246 22367 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:11.801330 22367 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:11.829519 22367 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.65:36283
I20240503 08:11:11.829591 22493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.65:36283 every 8 connection(s)
I20240503 08:11:11.830752 22367 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 08:11:11.833998 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 22367
I20240503 08:11:11.834197 21401 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 08:11:11.837611 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.66:0
--local_ip_for_outbound_sockets=127.20.230.66
--webserver_interface=127.20.230.66
--webserver_port=0
--tserver_master_addrs=127.20.230.126:40121
--builtin_ntp_servers=127.20.230.84:33777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:11:11.843183 22494 heartbeater.cc:347] Connected to a master server at 127.20.230.126:40121
I20240503 08:11:11.843442 22494 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:11.844097 22494 heartbeater.cc:511] Master 127.20.230.126:40121 requested a full tablet report, sending...
I20240503 08:11:11.846208 22307 ts_manager.cc:194] Registered new tserver with Master: 47fc50e3aaad476c9b9fb7b385093f23 (127.20.230.65:36283)
I20240503 08:11:11.848194 22307 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.65:39087
W20240503 08:11:12.136987 22497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:12.137259 22497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:12.137342 22497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:12.144352 22497 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:11:12.144464 22497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:12.144583 22497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.66
I20240503 08:11:12.153126 22497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:33777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.230.66
--webserver_port=0
--tserver_master_addrs=127.20.230.126:40121
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.66
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:12.154390 22497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:12.156052 22497 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:12.167006 22502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:12.167465 22503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:12.167969 22505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:12.168490 22497 server_base.cc:1034] running on GCE node
I20240503 08:11:12.169023 22497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:12.169910 22497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:12.171098 22497 hybrid_clock.cc:648] HybridClock initialized: now 1714723872171046 us; error 47 us; skew 500 ppm
I20240503 08:11:12.171406 22497 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:12.177068 22497 webserver.cc:460] Webserver started at http://127.20.230.66:44513/ using document root <none> and password file <none>
I20240503 08:11:12.177748 22497 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:12.177893 22497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:12.178211 22497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:12.180662 22497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "13870b7641a74621bdfe20cd49c0dbc9"
format_stamp: "Formatted at 2024-05-03 08:11:12 on dist-test-slave-6vp8"
I20240503 08:11:12.181526 22497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "13870b7641a74621bdfe20cd49c0dbc9"
format_stamp: "Formatted at 2024-05-03 08:11:12 on dist-test-slave-6vp8"
I20240503 08:11:12.186350 22497 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20240503 08:11:12.189451 22511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:12.190658 22497 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:11:12.190802 22497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "13870b7641a74621bdfe20cd49c0dbc9"
format_stamp: "Formatted at 2024-05-03 08:11:12 on dist-test-slave-6vp8"
I20240503 08:11:12.190971 22497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:12.229413 22497 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:12.230415 22497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:12.230698 22497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:12.231623 22497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:12.233069 22497 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:12.233161 22497 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:12.233266 22497 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:12.233342 22497 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:12.262406 22497 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.66:41099
I20240503 08:11:12.262485 22624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.66:41099 every 8 connection(s)
I20240503 08:11:12.263641 22497 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 08:11:12.273187 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 22497
I20240503 08:11:12.273383 21401 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 08:11:12.276721 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.67:0
--local_ip_for_outbound_sockets=127.20.230.67
--webserver_interface=127.20.230.67
--webserver_port=0
--tserver_master_addrs=127.20.230.126:40121
--builtin_ntp_servers=127.20.230.84:33777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:11:12.276980 22625 heartbeater.cc:347] Connected to a master server at 127.20.230.126:40121
I20240503 08:11:12.277238 22625 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:12.277971 22625 heartbeater.cc:511] Master 127.20.230.126:40121 requested a full tablet report, sending...
I20240503 08:11:12.279186 22307 ts_manager.cc:194] Registered new tserver with Master: 13870b7641a74621bdfe20cd49c0dbc9 (127.20.230.66:41099)
I20240503 08:11:12.279893 22307 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.66:53469
W20240503 08:11:12.591940 22628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:12.592219 22628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:12.592329 22628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:12.600205 22628 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:11:12.600317 22628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:12.600476 22628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.67
I20240503 08:11:12.609090 22628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:33777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.230.67
--webserver_port=0
--tserver_master_addrs=127.20.230.126:40121
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.67
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:12.610402 22628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:12.612080 22628 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:12.624467 22634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:12.624961 22628 server_base.cc:1034] running on GCE node
W20240503 08:11:12.625531 22636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:12.626000 22633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:12.627020 22628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:12.627873 22628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:12.629060 22628 hybrid_clock.cc:648] HybridClock initialized: now 1714723872629020 us; error 36 us; skew 500 ppm
I20240503 08:11:12.629345 22628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:12.631544 22628 webserver.cc:460] Webserver started at http://127.20.230.67:45485/ using document root <none> and password file <none>
I20240503 08:11:12.632227 22628 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:12.632344 22628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:12.632640 22628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:12.635046 22628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "73e51753b3ba48d4b7426a540f448a7a"
format_stamp: "Formatted at 2024-05-03 08:11:12 on dist-test-slave-6vp8"
I20240503 08:11:12.635738 22628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "73e51753b3ba48d4b7426a540f448a7a"
format_stamp: "Formatted at 2024-05-03 08:11:12 on dist-test-slave-6vp8"
I20240503 08:11:12.640333 22628 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 08:11:12.643397 22642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:12.644538 22628 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 08:11:12.644677 22628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "73e51753b3ba48d4b7426a540f448a7a"
format_stamp: "Formatted at 2024-05-03 08:11:12 on dist-test-slave-6vp8"
I20240503 08:11:12.644848 22628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:12.662767 22628 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:12.663492 22628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:12.663729 22628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:12.664562 22628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:12.665941 22628 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:12.666034 22628 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:12.666133 22628 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:12.666199 22628 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:12.694166 22628 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.67:41279
I20240503 08:11:12.694232 22754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.67:41279 every 8 connection(s)
I20240503 08:11:12.695422 22628 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:11:12.702147 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 22628
I20240503 08:11:12.702320 21401 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 08:11:12.705606 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.68:0
--local_ip_for_outbound_sockets=127.20.230.68
--webserver_interface=127.20.230.68
--webserver_port=0
--tserver_master_addrs=127.20.230.126:40121
--builtin_ntp_servers=127.20.230.84:33777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:11:12.707432 22755 heartbeater.cc:347] Connected to a master server at 127.20.230.126:40121
I20240503 08:11:12.707703 22755 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:12.708348 22755 heartbeater.cc:511] Master 127.20.230.126:40121 requested a full tablet report, sending...
I20240503 08:11:12.709542 22307 ts_manager.cc:194] Registered new tserver with Master: 73e51753b3ba48d4b7426a540f448a7a (127.20.230.67:41279)
I20240503 08:11:12.710343 22307 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.67:42037
I20240503 08:11:12.850978 22494 heartbeater.cc:503] Master 127.20.230.126:40121 was elected leader, sending a full tablet report...
W20240503 08:11:13.013806 22758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:13.014273 22758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:13.014384 22758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:13.021909 22758 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:11:13.022012 22758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:13.022127 22758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.68
I20240503 08:11:13.030790 22758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:33777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.230.68
--webserver_port=0
--tserver_master_addrs=127.20.230.126:40121
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.68
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:13.031987 22758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:13.033632 22758 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:13.045089 22763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:13.045542 22764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:13.046182 22766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:13.047036 22758 server_base.cc:1034] running on GCE node
I20240503 08:11:13.047585 22758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:13.048465 22758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:13.049669 22758 hybrid_clock.cc:648] HybridClock initialized: now 1714723873049618 us; error 47 us; skew 500 ppm
I20240503 08:11:13.050066 22758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:13.052146 22758 webserver.cc:460] Webserver started at http://127.20.230.68:32881/ using document root <none> and password file <none>
I20240503 08:11:13.052834 22758 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:13.052940 22758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:13.053234 22758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:13.055730 22758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "186fbc120ada4fb18c742d9647a1eaf3"
format_stamp: "Formatted at 2024-05-03 08:11:13 on dist-test-slave-6vp8"
I20240503 08:11:13.056438 22758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "186fbc120ada4fb18c742d9647a1eaf3"
format_stamp: "Formatted at 2024-05-03 08:11:13 on dist-test-slave-6vp8"
I20240503 08:11:13.061393 22758 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240503 08:11:13.064733 22772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:13.066048 22758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:11:13.066200 22758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "186fbc120ada4fb18c742d9647a1eaf3"
format_stamp: "Formatted at 2024-05-03 08:11:13 on dist-test-slave-6vp8"
I20240503 08:11:13.066382 22758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:13.080024 22758 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:13.080734 22758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:13.080952 22758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:13.081794 22758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:13.083359 22758 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:13.083453 22758 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:13.083545 22758 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:13.083601 22758 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:13.111907 22758 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.68:37403
I20240503 08:11:13.111958 22884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.68:37403 every 8 connection(s)
I20240503 08:11:13.113150 22758 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:11:13.120612 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 22758
I20240503 08:11:13.120810 21401 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 08:11:13.124117 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.69:0
--local_ip_for_outbound_sockets=127.20.230.69
--webserver_interface=127.20.230.69
--webserver_port=0
--tserver_master_addrs=127.20.230.126:40121
--builtin_ntp_servers=127.20.230.84:33777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:11:13.125303 22885 heartbeater.cc:347] Connected to a master server at 127.20.230.126:40121
I20240503 08:11:13.125613 22885 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:13.126482 22885 heartbeater.cc:511] Master 127.20.230.126:40121 requested a full tablet report, sending...
I20240503 08:11:13.127804 22307 ts_manager.cc:194] Registered new tserver with Master: 186fbc120ada4fb18c742d9647a1eaf3 (127.20.230.68:37403)
I20240503 08:11:13.128566 22307 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.68:48217
I20240503 08:11:13.282379 22625 heartbeater.cc:503] Master 127.20.230.126:40121 was elected leader, sending a full tablet report...
W20240503 08:11:13.437212 22888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:13.437490 22888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:13.437613 22888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:13.445192 22888 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:11:13.445299 22888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:13.445446 22888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.69
I20240503 08:11:13.454424 22888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:33777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.230.69
--webserver_port=0
--tserver_master_addrs=127.20.230.126:40121
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.69
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:13.455695 22888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:13.457381 22888 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:13.468674 22893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:13.469303 22896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:13.469801 22894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:13.470510 22888 server_base.cc:1034] running on GCE node
I20240503 08:11:13.471089 22888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:13.471964 22888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:13.473145 22888 hybrid_clock.cc:648] HybridClock initialized: now 1714723873473096 us; error 37 us; skew 500 ppm
I20240503 08:11:13.473433 22888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:13.475494 22888 webserver.cc:460] Webserver started at http://127.20.230.69:46811/ using document root <none> and password file <none>
I20240503 08:11:13.476132 22888 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:13.476233 22888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:13.476531 22888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:13.478998 22888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "efbd40aa19704f85bbf073eae5bc6fbc"
format_stamp: "Formatted at 2024-05-03 08:11:13 on dist-test-slave-6vp8"
I20240503 08:11:13.479709 22888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "efbd40aa19704f85bbf073eae5bc6fbc"
format_stamp: "Formatted at 2024-05-03 08:11:13 on dist-test-slave-6vp8"
I20240503 08:11:13.484589 22888 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240503 08:11:13.487668 22902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:13.488876 22888 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:13.489010 22888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "efbd40aa19704f85bbf073eae5bc6fbc"
format_stamp: "Formatted at 2024-05-03 08:11:13 on dist-test-slave-6vp8"
I20240503 08:11:13.489180 22888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:13.514735 22888 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:13.515625 22888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:13.515954 22888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:13.517120 22888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:13.519270 22888 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:13.519383 22888 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:13.519490 22888 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:13.519556 22888 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:13.549575 22888 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.69:40255
I20240503 08:11:13.549628 23014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.69:40255 every 8 connection(s)
I20240503 08:11:13.550951 22888 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:11:13.560062 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 22888
I20240503 08:11:13.560250 21401 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 08:11:13.564010 23015 heartbeater.cc:347] Connected to a master server at 127.20.230.126:40121
I20240503 08:11:13.564436 23015 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:13.565176 23015 heartbeater.cc:511] Master 127.20.230.126:40121 requested a full tablet report, sending...
I20240503 08:11:13.566561 22307 ts_manager.cc:194] Registered new tserver with Master: efbd40aa19704f85bbf073eae5bc6fbc (127.20.230.69:40255)
I20240503 08:11:13.567337 22307 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.69:60607
I20240503 08:11:13.576377 21401 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 08:11:13.590777 22307 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58564:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 08:11:13.647826 22690 tablet_service.cc:1467] Processing CreateTablet for tablet 6649209db9b14787a4097bdf400c486e (DEFAULT_TABLE table=TestTable [id=2d740b51b1c84db588617d3b38df2340]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:13.648056 22820 tablet_service.cc:1467] Processing CreateTablet for tablet 6649209db9b14787a4097bdf400c486e (DEFAULT_TABLE table=TestTable [id=2d740b51b1c84db588617d3b38df2340]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:13.649253 22820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6649209db9b14787a4097bdf400c486e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:13.649277 22690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6649209db9b14787a4097bdf400c486e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:13.649379 22950 tablet_service.cc:1467] Processing CreateTablet for tablet 6649209db9b14787a4097bdf400c486e (DEFAULT_TABLE table=TestTable [id=2d740b51b1c84db588617d3b38df2340]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:13.650610 22950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6649209db9b14787a4097bdf400c486e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:13.658740 23034 tablet_bootstrap.cc:492] T 6649209db9b14787a4097bdf400c486e P 186fbc120ada4fb18c742d9647a1eaf3: Bootstrap starting.
I20240503 08:11:13.661511 23034 tablet_bootstrap.cc:654] T 6649209db9b14787a4097bdf400c486e P 186fbc120ada4fb18c742d9647a1eaf3: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:13.662150 23035 tablet_bootstrap.cc:492] T 6649209db9b14787a4097bdf400c486e P 73e51753b3ba48d4b7426a540f448a7a: Bootstrap starting.
I20240503 08:11:13.662853 23034 log.cc:824] T 6649209db9b14787a4097bdf400c486e P 186fbc120ada4fb18c742d9647a1eaf3: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:13.663501 23036 tablet_bootstrap.cc:492] T 6649209db9b14787a4097bdf400c486e P efbd40aa19704f85bbf073eae5bc6fbc: Bootstrap starting.
I20240503 08:11:13.665889 23035 tablet_bootstrap.cc:654] T 6649209db9b14787a4097bdf400c486e P 73e51753b3ba48d4b7426a540f448a7a: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:13.666138 23036 tablet_bootstrap.cc:654] T 6649209db9b14787a4097bdf400c486e P efbd40aa19704f85bbf073eae5bc6fbc: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:13.667091 23035 log.cc:824] T 6649209db9b14787a4097bdf400c486e P 73e51753b3ba48d4b7426a540f448a7a: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:13.667189 23036 log.cc:824] T 6649209db9b14787a4097bdf400c486e P efbd40aa19704f85bbf073eae5bc6fbc: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:13.668433 23034 tablet_bootstrap.cc:492] T 6649209db9b14787a4097bdf400c486e P 186fbc120ada4fb18c742d9647a1eaf3: No bootstrap required, opened a new log
I20240503 08:11:13.668673 23034 ts_tablet_manager.cc:1367] T 6649209db9b14787a4097bdf400c486e P 186fbc120ada4fb18c742d9647a1eaf3: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20240503 08:11:13.669667 23035 tablet_bootstrap.cc:492] T 6649209db9b14787a4097bdf400c486e P 73e51753b3ba48d4b7426a540f448a7a: No bootstrap required, opened a new log
I20240503 08:11:13.669708 23036 tablet_bootstrap.cc:492] T 6649209db9b14787a4097bdf400c486e P efbd40aa19704f85bbf073eae5bc6fbc: No bootstrap required, opened a new log
I20240503 08:11:13.669945 23036 ts_tablet_manager.cc:1367] T 6649209db9b14787a4097bdf400c486e P efbd40aa19704f85bbf073eae5bc6fbc: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240503 08:11:13.669947 23035 ts_tablet_manager.cc:1367] T 6649209db9b14787a4097bdf400c486e P 73e51753b3ba48d4b7426a540f448a7a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20240503 08:11:13.674997 23034 raft_consensus.cc:348] T 6649209db9b14787a4097bdf400c486e P 186fbc120ada4fb18c742d9647a1eaf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbd40aa19704f85bbf073eae5bc6fbc" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 40255 } } peers { permanent_uuid: "73e51753b3ba48d4b7426a540f448a7a" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 41279 } } peers { permanent_uuid: "186fbc120ada4fb18c742d9647a1eaf3" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 37403 } }
I20240503 08:11:13.675315 23034 raft_consensus.cc:374] T 6649209db9b14787a4097bdf400c486e P 186fbc120ada4fb18c742d9647a1eaf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:13.675426 23034 raft_consensus.cc:729] T 6649209db9b14787a4097bdf400c486e P 186fbc120ada4fb18c742d9647a1eaf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186fbc120ada4fb18c742d9647a1eaf3, State: Initialized, Role: FOLLOWER
I20240503 08:11:13.676021 23034 consensus_queue.cc:260] T 6649209db9b14787a4097bdf400c486e P 186fbc120ada4fb18c742d9647a1eaf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbd40aa19704f85bbf073eae5bc6fbc" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 40255 } } peers { permanent_uuid: "73e51753b3ba48d4b7426a540f448a7a" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 41279 } } peers { permanent_uuid: "186fbc120ada4fb18c742d9647a1eaf3" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 37403 } }
I20240503 08:11:13.676985 22885 heartbeater.cc:503] Master 127.20.230.126:40121 was elected leader, sending a full tablet report...
I20240503 08:11:13.677523 23034 ts_tablet_manager.cc:1398] T 6649209db9b14787a4097bdf400c486e P 186fbc120ada4fb18c742d9647a1eaf3: Time spent starting tablet: real 0.009s user 0.003s sys 0.007s
I20240503 08:11:13.678256 23036 raft_consensus.cc:348] T 6649209db9b14787a4097bdf400c486e P efbd40aa19704f85bbf073eae5bc6fbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbd40aa19704f85bbf073eae5bc6fbc" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 40255 } } peers { permanent_uuid: "73e51753b3ba48d4b7426a540f448a7a" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 41279 } } peers { permanent_uuid: "186fbc120ada4fb18c742d9647a1eaf3" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 37403 } }
I20240503 08:11:13.678251 23035 raft_consensus.cc:348] T 6649209db9b14787a4097bdf400c486e P 73e51753b3ba48d4b7426a540f448a7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbd40aa19704f85bbf073eae5bc6fbc" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 40255 } } peers { permanent_uuid: "73e51753b3ba48d4b7426a540f448a7a" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 41279 } } peers { permanent_uuid: "186fbc120ada4fb18c742d9647a1eaf3" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 37403 } }
I20240503 08:11:13.678639 23035 raft_consensus.cc:374] T 6649209db9b14787a4097bdf400c486e P 73e51753b3ba48d4b7426a540f448a7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:13.678640 23036 raft_consensus.cc:374] T 6649209db9b14787a4097bdf400c486e P efbd40aa19704f85bbf073eae5bc6fbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:13.678740 23036 raft_consensus.cc:729] T 6649209db9b14787a4097bdf400c486e P efbd40aa19704f85bbf073eae5bc6fbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efbd40aa19704f85bbf073eae5bc6fbc, State: Initialized, Role: FOLLOWER
I20240503 08:11:13.678738 23035 raft_consensus.cc:729] T 6649209db9b14787a4097bdf400c486e P 73e51753b3ba48d4b7426a540f448a7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73e51753b3ba48d4b7426a540f448a7a, State: Initialized, Role: FOLLOWER
I20240503 08:11:13.679366 23036 consensus_queue.cc:260] T 6649209db9b14787a4097bdf400c486e P efbd40aa19704f85bbf073eae5bc6fbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbd40aa19704f85bbf073eae5bc6fbc" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 40255 } } peers { permanent_uuid: "73e51753b3ba48d4b7426a540f448a7a" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 41279 } } peers { permanent_uuid: "186fbc120ada4fb18c742d9647a1eaf3" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 37403 } }
I20240503 08:11:13.679366 23035 consensus_queue.cc:260] T 6649209db9b14787a4097bdf400c486e P 73e51753b3ba48d4b7426a540f448a7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbd40aa19704f85bbf073eae5bc6fbc" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 40255 } } peers { permanent_uuid: "73e51753b3ba48d4b7426a540f448a7a" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 41279 } } peers { permanent_uuid: "186fbc120ada4fb18c742d9647a1eaf3" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 37403 } }
I20240503 08:11:13.680367 23015 heartbeater.cc:503] Master 127.20.230.126:40121 was elected leader, sending a full tablet report...
I20240503 08:11:13.680963 23036 ts_tablet_manager.cc:1398] T 6649209db9b14787a4097bdf400c486e P efbd40aa19704f85bbf073eae5bc6fbc: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20240503 08:11:13.681062 22755 heartbeater.cc:503] Master 127.20.230.126:40121 was elected leader, sending a full tablet report...
I20240503 08:11:13.683837 23035 ts_tablet_manager.cc:1398] T 6649209db9b14787a4097bdf400c486e P 73e51753b3ba48d4b7426a540f448a7a: Time spent starting tablet: real 0.014s user 0.010s sys 0.001s
W20240503 08:11:13.697299 22756 tablet.cc:2369] T 6649209db9b14787a4097bdf400c486e P 73e51753b3ba48d4b7426a540f448a7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:11:13.802497 23016 tablet.cc:2369] T 6649209db9b14787a4097bdf400c486e P efbd40aa19704f85bbf073eae5bc6fbc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:11:13.833986 23042 raft_consensus.cc:482] T 6649209db9b14787a4097bdf400c486e P 73e51753b3ba48d4b7426a540f448a7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:13.834232 23042 raft_consensus.cc:504] T 6649209db9b14787a4097bdf400c486e P 73e51753b3ba48d4b7426a540f448a7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbd40aa19704f85bbf073eae5bc6fbc" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 40255 } } peers { permanent_uuid: "73e51753b3ba48d4b7426a540f448a7a" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 41279 } } peers { permanent_uuid: "186fbc120ada4fb18c742d9647a1eaf3" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 37403 } }
I20240503 08:11:13.835858 23042 leader_election.cc:288] T 6649209db9b14787a4097bdf400c486e P 73e51753b3ba48d4b7426a540f448a7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efbd40aa19704f85bbf073eae5bc6fbc (127.20.230.69:40255), 186fbc120ada4fb18c742d9647a1eaf3 (127.20.230.68:37403)
I20240503 08:11:13.843962 22970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6649209db9b14787a4097bdf400c486e" candidate_uuid: "73e51753b3ba48d4b7426a540f448a7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efbd40aa19704f85bbf073eae5bc6fbc" is_pre_election: true
I20240503 08:11:13.844044 22840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6649209db9b14787a4097bdf400c486e" candidate_uuid: "73e51753b3ba48d4b7426a540f448a7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186fbc120ada4fb18c742d9647a1eaf3" is_pre_election: true
I20240503 08:11:13.844458 22970 raft_consensus.cc:2454] T 6649209db9b14787a4097bdf400c486e P efbd40aa19704f85bbf073eae5bc6fbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73e51753b3ba48d4b7426a540f448a7a in term 0.
I20240503 08:11:13.844514 22840 raft_consensus.cc:2454] T 6649209db9b14787a4097bdf400c486e P 186fbc120ada4fb18c742d9647a1eaf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73e51753b3ba48d4b7426a540f448a7a in term 0.
I20240503 08:11:13.845062 22644 leader_election.cc:302] T 6649209db9b14787a4097bdf400c486e P 73e51753b3ba48d4b7426a540f448a7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 186fbc120ada4fb18c742d9647a1eaf3, 73e51753b3ba48d4b7426a540f448a7a, efbd40aa19704f85bbf073eae5bc6fbc; no voters:
I20240503 08:11:13.845403 23042 raft_consensus.cc:2781] T 6649209db9b14787a4097bdf400c486e P 73e51753b3ba48d4b7426a540f448a7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:13.845535 23042 raft_consensus.cc:482] T 6649209db9b14787a4097bdf400c486e P 73e51753b3ba48d4b7426a540f448a7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:13.845618 23042 raft_consensus.cc:3037] T 6649209db9b14787a4097bdf400c486e P 73e51753b3ba48d4b7426a540f448a7a [term 0 FOLLOWER]: Advancing to term 1
W20240503 08:11:13.864969 22886 tablet.cc:2369] T 6649209db9b14787a4097bdf400c486e P 186fbc120ada4fb18c742d9647a1eaf3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:11:13.918740 23042 raft_consensus.cc:504] T 6649209db9b14787a4097bdf400c486e P 73e51753b3ba48d4b7426a540f448a7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbd40aa19704f85bbf073eae5bc6fbc" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 40255 } } peers { permanent_uuid: "73e51753b3ba48d4b7426a540f448a7a" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 41279 } } peers { permanent_uuid: "186fbc120ada4fb18c742d9647a1eaf3" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 37403 } }
I20240503 08:11:13.919492 23042 leader_election.cc:288] T 6649209db9b14787a4097bdf400c486e P 73e51753b3ba48d4b7426a540f448a7a [CANDIDATE]: Term 1 election: Requested vote from peers efbd40aa19704f85bbf073eae5bc6fbc (127.20.230.69:40255), 186fbc120ada4fb18c742d9647a1eaf3 (127.20.230.68:37403)
I20240503 08:11:13.920092 22840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6649209db9b14787a4097bdf400c486e" candidate_uuid: "73e51753b3ba48d4b7426a540f448a7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186fbc120ada4fb18c742d9647a1eaf3"
I20240503 08:11:13.920073 22970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6649209db9b14787a4097bdf400c486e" candidate_uuid: "73e51753b3ba48d4b7426a540f448a7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efbd40aa19704f85bbf073eae5bc6fbc"
I20240503 08:11:13.920302 22840 raft_consensus.cc:3037] T 6649209db9b14787a4097bdf400c486e P 186fbc120ada4fb18c742d9647a1eaf3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:13.920302 22970 raft_consensus.cc:3037] T 6649209db9b14787a4097bdf400c486e P efbd40aa19704f85bbf073eae5bc6fbc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:14.020869 22970 raft_consensus.cc:2454] T 6649209db9b14787a4097bdf400c486e P efbd40aa19704f85bbf073eae5bc6fbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73e51753b3ba48d4b7426a540f448a7a in term 1.
I20240503 08:11:14.020872 22840 raft_consensus.cc:2454] T 6649209db9b14787a4097bdf400c486e P 186fbc120ada4fb18c742d9647a1eaf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73e51753b3ba48d4b7426a540f448a7a in term 1.
I20240503 08:11:14.021559 22646 leader_election.cc:302] T 6649209db9b14787a4097bdf400c486e P 73e51753b3ba48d4b7426a540f448a7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186fbc120ada4fb18c742d9647a1eaf3, 73e51753b3ba48d4b7426a540f448a7a; no voters:
I20240503 08:11:14.021932 23042 raft_consensus.cc:2781] T 6649209db9b14787a4097bdf400c486e P 73e51753b3ba48d4b7426a540f448a7a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:14.022384 23042 raft_consensus.cc:686] T 6649209db9b14787a4097bdf400c486e P 73e51753b3ba48d4b7426a540f448a7a [term 1 LEADER]: Becoming Leader. State: Replica: 73e51753b3ba48d4b7426a540f448a7a, State: Running, Role: LEADER
I20240503 08:11:14.022903 23042 consensus_queue.cc:237] T 6649209db9b14787a4097bdf400c486e P 73e51753b3ba48d4b7426a540f448a7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbd40aa19704f85bbf073eae5bc6fbc" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 40255 } } peers { permanent_uuid: "73e51753b3ba48d4b7426a540f448a7a" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 41279 } } peers { permanent_uuid: "186fbc120ada4fb18c742d9647a1eaf3" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 37403 } }
I20240503 08:11:14.027447 22306 catalog_manager.cc:5484] T 6649209db9b14787a4097bdf400c486e P 73e51753b3ba48d4b7426a540f448a7a reported cstate change: term changed from 0 to 1, leader changed from <none> to 73e51753b3ba48d4b7426a540f448a7a (127.20.230.67). New cstate: current_term: 1 leader_uuid: "73e51753b3ba48d4b7426a540f448a7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbd40aa19704f85bbf073eae5bc6fbc" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 40255 } } peers { permanent_uuid: "73e51753b3ba48d4b7426a540f448a7a" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 41279 } } peers { permanent_uuid: "186fbc120ada4fb18c742d9647a1eaf3" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 37403 } } }
I20240503 08:11:14.064249 21401 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 08:11:14.066345 21401 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 47fc50e3aaad476c9b9fb7b385093f23 to finish bootstrapping
I20240503 08:11:14.076014 21401 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 13870b7641a74621bdfe20cd49c0dbc9 to finish bootstrapping
I20240503 08:11:14.084370 21401 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 73e51753b3ba48d4b7426a540f448a7a to finish bootstrapping
I20240503 08:11:14.091184 21401 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 186fbc120ada4fb18c742d9647a1eaf3 to finish bootstrapping
I20240503 08:11:14.097752 21401 ts_itest-base.cc:233] Waiting for 1 tablets on tserver efbd40aa19704f85bbf073eae5bc6fbc to finish bootstrapping
I20240503 08:11:14.104369 21401 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8bBFGA/build/asan/bin/kudu pid 22367
I20240503 08:11:14.105244 22489 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:14.188717 21401 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8bBFGA/build/asan/bin/kudu with pid 22367
I20240503 08:11:14.476213 23042 consensus_queue.cc:1035] T 6649209db9b14787a4097bdf400c486e P 73e51753b3ba48d4b7426a540f448a7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efbd40aa19704f85bbf073eae5bc6fbc" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 40255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:14.485189 23042 consensus_queue.cc:1035] T 6649209db9b14787a4097bdf400c486e P 73e51753b3ba48d4b7426a540f448a7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "186fbc120ada4fb18c742d9647a1eaf3" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 37403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:14.646946 22820 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:14.657624 22560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:14.666049 22950 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:14.666083 22690 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:14.785521 22560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:14.786489 22820 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:14.798202 22690 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:14.799283 22950 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:14.951452 22560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:14.955651 22820 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:14.956602 22690 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:14.960516 22950 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:15.543124 22560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:15.546056 22820 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:15.567195 22950 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:15.567916 22690 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:15.684943 22560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:15.686058 22690 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:15.697975 22950 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:15.706652 22820 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:15.858824 22820 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:15.860493 22560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:15.862974 22690 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:15.863912 22950 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:16.007730 21401 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8bBFGA/build/asan/bin/kudu pid 22497
I20240503 08:11:16.009042 22620 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:16.101105 21401 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8bBFGA/build/asan/bin/kudu with pid 22497
I20240503 08:11:16.115372 21401 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8bBFGA/build/asan/bin/kudu pid 22628
I20240503 08:11:16.116391 22749 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:16.214337 21401 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8bBFGA/build/asan/bin/kudu with pid 22628
I20240503 08:11:16.228642 21401 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8bBFGA/build/asan/bin/kudu pid 22758
I20240503 08:11:16.229574 22880 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:16.328325 21401 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8bBFGA/build/asan/bin/kudu with pid 22758
I20240503 08:11:16.342316 21401 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8bBFGA/build/asan/bin/kudu pid 22888
I20240503 08:11:16.343262 23010 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:16.441871 21401 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8bBFGA/build/asan/bin/kudu with pid 22888
I20240503 08:11:16.456138 21401 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8bBFGA/build/asan/bin/kudu pid 22276
I20240503 08:11:16.457193 22337 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:16.571079 21401 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8bBFGA/build/asan/bin/kudu with pid 22276
2024-05-03T08:11:16Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.TabletServerIsDown/0 (5696 ms)
[----------] 1 test from RebalanceStartCriteriaTest (5696 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/0
I20240503 08:11:16.607941 21401 test_util.cc:268] Using random seed: 1133352879
I20240503 08:11:16.609704 21401 ts_itest-base.cc:112] Starting cluster with:
I20240503 08:11:16.609799 21401 ts_itest-base.cc:113] --------------
I20240503 08:11:16.609853 21401 ts_itest-base.cc:114] 7 tablet servers
I20240503 08:11:16.609894 21401 ts_itest-base.cc:115] 1 replicas per TS
I20240503 08:11:16.609938 21401 ts_itest-base.cc:116] --------------
2024-05-03T08:11:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:11:16Z Disabled control of system clock
I20240503 08:11:16.632722 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.230.126:38671
--webserver_interface=127.20.230.126
--webserver_port=0
--builtin_ntp_servers=127.20.230.84:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.230.126:38671
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240503 08:11:16.952659 23255 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 08:11:16.952986 23255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:16.953080 23255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:16.953183 23255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:16.960677 23255 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:11:16.960786 23255 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:11:16.960852 23255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:16.960916 23255 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:11:16.960992 23255 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:11:16.970172 23255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.20.230.126:38671
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.230.126:38671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.230.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:16.971535 23255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:16.973335 23255 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:16.984076 23260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:16.984828 23261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:16.985432 23263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:16.986200 23255 server_base.cc:1034] running on GCE node
I20240503 08:11:16.986979 23255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:16.988356 23255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:16.989583 23255 hybrid_clock.cc:648] HybridClock initialized: now 1714723876989509 us; error 51 us; skew 500 ppm
I20240503 08:11:16.989931 23255 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:16.992180 23255 webserver.cc:460] Webserver started at http://127.20.230.126:44897/ using document root <none> and password file <none>
I20240503 08:11:16.992888 23255 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:16.993021 23255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:16.993324 23255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:16.995981 23255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "0842171f0fd24fa69ee8e4099dc61148"
format_stamp: "Formatted at 2024-05-03 08:11:16 on dist-test-slave-6vp8"
I20240503 08:11:16.996722 23255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "0842171f0fd24fa69ee8e4099dc61148"
format_stamp: "Formatted at 2024-05-03 08:11:16 on dist-test-slave-6vp8"
I20240503 08:11:17.002187 23255 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20240503 08:11:17.005894 23269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:17.007505 23255 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 08:11:17.007694 23255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/wal
uuid: "0842171f0fd24fa69ee8e4099dc61148"
format_stamp: "Formatted at 2024-05-03 08:11:16 on dist-test-slave-6vp8"
I20240503 08:11:17.007933 23255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:17.021397 23255 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:17.022266 23255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:17.022572 23255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:17.040299 23255 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.126:38671
I20240503 08:11:17.040333 23320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.126:38671 every 8 connection(s)
I20240503 08:11:17.041734 23255 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 08:11:17.045539 23321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:17.048094 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 23255
I20240503 08:11:17.048271 21401 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 08:11:17.056599 23321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0842171f0fd24fa69ee8e4099dc61148: Bootstrap starting.
I20240503 08:11:17.060047 23321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0842171f0fd24fa69ee8e4099dc61148: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:17.061223 23321 log.cc:824] T 00000000000000000000000000000000 P 0842171f0fd24fa69ee8e4099dc61148: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:17.064250 23321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0842171f0fd24fa69ee8e4099dc61148: No bootstrap required, opened a new log
I20240503 08:11:17.071084 23321 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0842171f0fd24fa69ee8e4099dc61148 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0842171f0fd24fa69ee8e4099dc61148" member_type: VOTER last_known_addr { host: "127.20.230.126" port: 38671 } }
I20240503 08:11:17.071460 23321 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0842171f0fd24fa69ee8e4099dc61148 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:17.071575 23321 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0842171f0fd24fa69ee8e4099dc61148 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0842171f0fd24fa69ee8e4099dc61148, State: Initialized, Role: FOLLOWER
I20240503 08:11:17.072263 23321 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0842171f0fd24fa69ee8e4099dc61148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0842171f0fd24fa69ee8e4099dc61148" member_type: VOTER last_known_addr { host: "127.20.230.126" port: 38671 } }
I20240503 08:11:17.072469 23321 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0842171f0fd24fa69ee8e4099dc61148 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:17.072592 23321 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0842171f0fd24fa69ee8e4099dc61148 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:17.072734 23321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0842171f0fd24fa69ee8e4099dc61148 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:17.074829 23321 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0842171f0fd24fa69ee8e4099dc61148 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0842171f0fd24fa69ee8e4099dc61148" member_type: VOTER last_known_addr { host: "127.20.230.126" port: 38671 } }
I20240503 08:11:17.075333 23321 leader_election.cc:302] T 00000000000000000000000000000000 P 0842171f0fd24fa69ee8e4099dc61148 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0842171f0fd24fa69ee8e4099dc61148; no voters:
I20240503 08:11:17.075946 23321 leader_election.cc:288] T 00000000000000000000000000000000 P 0842171f0fd24fa69ee8e4099dc61148 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:17.076097 23326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0842171f0fd24fa69ee8e4099dc61148 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:17.076678 23326 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0842171f0fd24fa69ee8e4099dc61148 [term 1 LEADER]: Becoming Leader. State: Replica: 0842171f0fd24fa69ee8e4099dc61148, State: Running, Role: LEADER
I20240503 08:11:17.077440 23326 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0842171f0fd24fa69ee8e4099dc61148 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0842171f0fd24fa69ee8e4099dc61148" member_type: VOTER last_known_addr { host: "127.20.230.126" port: 38671 } }
I20240503 08:11:17.078387 23321 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0842171f0fd24fa69ee8e4099dc61148 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:11:17.080314 23327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0842171f0fd24fa69ee8e4099dc61148 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0842171f0fd24fa69ee8e4099dc61148" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0842171f0fd24fa69ee8e4099dc61148" member_type: VOTER last_known_addr { host: "127.20.230.126" port: 38671 } } }
I20240503 08:11:17.080359 23328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0842171f0fd24fa69ee8e4099dc61148 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0842171f0fd24fa69ee8e4099dc61148. Latest consensus state: current_term: 1 leader_uuid: "0842171f0fd24fa69ee8e4099dc61148" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0842171f0fd24fa69ee8e4099dc61148" member_type: VOTER last_known_addr { host: "127.20.230.126" port: 38671 } } }
I20240503 08:11:17.080690 23327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0842171f0fd24fa69ee8e4099dc61148 [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:17.080687 23328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0842171f0fd24fa69ee8e4099dc61148 [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:17.085353 23332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:11:17.091558 23332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:11:17.100257 23332 catalog_manager.cc:1347] Generated new cluster ID: 4d6a06e57bc145509cc5a213d171ceb3
I20240503 08:11:17.100402 23332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:11:17.107925 23332 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:11:17.109280 23332 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:11:17.117059 23332 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 0842171f0fd24fa69ee8e4099dc61148: Generated new TSK 0
I20240503 08:11:17.117959 23332 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:11:17.129043 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.65:0
--local_ip_for_outbound_sockets=127.20.230.65
--webserver_interface=127.20.230.65
--webserver_port=0
--tserver_master_addrs=127.20.230.126:38671
--builtin_ntp_servers=127.20.230.84:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 08:11:17.448727 23345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:17.448982 23345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:17.449106 23345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:17.456357 23345 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:11:17.456463 23345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:17.456617 23345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.65
I20240503 08:11:17.465209 23345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.230.65
--webserver_port=0
--tserver_master_addrs=127.20.230.126:38671
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.65
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:17.466416 23345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:17.468039 23345 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:17.479097 23350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:17.479821 23351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:17.480398 23353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:17.481098 23345 server_base.cc:1034] running on GCE node
I20240503 08:11:17.481631 23345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:17.482494 23345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:17.483688 23345 hybrid_clock.cc:648] HybridClock initialized: now 1714723877483647 us; error 36 us; skew 500 ppm
I20240503 08:11:17.483966 23345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:17.491255 23345 webserver.cc:460] Webserver started at http://127.20.230.65:39981/ using document root <none> and password file <none>
I20240503 08:11:17.492019 23345 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:17.492136 23345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:17.492458 23345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:17.495108 23345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "76e30a81c8b8451898078e9636a1104b"
format_stamp: "Formatted at 2024-05-03 08:11:17 on dist-test-slave-6vp8"
I20240503 08:11:17.495888 23345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "76e30a81c8b8451898078e9636a1104b"
format_stamp: "Formatted at 2024-05-03 08:11:17 on dist-test-slave-6vp8"
I20240503 08:11:17.500738 23345 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20240503 08:11:17.503969 23359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:17.505275 23345 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240503 08:11:17.505421 23345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "76e30a81c8b8451898078e9636a1104b"
format_stamp: "Formatted at 2024-05-03 08:11:17 on dist-test-slave-6vp8"
I20240503 08:11:17.505601 23345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:17.522437 23345 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:17.523181 23345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:17.523437 23345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:17.524299 23345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:17.525744 23345 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:17.525856 23345 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:17.525961 23345 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:17.526037 23345 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:17.554172 23345 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.65:37573
I20240503 08:11:17.554221 23471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.65:37573 every 8 connection(s)
I20240503 08:11:17.555402 23345 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 08:11:17.565694 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 23345
I20240503 08:11:17.565946 21401 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 08:11:17.567193 23472 heartbeater.cc:347] Connected to a master server at 127.20.230.126:38671
I20240503 08:11:17.567457 23472 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:17.568117 23472 heartbeater.cc:511] Master 127.20.230.126:38671 requested a full tablet report, sending...
I20240503 08:11:17.569083 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.66:0
--local_ip_for_outbound_sockets=127.20.230.66
--webserver_interface=127.20.230.66
--webserver_port=0
--tserver_master_addrs=127.20.230.126:38671
--builtin_ntp_servers=127.20.230.84:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:11:17.570052 23286 ts_manager.cc:194] Registered new tserver with Master: 76e30a81c8b8451898078e9636a1104b (127.20.230.65:37573)
I20240503 08:11:17.571463 23286 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.65:49911
W20240503 08:11:17.881443 23475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:17.881728 23475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:17.881880 23475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:17.889173 23475 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:11:17.889318 23475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:17.889467 23475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.66
I20240503 08:11:17.898339 23475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.230.66
--webserver_port=0
--tserver_master_addrs=127.20.230.126:38671
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.66
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:17.899601 23475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:17.901242 23475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:17.912369 23480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:17.912904 23483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:17.913658 23481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:17.914204 23475 server_base.cc:1034] running on GCE node
I20240503 08:11:17.914719 23475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:17.915575 23475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:17.916770 23475 hybrid_clock.cc:648] HybridClock initialized: now 1714723877916730 us; error 34 us; skew 500 ppm
I20240503 08:11:17.917052 23475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:17.919124 23475 webserver.cc:460] Webserver started at http://127.20.230.66:42489/ using document root <none> and password file <none>
I20240503 08:11:17.919791 23475 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:17.919898 23475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:17.920174 23475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:17.922605 23475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "30d88bcd65b347dabcbac54465e52b21"
format_stamp: "Formatted at 2024-05-03 08:11:17 on dist-test-slave-6vp8"
I20240503 08:11:17.923321 23475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "30d88bcd65b347dabcbac54465e52b21"
format_stamp: "Formatted at 2024-05-03 08:11:17 on dist-test-slave-6vp8"
I20240503 08:11:17.927947 23475 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 08:11:17.931039 23489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:17.932363 23475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:11:17.932509 23475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "30d88bcd65b347dabcbac54465e52b21"
format_stamp: "Formatted at 2024-05-03 08:11:17 on dist-test-slave-6vp8"
I20240503 08:11:17.932685 23475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:17.962661 23475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:17.963431 23475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:17.963696 23475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:17.964771 23475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:17.966387 23475 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:17.966485 23475 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:17.966609 23475 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:17.966679 23475 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:17.995455 23475 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.66:33351
I20240503 08:11:17.995517 23601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.66:33351 every 8 connection(s)
I20240503 08:11:17.996672 23475 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 08:11:18.003265 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 23475
I20240503 08:11:18.003448 21401 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 08:11:18.006841 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.67:0
--local_ip_for_outbound_sockets=127.20.230.67
--webserver_interface=127.20.230.67
--webserver_port=0
--tserver_master_addrs=127.20.230.126:38671
--builtin_ntp_servers=127.20.230.84:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:11:18.011387 23602 heartbeater.cc:347] Connected to a master server at 127.20.230.126:38671
I20240503 08:11:18.011659 23602 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:18.012413 23602 heartbeater.cc:511] Master 127.20.230.126:38671 requested a full tablet report, sending...
I20240503 08:11:18.013741 23286 ts_manager.cc:194] Registered new tserver with Master: 30d88bcd65b347dabcbac54465e52b21 (127.20.230.66:33351)
I20240503 08:11:18.014480 23286 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.66:33563
W20240503 08:11:18.305353 23605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:18.305647 23605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:18.305801 23605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:18.313108 23605 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:11:18.313228 23605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:18.313372 23605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.67
I20240503 08:11:18.321983 23605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.230.67
--webserver_port=0
--tserver_master_addrs=127.20.230.126:38671
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.67
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:18.323210 23605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:18.324831 23605 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:18.336349 23613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:18.337024 23610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:18.337400 23611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:18.338050 23605 server_base.cc:1034] running on GCE node
I20240503 08:11:18.338626 23605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:18.339493 23605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:18.340698 23605 hybrid_clock.cc:648] HybridClock initialized: now 1714723878340653 us; error 45 us; skew 500 ppm
I20240503 08:11:18.340993 23605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:18.343078 23605 webserver.cc:460] Webserver started at http://127.20.230.67:38497/ using document root <none> and password file <none>
I20240503 08:11:18.343837 23605 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:18.343955 23605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:18.344241 23605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:18.346725 23605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "5ea58b5961b54598878bad3f5660913c"
format_stamp: "Formatted at 2024-05-03 08:11:18 on dist-test-slave-6vp8"
I20240503 08:11:18.347457 23605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "5ea58b5961b54598878bad3f5660913c"
format_stamp: "Formatted at 2024-05-03 08:11:18 on dist-test-slave-6vp8"
I20240503 08:11:18.352394 23605 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 08:11:18.360206 23619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:18.361430 23605 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 08:11:18.361580 23605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "5ea58b5961b54598878bad3f5660913c"
format_stamp: "Formatted at 2024-05-03 08:11:18 on dist-test-slave-6vp8"
I20240503 08:11:18.361753 23605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:18.380561 23605 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:18.381314 23605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:18.381569 23605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:18.382586 23605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:18.383934 23605 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:18.384025 23605 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:18.384124 23605 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:18.384188 23605 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:18.412191 23605 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.67:37877
I20240503 08:11:18.412272 23731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.67:37877 every 8 connection(s)
I20240503 08:11:18.413426 23605 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:11:18.422375 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 23605
I20240503 08:11:18.422567 21401 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 08:11:18.425177 23732 heartbeater.cc:347] Connected to a master server at 127.20.230.126:38671
I20240503 08:11:18.425444 23732 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:18.425736 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.68:0
--local_ip_for_outbound_sockets=127.20.230.68
--webserver_interface=127.20.230.68
--webserver_port=0
--tserver_master_addrs=127.20.230.126:38671
--builtin_ntp_servers=127.20.230.84:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:11:18.426175 23732 heartbeater.cc:511] Master 127.20.230.126:38671 requested a full tablet report, sending...
I20240503 08:11:18.427480 23286 ts_manager.cc:194] Registered new tserver with Master: 5ea58b5961b54598878bad3f5660913c (127.20.230.67:37877)
I20240503 08:11:18.428436 23286 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.67:39147
I20240503 08:11:18.574041 23472 heartbeater.cc:503] Master 127.20.230.126:38671 was elected leader, sending a full tablet report...
W20240503 08:11:18.732395 23735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:18.732661 23735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:18.732774 23735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:18.740072 23735 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:11:18.740185 23735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:18.740329 23735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.68
I20240503 08:11:18.748991 23735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.230.68
--webserver_port=0
--tserver_master_addrs=127.20.230.126:38671
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.68
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:18.750341 23735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:18.752092 23735 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:18.763873 23740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:18.764581 23741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:18.765184 23743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:18.765866 23735 server_base.cc:1034] running on GCE node
I20240503 08:11:18.766460 23735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:18.767341 23735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:18.768546 23735 hybrid_clock.cc:648] HybridClock initialized: now 1714723878768495 us; error 44 us; skew 500 ppm
I20240503 08:11:18.768862 23735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:18.771126 23735 webserver.cc:460] Webserver started at http://127.20.230.68:43945/ using document root <none> and password file <none>
I20240503 08:11:18.771834 23735 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:18.771950 23735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:18.772261 23735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:18.774814 23735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "51d0bc1740db4723a71f1d56b7e6e6b7"
format_stamp: "Formatted at 2024-05-03 08:11:18 on dist-test-slave-6vp8"
I20240503 08:11:18.775564 23735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "51d0bc1740db4723a71f1d56b7e6e6b7"
format_stamp: "Formatted at 2024-05-03 08:11:18 on dist-test-slave-6vp8"
I20240503 08:11:18.780623 23735 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240503 08:11:18.784147 23749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:18.785502 23735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 08:11:18.785645 23735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "51d0bc1740db4723a71f1d56b7e6e6b7"
format_stamp: "Formatted at 2024-05-03 08:11:18 on dist-test-slave-6vp8"
I20240503 08:11:18.785866 23735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:18.804607 23735 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:18.805420 23735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:18.805680 23735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:18.806605 23735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:18.808094 23735 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:18.808182 23735 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:18.808292 23735 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:18.808357 23735 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:18.837563 23735 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.68:44977
I20240503 08:11:18.837623 23861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.68:44977 every 8 connection(s)
I20240503 08:11:18.838865 23735 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:11:18.841248 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 23735
I20240503 08:11:18.841416 21401 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 08:11:18.845118 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.69:0
--local_ip_for_outbound_sockets=127.20.230.69
--webserver_interface=127.20.230.69
--webserver_port=0
--tserver_master_addrs=127.20.230.126:38671
--builtin_ntp_servers=127.20.230.84:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:11:18.852348 23862 heartbeater.cc:347] Connected to a master server at 127.20.230.126:38671
I20240503 08:11:18.852641 23862 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:18.853387 23862 heartbeater.cc:511] Master 127.20.230.126:38671 requested a full tablet report, sending...
I20240503 08:11:18.854815 23286 ts_manager.cc:194] Registered new tserver with Master: 51d0bc1740db4723a71f1d56b7e6e6b7 (127.20.230.68:44977)
I20240503 08:11:18.855870 23286 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.68:47079
I20240503 08:11:19.016782 23602 heartbeater.cc:503] Master 127.20.230.126:38671 was elected leader, sending a full tablet report...
W20240503 08:11:19.157727 23865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:19.157999 23865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:19.158119 23865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:19.165216 23865 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:11:19.165340 23865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:19.165477 23865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.69
I20240503 08:11:19.173921 23865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.230.69
--webserver_port=0
--tserver_master_addrs=127.20.230.126:38671
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.69
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:19.175055 23865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:19.176635 23865 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:19.187852 23870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:19.188700 23871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:19.189334 23873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:19.190016 23865 server_base.cc:1034] running on GCE node
I20240503 08:11:19.190547 23865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:19.191401 23865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:19.192606 23865 hybrid_clock.cc:648] HybridClock initialized: now 1714723879192567 us; error 41 us; skew 500 ppm
I20240503 08:11:19.192875 23865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:19.194900 23865 webserver.cc:460] Webserver started at http://127.20.230.69:43303/ using document root <none> and password file <none>
I20240503 08:11:19.195534 23865 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:19.195629 23865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:19.195909 23865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:19.198259 23865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "509c32504f544f8b90ebf66b6dd951f7"
format_stamp: "Formatted at 2024-05-03 08:11:19 on dist-test-slave-6vp8"
I20240503 08:11:19.198968 23865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "509c32504f544f8b90ebf66b6dd951f7"
format_stamp: "Formatted at 2024-05-03 08:11:19 on dist-test-slave-6vp8"
I20240503 08:11:19.203409 23865 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 08:11:19.206357 23879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:19.207528 23865 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:11:19.207652 23865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "509c32504f544f8b90ebf66b6dd951f7"
format_stamp: "Formatted at 2024-05-03 08:11:19 on dist-test-slave-6vp8"
I20240503 08:11:19.207854 23865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:19.232429 23865 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:19.233211 23865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:19.233469 23865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:19.234413 23865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:19.235766 23865 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:19.235846 23865 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:19.235946 23865 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:19.236009 23865 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:19.263801 23865 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.69:41435
I20240503 08:11:19.263851 23991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.69:41435 every 8 connection(s)
I20240503 08:11:19.265043 23865 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:11:19.273306 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 23865
I20240503 08:11:19.273519 21401 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 08:11:19.277184 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.70:0
--local_ip_for_outbound_sockets=127.20.230.70
--webserver_interface=127.20.230.70
--webserver_port=0
--tserver_master_addrs=127.20.230.126:38671
--builtin_ntp_servers=127.20.230.84:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:11:19.278491 23992 heartbeater.cc:347] Connected to a master server at 127.20.230.126:38671
I20240503 08:11:19.278820 23992 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:19.279564 23992 heartbeater.cc:511] Master 127.20.230.126:38671 requested a full tablet report, sending...
I20240503 08:11:19.280829 23286 ts_manager.cc:194] Registered new tserver with Master: 509c32504f544f8b90ebf66b6dd951f7 (127.20.230.69:41435)
I20240503 08:11:19.281596 23286 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.69:35763
I20240503 08:11:19.430760 23732 heartbeater.cc:503] Master 127.20.230.126:38671 was elected leader, sending a full tablet report...
W20240503 08:11:19.585199 23995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:19.585484 23995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:19.585602 23995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:19.592849 23995 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:11:19.592975 23995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:19.593125 23995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.70
I20240503 08:11:19.601812 23995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.20.230.70
--webserver_port=0
--tserver_master_addrs=127.20.230.126:38671
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.70
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:19.603091 23995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:19.604763 23995 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:19.616242 24000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:19.616786 24003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:19.617537 24001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:19.618119 23995 server_base.cc:1034] running on GCE node
I20240503 08:11:19.618646 23995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:19.619499 23995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:19.620688 23995 hybrid_clock.cc:648] HybridClock initialized: now 1714723879620647 us; error 41 us; skew 500 ppm
I20240503 08:11:19.620970 23995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:19.622956 23995 webserver.cc:460] Webserver started at http://127.20.230.70:41101/ using document root <none> and password file <none>
I20240503 08:11:19.623617 23995 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:19.623721 23995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:19.624006 23995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:19.626416 23995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "c82f2e5e9e13405d9328620179077098"
format_stamp: "Formatted at 2024-05-03 08:11:19 on dist-test-slave-6vp8"
I20240503 08:11:19.627138 23995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "c82f2e5e9e13405d9328620179077098"
format_stamp: "Formatted at 2024-05-03 08:11:19 on dist-test-slave-6vp8"
I20240503 08:11:19.631812 23995 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20240503 08:11:19.634842 24009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:19.636014 23995 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:11:19.636153 23995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "c82f2e5e9e13405d9328620179077098"
format_stamp: "Formatted at 2024-05-03 08:11:19 on dist-test-slave-6vp8"
I20240503 08:11:19.636319 23995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:19.651837 23995 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:19.652616 23995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:19.652863 23995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:19.653705 23995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:19.655028 23995 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:19.655118 23995 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:19.655220 23995 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:19.655287 23995 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:19.683087 23995 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.70:36811
I20240503 08:11:19.683130 24121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.70:36811 every 8 connection(s)
I20240503 08:11:19.684304 23995 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 08:11:19.692322 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 23995
I20240503 08:11:19.692492 21401 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 08:11:19.695690 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.71:0
--local_ip_for_outbound_sockets=127.20.230.71
--webserver_interface=127.20.230.71
--webserver_port=0
--tserver_master_addrs=127.20.230.126:38671
--builtin_ntp_servers=127.20.230.84:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:11:19.696283 24122 heartbeater.cc:347] Connected to a master server at 127.20.230.126:38671
I20240503 08:11:19.696542 24122 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:19.697190 24122 heartbeater.cc:511] Master 127.20.230.126:38671 requested a full tablet report, sending...
I20240503 08:11:19.698375 23286 ts_manager.cc:194] Registered new tserver with Master: c82f2e5e9e13405d9328620179077098 (127.20.230.70:36811)
I20240503 08:11:19.699071 23286 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.70:33697
I20240503 08:11:19.858345 23862 heartbeater.cc:503] Master 127.20.230.126:38671 was elected leader, sending a full tablet report...
W20240503 08:11:20.003225 24125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:20.003490 24125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:20.003577 24125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:20.010643 24125 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:11:20.010756 24125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:20.010870 24125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.71
I20240503 08:11:20.020282 24125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.20.230.71
--webserver_port=0
--tserver_master_addrs=127.20.230.126:38671
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.71
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:20.021536 24125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:20.023193 24125 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:20.034818 24130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:20.035356 24133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:20.036125 24131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:20.036710 24125 server_base.cc:1034] running on GCE node
I20240503 08:11:20.037217 24125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:20.038123 24125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:20.039309 24125 hybrid_clock.cc:648] HybridClock initialized: now 1714723880039261 us; error 38 us; skew 500 ppm
I20240503 08:11:20.039630 24125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:20.046934 24125 webserver.cc:460] Webserver started at http://127.20.230.71:45417/ using document root <none> and password file <none>
I20240503 08:11:20.047632 24125 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:20.047735 24125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:20.048022 24125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:20.050521 24125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "7823cb7f30b045209bc5ccfb110bf199"
format_stamp: "Formatted at 2024-05-03 08:11:20 on dist-test-slave-6vp8"
I20240503 08:11:20.051257 24125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "7823cb7f30b045209bc5ccfb110bf199"
format_stamp: "Formatted at 2024-05-03 08:11:20 on dist-test-slave-6vp8"
I20240503 08:11:20.056510 24125 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 08:11:20.059733 24139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:20.061002 24125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240503 08:11:20.061143 24125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "7823cb7f30b045209bc5ccfb110bf199"
format_stamp: "Formatted at 2024-05-03 08:11:20 on dist-test-slave-6vp8"
I20240503 08:11:20.061322 24125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:20.076889 24125 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:20.077661 24125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:20.077934 24125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:20.079268 24125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:20.080725 24125 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:20.080806 24125 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:20.080906 24125 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:20.080969 24125 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:20.111984 24125 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.71:38073
I20240503 08:11:20.111985 24251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.71:38073 every 8 connection(s)
I20240503 08:11:20.113298 24125 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 08:11:20.120334 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 24125
I20240503 08:11:20.120522 21401 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 08:11:20.125677 24252 heartbeater.cc:347] Connected to a master server at 127.20.230.126:38671
I20240503 08:11:20.125964 24252 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:20.126642 24252 heartbeater.cc:511] Master 127.20.230.126:38671 requested a full tablet report, sending...
I20240503 08:11:20.127835 23286 ts_manager.cc:194] Registered new tserver with Master: 7823cb7f30b045209bc5ccfb110bf199 (127.20.230.71:38073)
I20240503 08:11:20.128563 23286 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.71:44253
I20240503 08:11:20.135996 21401 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 08:11:20.151522 23286 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58436:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 08:11:20.179158 24057 tablet_service.cc:1467] Processing CreateTablet for tablet 7879251014c84b11a0f49d707d80d85b (DEFAULT_TABLE table=TestTable [id=dddf1828352f4800a880448de4f7ab8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:20.180409 24057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7879251014c84b11a0f49d707d80d85b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:20.191102 24266 tablet_bootstrap.cc:492] T 7879251014c84b11a0f49d707d80d85b P c82f2e5e9e13405d9328620179077098: Bootstrap starting.
I20240503 08:11:20.194108 24266 tablet_bootstrap.cc:654] T 7879251014c84b11a0f49d707d80d85b P c82f2e5e9e13405d9328620179077098: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:20.195091 24266 log.cc:824] T 7879251014c84b11a0f49d707d80d85b P c82f2e5e9e13405d9328620179077098: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:20.197230 24266 tablet_bootstrap.cc:492] T 7879251014c84b11a0f49d707d80d85b P c82f2e5e9e13405d9328620179077098: No bootstrap required, opened a new log
I20240503 08:11:20.197456 24266 ts_tablet_manager.cc:1367] T 7879251014c84b11a0f49d707d80d85b P c82f2e5e9e13405d9328620179077098: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20240503 08:11:20.203698 24266 raft_consensus.cc:348] T 7879251014c84b11a0f49d707d80d85b P c82f2e5e9e13405d9328620179077098 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c82f2e5e9e13405d9328620179077098" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 36811 } }
I20240503 08:11:20.203980 24266 raft_consensus.cc:374] T 7879251014c84b11a0f49d707d80d85b P c82f2e5e9e13405d9328620179077098 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:20.204077 24266 raft_consensus.cc:729] T 7879251014c84b11a0f49d707d80d85b P c82f2e5e9e13405d9328620179077098 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c82f2e5e9e13405d9328620179077098, State: Initialized, Role: FOLLOWER
I20240503 08:11:20.204650 24266 consensus_queue.cc:260] T 7879251014c84b11a0f49d707d80d85b P c82f2e5e9e13405d9328620179077098 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c82f2e5e9e13405d9328620179077098" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 36811 } }
I20240503 08:11:20.204835 24266 raft_consensus.cc:388] T 7879251014c84b11a0f49d707d80d85b P c82f2e5e9e13405d9328620179077098 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:20.204948 24266 raft_consensus.cc:482] T 7879251014c84b11a0f49d707d80d85b P c82f2e5e9e13405d9328620179077098 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:20.205077 24266 raft_consensus.cc:3037] T 7879251014c84b11a0f49d707d80d85b P c82f2e5e9e13405d9328620179077098 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:20.206924 24266 raft_consensus.cc:504] T 7879251014c84b11a0f49d707d80d85b P c82f2e5e9e13405d9328620179077098 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c82f2e5e9e13405d9328620179077098" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 36811 } }
I20240503 08:11:20.207362 24266 leader_election.cc:302] T 7879251014c84b11a0f49d707d80d85b P c82f2e5e9e13405d9328620179077098 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c82f2e5e9e13405d9328620179077098; no voters:
I20240503 08:11:20.208027 24266 leader_election.cc:288] T 7879251014c84b11a0f49d707d80d85b P c82f2e5e9e13405d9328620179077098 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:20.208120 24268 raft_consensus.cc:2781] T 7879251014c84b11a0f49d707d80d85b P c82f2e5e9e13405d9328620179077098 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:20.208575 24268 raft_consensus.cc:686] T 7879251014c84b11a0f49d707d80d85b P c82f2e5e9e13405d9328620179077098 [term 1 LEADER]: Becoming Leader. State: Replica: c82f2e5e9e13405d9328620179077098, State: Running, Role: LEADER
I20240503 08:11:20.209168 24268 consensus_queue.cc:237] T 7879251014c84b11a0f49d707d80d85b P c82f2e5e9e13405d9328620179077098 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c82f2e5e9e13405d9328620179077098" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 36811 } }
I20240503 08:11:20.209414 24122 heartbeater.cc:503] Master 127.20.230.126:38671 was elected leader, sending a full tablet report...
I20240503 08:11:20.210136 24266 ts_tablet_manager.cc:1398] T 7879251014c84b11a0f49d707d80d85b P c82f2e5e9e13405d9328620179077098: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20240503 08:11:20.214349 23286 catalog_manager.cc:5484] T 7879251014c84b11a0f49d707d80d85b P c82f2e5e9e13405d9328620179077098 reported cstate change: term changed from 0 to 1, leader changed from <none> to c82f2e5e9e13405d9328620179077098 (127.20.230.70). New cstate: current_term: 1 leader_uuid: "c82f2e5e9e13405d9328620179077098" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c82f2e5e9e13405d9328620179077098" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 36811 } } }
I20240503 08:11:20.223169 21401 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 08:11:20.225347 21401 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 76e30a81c8b8451898078e9636a1104b to finish bootstrapping
I20240503 08:11:20.233572 21401 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 30d88bcd65b347dabcbac54465e52b21 to finish bootstrapping
I20240503 08:11:20.241636 21401 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 5ea58b5961b54598878bad3f5660913c to finish bootstrapping
I20240503 08:11:20.249066 21401 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 51d0bc1740db4723a71f1d56b7e6e6b7 to finish bootstrapping
I20240503 08:11:20.256808 21401 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 509c32504f544f8b90ebf66b6dd951f7 to finish bootstrapping
I20240503 08:11:20.264497 21401 ts_itest-base.cc:233] Waiting for 1 tablets on tserver c82f2e5e9e13405d9328620179077098 to finish bootstrapping
I20240503 08:11:20.271890 21401 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 7823cb7f30b045209bc5ccfb110bf199 to finish bootstrapping
I20240503 08:11:20.280591 21401 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8bBFGA/build/asan/bin/kudu pid 23605
I20240503 08:11:20.281507 23727 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:20.283941 23992 heartbeater.cc:503] Master 127.20.230.126:38671 was elected leader, sending a full tablet report...
I20240503 08:11:20.364876 21401 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8bBFGA/build/asan/bin/kudu with pid 23605
I20240503 08:11:20.377247 21401 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8bBFGA/build/asan/bin/kudu pid 23735
I20240503 08:11:20.378376 23857 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:20.458555 21401 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8bBFGA/build/asan/bin/kudu with pid 23735
I20240503 08:11:20.471254 21401 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8bBFGA/build/asan/bin/kudu pid 23865
I20240503 08:11:20.472370 23987 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:20.550829 21401 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8bBFGA/build/asan/bin/kudu with pid 23865
I20240503 08:11:20.562916 21401 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8bBFGA/build/asan/bin/kudu pid 23995
I20240503 08:11:20.563936 24117 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:20.648521 21401 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8bBFGA/build/asan/bin/kudu with pid 23995
I20240503 08:11:20.661633 21401 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8bBFGA/build/asan/bin/kudu pid 24125
I20240503 08:11:20.662761 24247 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:20.741343 21401 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8bBFGA/build/asan/bin/kudu with pid 24125
I20240503 08:11:24.505445 23285 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58436:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:11:24.524852 23537 tablet_service.cc:1467] Processing CreateTablet for tablet 6bae8d67e9894646aae3f6230eda07c7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e75f9b41b7294dddba472b9ede90ddcb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:24.524864 23407 tablet_service.cc:1467] Processing CreateTablet for tablet 494f9374df1d45f5a0a8f7705e72557a (DEFAULT_TABLE table=rebalance_test_table_0 [id=e75f9b41b7294dddba472b9ede90ddcb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:24.524906 23406 tablet_service.cc:1467] Processing CreateTablet for tablet 78b4f0eae3d14b97b8f74cbc24389044 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e75f9b41b7294dddba472b9ede90ddcb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:24.526347 23537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bae8d67e9894646aae3f6230eda07c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:24.526386 23407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 494f9374df1d45f5a0a8f7705e72557a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:24.526831 23406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78b4f0eae3d14b97b8f74cbc24389044. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:24.537700 24293 tablet_bootstrap.cc:492] T 78b4f0eae3d14b97b8f74cbc24389044 P 76e30a81c8b8451898078e9636a1104b: Bootstrap starting.
I20240503 08:11:24.540701 24293 tablet_bootstrap.cc:654] T 78b4f0eae3d14b97b8f74cbc24389044 P 76e30a81c8b8451898078e9636a1104b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:24.541711 24293 log.cc:824] T 78b4f0eae3d14b97b8f74cbc24389044 P 76e30a81c8b8451898078e9636a1104b: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:24.542521 24294 tablet_bootstrap.cc:492] T 6bae8d67e9894646aae3f6230eda07c7 P 30d88bcd65b347dabcbac54465e52b21: Bootstrap starting.
I20240503 08:11:24.544835 24293 tablet_bootstrap.cc:492] T 78b4f0eae3d14b97b8f74cbc24389044 P 76e30a81c8b8451898078e9636a1104b: No bootstrap required, opened a new log
I20240503 08:11:24.545114 24293 ts_tablet_manager.cc:1367] T 78b4f0eae3d14b97b8f74cbc24389044 P 76e30a81c8b8451898078e9636a1104b: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240503 08:11:24.545845 24294 tablet_bootstrap.cc:654] T 6bae8d67e9894646aae3f6230eda07c7 P 30d88bcd65b347dabcbac54465e52b21: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:24.547036 24294 log.cc:824] T 6bae8d67e9894646aae3f6230eda07c7 P 30d88bcd65b347dabcbac54465e52b21: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:24.550058 24294 tablet_bootstrap.cc:492] T 6bae8d67e9894646aae3f6230eda07c7 P 30d88bcd65b347dabcbac54465e52b21: No bootstrap required, opened a new log
I20240503 08:11:24.550330 24294 ts_tablet_manager.cc:1367] T 6bae8d67e9894646aae3f6230eda07c7 P 30d88bcd65b347dabcbac54465e52b21: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.006s
I20240503 08:11:24.553577 24293 raft_consensus.cc:348] T 78b4f0eae3d14b97b8f74cbc24389044 P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } }
I20240503 08:11:24.554003 24293 raft_consensus.cc:374] T 78b4f0eae3d14b97b8f74cbc24389044 P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:24.554104 24293 raft_consensus.cc:729] T 78b4f0eae3d14b97b8f74cbc24389044 P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76e30a81c8b8451898078e9636a1104b, State: Initialized, Role: FOLLOWER
I20240503 08:11:24.554884 24293 consensus_queue.cc:260] T 78b4f0eae3d14b97b8f74cbc24389044 P 76e30a81c8b8451898078e9636a1104b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } }
I20240503 08:11:24.555130 24293 raft_consensus.cc:388] T 78b4f0eae3d14b97b8f74cbc24389044 P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:24.555270 24293 raft_consensus.cc:482] T 78b4f0eae3d14b97b8f74cbc24389044 P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:24.555436 24293 raft_consensus.cc:3037] T 78b4f0eae3d14b97b8f74cbc24389044 P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:24.558511 24293 raft_consensus.cc:504] T 78b4f0eae3d14b97b8f74cbc24389044 P 76e30a81c8b8451898078e9636a1104b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } }
I20240503 08:11:24.558737 24294 raft_consensus.cc:348] T 6bae8d67e9894646aae3f6230eda07c7 P 30d88bcd65b347dabcbac54465e52b21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } }
I20240503 08:11:24.559123 24294 raft_consensus.cc:374] T 6bae8d67e9894646aae3f6230eda07c7 P 30d88bcd65b347dabcbac54465e52b21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:24.559073 24293 leader_election.cc:302] T 78b4f0eae3d14b97b8f74cbc24389044 P 76e30a81c8b8451898078e9636a1104b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76e30a81c8b8451898078e9636a1104b; no voters:
I20240503 08:11:24.559222 24294 raft_consensus.cc:729] T 6bae8d67e9894646aae3f6230eda07c7 P 30d88bcd65b347dabcbac54465e52b21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30d88bcd65b347dabcbac54465e52b21, State: Initialized, Role: FOLLOWER
I20240503 08:11:24.559841 24293 leader_election.cc:288] T 78b4f0eae3d14b97b8f74cbc24389044 P 76e30a81c8b8451898078e9636a1104b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:24.559959 24297 raft_consensus.cc:2781] T 78b4f0eae3d14b97b8f74cbc24389044 P 76e30a81c8b8451898078e9636a1104b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:24.559917 24294 consensus_queue.cc:260] T 6bae8d67e9894646aae3f6230eda07c7 P 30d88bcd65b347dabcbac54465e52b21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } }
I20240503 08:11:24.560221 24294 raft_consensus.cc:388] T 6bae8d67e9894646aae3f6230eda07c7 P 30d88bcd65b347dabcbac54465e52b21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:24.560407 24294 raft_consensus.cc:482] T 6bae8d67e9894646aae3f6230eda07c7 P 30d88bcd65b347dabcbac54465e52b21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:24.560626 24294 raft_consensus.cc:3037] T 6bae8d67e9894646aae3f6230eda07c7 P 30d88bcd65b347dabcbac54465e52b21 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:24.563839 24294 raft_consensus.cc:504] T 6bae8d67e9894646aae3f6230eda07c7 P 30d88bcd65b347dabcbac54465e52b21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } }
I20240503 08:11:24.564456 24294 leader_election.cc:302] T 6bae8d67e9894646aae3f6230eda07c7 P 30d88bcd65b347dabcbac54465e52b21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30d88bcd65b347dabcbac54465e52b21; no voters:
I20240503 08:11:24.564868 24297 raft_consensus.cc:686] T 78b4f0eae3d14b97b8f74cbc24389044 P 76e30a81c8b8451898078e9636a1104b [term 1 LEADER]: Becoming Leader. State: Replica: 76e30a81c8b8451898078e9636a1104b, State: Running, Role: LEADER
I20240503 08:11:24.565171 24294 leader_election.cc:288] T 6bae8d67e9894646aae3f6230eda07c7 P 30d88bcd65b347dabcbac54465e52b21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:24.565266 24293 ts_tablet_manager.cc:1398] T 78b4f0eae3d14b97b8f74cbc24389044 P 76e30a81c8b8451898078e9636a1104b: Time spent starting tablet: real 0.020s user 0.013s sys 0.008s
I20240503 08:11:24.565287 24300 raft_consensus.cc:2781] T 6bae8d67e9894646aae3f6230eda07c7 P 30d88bcd65b347dabcbac54465e52b21 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:24.565490 24297 consensus_queue.cc:237] T 78b4f0eae3d14b97b8f74cbc24389044 P 76e30a81c8b8451898078e9636a1104b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } }
I20240503 08:11:24.565728 24293 tablet_bootstrap.cc:492] T 494f9374df1d45f5a0a8f7705e72557a P 76e30a81c8b8451898078e9636a1104b: Bootstrap starting.
I20240503 08:11:24.567546 24293 tablet_bootstrap.cc:654] T 494f9374df1d45f5a0a8f7705e72557a P 76e30a81c8b8451898078e9636a1104b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:24.572062 24300 raft_consensus.cc:686] T 6bae8d67e9894646aae3f6230eda07c7 P 30d88bcd65b347dabcbac54465e52b21 [term 1 LEADER]: Becoming Leader. State: Replica: 30d88bcd65b347dabcbac54465e52b21, State: Running, Role: LEADER
I20240503 08:11:24.572739 24300 consensus_queue.cc:237] T 6bae8d67e9894646aae3f6230eda07c7 P 30d88bcd65b347dabcbac54465e52b21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } }
I20240503 08:11:24.573856 24294 ts_tablet_manager.cc:1398] T 6bae8d67e9894646aae3f6230eda07c7 P 30d88bcd65b347dabcbac54465e52b21: Time spent starting tablet: real 0.023s user 0.014s sys 0.008s
I20240503 08:11:24.575408 24293 tablet_bootstrap.cc:492] T 494f9374df1d45f5a0a8f7705e72557a P 76e30a81c8b8451898078e9636a1104b: No bootstrap required, opened a new log
I20240503 08:11:24.575585 24293 ts_tablet_manager.cc:1367] T 494f9374df1d45f5a0a8f7705e72557a P 76e30a81c8b8451898078e9636a1104b: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20240503 08:11:24.576622 24293 raft_consensus.cc:348] T 494f9374df1d45f5a0a8f7705e72557a P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } }
I20240503 08:11:24.576858 24293 raft_consensus.cc:374] T 494f9374df1d45f5a0a8f7705e72557a P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:24.577004 24293 raft_consensus.cc:729] T 494f9374df1d45f5a0a8f7705e72557a P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76e30a81c8b8451898078e9636a1104b, State: Initialized, Role: FOLLOWER
I20240503 08:11:24.577358 24293 consensus_queue.cc:260] T 494f9374df1d45f5a0a8f7705e72557a P 76e30a81c8b8451898078e9636a1104b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } }
I20240503 08:11:24.577615 24293 raft_consensus.cc:388] T 494f9374df1d45f5a0a8f7705e72557a P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:24.577744 24293 raft_consensus.cc:482] T 494f9374df1d45f5a0a8f7705e72557a P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:24.578043 24293 raft_consensus.cc:3037] T 494f9374df1d45f5a0a8f7705e72557a P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:24.578413 23285 catalog_manager.cc:5484] T 6bae8d67e9894646aae3f6230eda07c7 P 30d88bcd65b347dabcbac54465e52b21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30d88bcd65b347dabcbac54465e52b21 (127.20.230.66). New cstate: current_term: 1 leader_uuid: "30d88bcd65b347dabcbac54465e52b21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } }
I20240503 08:11:24.578413 23286 catalog_manager.cc:5484] T 78b4f0eae3d14b97b8f74cbc24389044 P 76e30a81c8b8451898078e9636a1104b reported cstate change: term changed from 0 to 1, leader changed from <none> to 76e30a81c8b8451898078e9636a1104b (127.20.230.65). New cstate: current_term: 1 leader_uuid: "76e30a81c8b8451898078e9636a1104b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } }
I20240503 08:11:24.580837 24293 raft_consensus.cc:504] T 494f9374df1d45f5a0a8f7705e72557a P 76e30a81c8b8451898078e9636a1104b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } }
I20240503 08:11:24.581123 24293 leader_election.cc:302] T 494f9374df1d45f5a0a8f7705e72557a P 76e30a81c8b8451898078e9636a1104b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76e30a81c8b8451898078e9636a1104b; no voters:
I20240503 08:11:24.581444 24297 raft_consensus.cc:2781] T 494f9374df1d45f5a0a8f7705e72557a P 76e30a81c8b8451898078e9636a1104b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:24.581586 24297 raft_consensus.cc:686] T 494f9374df1d45f5a0a8f7705e72557a P 76e30a81c8b8451898078e9636a1104b [term 1 LEADER]: Becoming Leader. State: Replica: 76e30a81c8b8451898078e9636a1104b, State: Running, Role: LEADER
I20240503 08:11:24.581835 24297 consensus_queue.cc:237] T 494f9374df1d45f5a0a8f7705e72557a P 76e30a81c8b8451898078e9636a1104b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } }
I20240503 08:11:24.581941 24293 leader_election.cc:288] T 494f9374df1d45f5a0a8f7705e72557a P 76e30a81c8b8451898078e9636a1104b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:24.582340 24293 ts_tablet_manager.cc:1398] T 494f9374df1d45f5a0a8f7705e72557a P 76e30a81c8b8451898078e9636a1104b: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240503 08:11:24.585016 23285 catalog_manager.cc:5484] T 494f9374df1d45f5a0a8f7705e72557a P 76e30a81c8b8451898078e9636a1104b reported cstate change: term changed from 0 to 1, leader changed from <none> to 76e30a81c8b8451898078e9636a1104b (127.20.230.65). New cstate: current_term: 1 leader_uuid: "76e30a81c8b8451898078e9636a1104b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } }
W20240503 08:11:24.906423 24309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:24.906754 24309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:24.913980 24309 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 130.628 ms
time per row: 0.0653138 ms
I20240503 08:11:25.178623 23281 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58436:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:11:25.186448 23537 tablet_service.cc:1467] Processing CreateTablet for tablet 43fb21a411474f62afad731b47bad5a9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=be175f2340dc4b2a947a393d8a2bd2c3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:25.186919 23537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43fb21a411474f62afad731b47bad5a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:25.187008 23407 tablet_service.cc:1467] Processing CreateTablet for tablet 6bb8e518a6b6496fb52f41f0b6914ff4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=be175f2340dc4b2a947a393d8a2bd2c3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:25.187021 23536 tablet_service.cc:1467] Processing CreateTablet for tablet 2449909504504d1e84cf34f1af673af2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=be175f2340dc4b2a947a393d8a2bd2c3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:25.187397 23407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bb8e518a6b6496fb52f41f0b6914ff4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:25.187448 23536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2449909504504d1e84cf34f1af673af2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:25.192881 24351 tablet_bootstrap.cc:492] T 2449909504504d1e84cf34f1af673af2 P 30d88bcd65b347dabcbac54465e52b21: Bootstrap starting.
I20240503 08:11:25.193593 24352 tablet_bootstrap.cc:492] T 6bb8e518a6b6496fb52f41f0b6914ff4 P 76e30a81c8b8451898078e9636a1104b: Bootstrap starting.
I20240503 08:11:25.194738 24351 tablet_bootstrap.cc:654] T 2449909504504d1e84cf34f1af673af2 P 30d88bcd65b347dabcbac54465e52b21: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:25.195307 24352 tablet_bootstrap.cc:654] T 6bb8e518a6b6496fb52f41f0b6914ff4 P 76e30a81c8b8451898078e9636a1104b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:25.196635 24351 tablet_bootstrap.cc:492] T 2449909504504d1e84cf34f1af673af2 P 30d88bcd65b347dabcbac54465e52b21: No bootstrap required, opened a new log
I20240503 08:11:25.196760 24351 ts_tablet_manager.cc:1367] T 2449909504504d1e84cf34f1af673af2 P 30d88bcd65b347dabcbac54465e52b21: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 08:11:25.197000 24352 tablet_bootstrap.cc:492] T 6bb8e518a6b6496fb52f41f0b6914ff4 P 76e30a81c8b8451898078e9636a1104b: No bootstrap required, opened a new log
I20240503 08:11:25.197113 24352 ts_tablet_manager.cc:1367] T 6bb8e518a6b6496fb52f41f0b6914ff4 P 76e30a81c8b8451898078e9636a1104b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:11:25.197499 24351 raft_consensus.cc:348] T 2449909504504d1e84cf34f1af673af2 P 30d88bcd65b347dabcbac54465e52b21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } }
I20240503 08:11:25.197641 24351 raft_consensus.cc:374] T 2449909504504d1e84cf34f1af673af2 P 30d88bcd65b347dabcbac54465e52b21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:25.197640 24352 raft_consensus.cc:348] T 6bb8e518a6b6496fb52f41f0b6914ff4 P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } }
I20240503 08:11:25.197726 24351 raft_consensus.cc:729] T 2449909504504d1e84cf34f1af673af2 P 30d88bcd65b347dabcbac54465e52b21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30d88bcd65b347dabcbac54465e52b21, State: Initialized, Role: FOLLOWER
I20240503 08:11:25.197813 24352 raft_consensus.cc:374] T 6bb8e518a6b6496fb52f41f0b6914ff4 P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:25.197903 24352 raft_consensus.cc:729] T 6bb8e518a6b6496fb52f41f0b6914ff4 P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76e30a81c8b8451898078e9636a1104b, State: Initialized, Role: FOLLOWER
I20240503 08:11:25.197973 24351 consensus_queue.cc:260] T 2449909504504d1e84cf34f1af673af2 P 30d88bcd65b347dabcbac54465e52b21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } }
I20240503 08:11:25.198148 24351 raft_consensus.cc:388] T 2449909504504d1e84cf34f1af673af2 P 30d88bcd65b347dabcbac54465e52b21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:25.198144 24352 consensus_queue.cc:260] T 6bb8e518a6b6496fb52f41f0b6914ff4 P 76e30a81c8b8451898078e9636a1104b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } }
I20240503 08:11:25.198223 24351 raft_consensus.cc:482] T 2449909504504d1e84cf34f1af673af2 P 30d88bcd65b347dabcbac54465e52b21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:25.198330 24351 raft_consensus.cc:3037] T 2449909504504d1e84cf34f1af673af2 P 30d88bcd65b347dabcbac54465e52b21 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:25.198328 24352 raft_consensus.cc:388] T 6bb8e518a6b6496fb52f41f0b6914ff4 P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:25.198392 24352 raft_consensus.cc:482] T 6bb8e518a6b6496fb52f41f0b6914ff4 P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:25.198498 24352 raft_consensus.cc:3037] T 6bb8e518a6b6496fb52f41f0b6914ff4 P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:25.200347 24351 raft_consensus.cc:504] T 2449909504504d1e84cf34f1af673af2 P 30d88bcd65b347dabcbac54465e52b21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } }
I20240503 08:11:25.200410 24352 raft_consensus.cc:504] T 6bb8e518a6b6496fb52f41f0b6914ff4 P 76e30a81c8b8451898078e9636a1104b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } }
I20240503 08:11:25.200556 24351 leader_election.cc:302] T 2449909504504d1e84cf34f1af673af2 P 30d88bcd65b347dabcbac54465e52b21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30d88bcd65b347dabcbac54465e52b21; no voters:
I20240503 08:11:25.200613 24352 leader_election.cc:302] T 6bb8e518a6b6496fb52f41f0b6914ff4 P 76e30a81c8b8451898078e9636a1104b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76e30a81c8b8451898078e9636a1104b; no voters:
I20240503 08:11:25.200721 24351 leader_election.cc:288] T 2449909504504d1e84cf34f1af673af2 P 30d88bcd65b347dabcbac54465e52b21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:25.200801 24352 leader_election.cc:288] T 6bb8e518a6b6496fb52f41f0b6914ff4 P 76e30a81c8b8451898078e9636a1104b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:25.200788 24300 raft_consensus.cc:2781] T 2449909504504d1e84cf34f1af673af2 P 30d88bcd65b347dabcbac54465e52b21 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:25.200883 24299 raft_consensus.cc:2781] T 6bb8e518a6b6496fb52f41f0b6914ff4 P 76e30a81c8b8451898078e9636a1104b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:25.201390 24352 ts_tablet_manager.cc:1398] T 6bb8e518a6b6496fb52f41f0b6914ff4 P 76e30a81c8b8451898078e9636a1104b: Time spent starting tablet: real 0.004s user 0.001s sys 0.004s
I20240503 08:11:25.201813 24300 raft_consensus.cc:686] T 2449909504504d1e84cf34f1af673af2 P 30d88bcd65b347dabcbac54465e52b21 [term 1 LEADER]: Becoming Leader. State: Replica: 30d88bcd65b347dabcbac54465e52b21, State: Running, Role: LEADER
I20240503 08:11:25.202054 24300 consensus_queue.cc:237] T 2449909504504d1e84cf34f1af673af2 P 30d88bcd65b347dabcbac54465e52b21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } }
I20240503 08:11:25.202433 24299 raft_consensus.cc:686] T 6bb8e518a6b6496fb52f41f0b6914ff4 P 76e30a81c8b8451898078e9636a1104b [term 1 LEADER]: Becoming Leader. State: Replica: 76e30a81c8b8451898078e9636a1104b, State: Running, Role: LEADER
I20240503 08:11:25.202749 24299 consensus_queue.cc:237] T 6bb8e518a6b6496fb52f41f0b6914ff4 P 76e30a81c8b8451898078e9636a1104b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } }
I20240503 08:11:25.204989 23281 catalog_manager.cc:5484] T 6bb8e518a6b6496fb52f41f0b6914ff4 P 76e30a81c8b8451898078e9636a1104b reported cstate change: term changed from 0 to 1, leader changed from <none> to 76e30a81c8b8451898078e9636a1104b (127.20.230.65). New cstate: current_term: 1 leader_uuid: "76e30a81c8b8451898078e9636a1104b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } }
I20240503 08:11:25.206357 24351 ts_tablet_manager.cc:1398] T 2449909504504d1e84cf34f1af673af2 P 30d88bcd65b347dabcbac54465e52b21: Time spent starting tablet: real 0.009s user 0.006s sys 0.003s
I20240503 08:11:25.206620 24351 tablet_bootstrap.cc:492] T 43fb21a411474f62afad731b47bad5a9 P 30d88bcd65b347dabcbac54465e52b21: Bootstrap starting.
I20240503 08:11:25.207880 23282 catalog_manager.cc:5484] T 2449909504504d1e84cf34f1af673af2 P 30d88bcd65b347dabcbac54465e52b21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30d88bcd65b347dabcbac54465e52b21 (127.20.230.66). New cstate: current_term: 1 leader_uuid: "30d88bcd65b347dabcbac54465e52b21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } }
I20240503 08:11:25.208545 24351 tablet_bootstrap.cc:654] T 43fb21a411474f62afad731b47bad5a9 P 30d88bcd65b347dabcbac54465e52b21: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:25.210695 24351 tablet_bootstrap.cc:492] T 43fb21a411474f62afad731b47bad5a9 P 30d88bcd65b347dabcbac54465e52b21: No bootstrap required, opened a new log
I20240503 08:11:25.210891 24351 ts_tablet_manager.cc:1367] T 43fb21a411474f62afad731b47bad5a9 P 30d88bcd65b347dabcbac54465e52b21: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s
I20240503 08:11:25.211556 24351 raft_consensus.cc:348] T 43fb21a411474f62afad731b47bad5a9 P 30d88bcd65b347dabcbac54465e52b21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } }
I20240503 08:11:25.211710 24351 raft_consensus.cc:374] T 43fb21a411474f62afad731b47bad5a9 P 30d88bcd65b347dabcbac54465e52b21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:25.211781 24351 raft_consensus.cc:729] T 43fb21a411474f62afad731b47bad5a9 P 30d88bcd65b347dabcbac54465e52b21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30d88bcd65b347dabcbac54465e52b21, State: Initialized, Role: FOLLOWER
I20240503 08:11:25.212064 24351 consensus_queue.cc:260] T 43fb21a411474f62afad731b47bad5a9 P 30d88bcd65b347dabcbac54465e52b21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } }
I20240503 08:11:25.212250 24351 raft_consensus.cc:388] T 43fb21a411474f62afad731b47bad5a9 P 30d88bcd65b347dabcbac54465e52b21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:25.212344 24351 raft_consensus.cc:482] T 43fb21a411474f62afad731b47bad5a9 P 30d88bcd65b347dabcbac54465e52b21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:25.212435 24351 raft_consensus.cc:3037] T 43fb21a411474f62afad731b47bad5a9 P 30d88bcd65b347dabcbac54465e52b21 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:25.214591 24351 raft_consensus.cc:504] T 43fb21a411474f62afad731b47bad5a9 P 30d88bcd65b347dabcbac54465e52b21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } }
I20240503 08:11:25.214781 24351 leader_election.cc:302] T 43fb21a411474f62afad731b47bad5a9 P 30d88bcd65b347dabcbac54465e52b21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30d88bcd65b347dabcbac54465e52b21; no voters:
I20240503 08:11:25.214947 24351 leader_election.cc:288] T 43fb21a411474f62afad731b47bad5a9 P 30d88bcd65b347dabcbac54465e52b21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:25.215013 24356 raft_consensus.cc:2781] T 43fb21a411474f62afad731b47bad5a9 P 30d88bcd65b347dabcbac54465e52b21 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:25.215168 24356 raft_consensus.cc:686] T 43fb21a411474f62afad731b47bad5a9 P 30d88bcd65b347dabcbac54465e52b21 [term 1 LEADER]: Becoming Leader. State: Replica: 30d88bcd65b347dabcbac54465e52b21, State: Running, Role: LEADER
I20240503 08:11:25.215334 24351 ts_tablet_manager.cc:1398] T 43fb21a411474f62afad731b47bad5a9 P 30d88bcd65b347dabcbac54465e52b21: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240503 08:11:25.215420 24356 consensus_queue.cc:237] T 43fb21a411474f62afad731b47bad5a9 P 30d88bcd65b347dabcbac54465e52b21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } }
I20240503 08:11:25.217197 23281 catalog_manager.cc:5484] T 43fb21a411474f62afad731b47bad5a9 P 30d88bcd65b347dabcbac54465e52b21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30d88bcd65b347dabcbac54465e52b21 (127.20.230.66). New cstate: current_term: 1 leader_uuid: "30d88bcd65b347dabcbac54465e52b21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } }
W20240503 08:11:25.538359 24362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:25.538734 24362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:25.546234 24362 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 102.957 ms
time per row: 0.0514785 ms
-----------------------------------------------------
Suppressions used:
count bytes template
14 3624 libcrypto.so.1.1
-----------------------------------------------------
I20240503 08:11:26.636742 23281 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58436:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:11:26.645085 23407 tablet_service.cc:1467] Processing CreateTablet for tablet 0d69b1adfa064beb82bdcc16d53717d7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9e0052fa5a20450fad9ea538bf678fcd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:26.645431 23536 tablet_service.cc:1467] Processing CreateTablet for tablet b2779eae1c08452ca7b5b0874bbac47e (DEFAULT_TABLE table=rebalance_test_table_2 [id=9e0052fa5a20450fad9ea538bf678fcd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:26.645524 23407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d69b1adfa064beb82bdcc16d53717d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:26.645502 23406 tablet_service.cc:1467] Processing CreateTablet for tablet b19203edb44242acb3ad7dfbea6459a2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9e0052fa5a20450fad9ea538bf678fcd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:26.645874 23536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2779eae1c08452ca7b5b0874bbac47e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:26.645877 23406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b19203edb44242acb3ad7dfbea6459a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:26.650079 24405 tablet_bootstrap.cc:492] T 0d69b1adfa064beb82bdcc16d53717d7 P 76e30a81c8b8451898078e9636a1104b: Bootstrap starting.
I20240503 08:11:26.651299 24405 tablet_bootstrap.cc:654] T 0d69b1adfa064beb82bdcc16d53717d7 P 76e30a81c8b8451898078e9636a1104b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:26.653113 24406 tablet_bootstrap.cc:492] T b2779eae1c08452ca7b5b0874bbac47e P 30d88bcd65b347dabcbac54465e52b21: Bootstrap starting.
I20240503 08:11:26.653314 24405 tablet_bootstrap.cc:492] T 0d69b1adfa064beb82bdcc16d53717d7 P 76e30a81c8b8451898078e9636a1104b: No bootstrap required, opened a new log
I20240503 08:11:26.653528 24405 ts_tablet_manager.cc:1367] T 0d69b1adfa064beb82bdcc16d53717d7 P 76e30a81c8b8451898078e9636a1104b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 08:11:26.654150 24405 raft_consensus.cc:348] T 0d69b1adfa064beb82bdcc16d53717d7 P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } }
I20240503 08:11:26.654285 24405 raft_consensus.cc:374] T 0d69b1adfa064beb82bdcc16d53717d7 P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:26.654367 24405 raft_consensus.cc:729] T 0d69b1adfa064beb82bdcc16d53717d7 P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76e30a81c8b8451898078e9636a1104b, State: Initialized, Role: FOLLOWER
I20240503 08:11:26.654569 24405 consensus_queue.cc:260] T 0d69b1adfa064beb82bdcc16d53717d7 P 76e30a81c8b8451898078e9636a1104b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } }
I20240503 08:11:26.654634 24406 tablet_bootstrap.cc:654] T b2779eae1c08452ca7b5b0874bbac47e P 30d88bcd65b347dabcbac54465e52b21: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:26.654704 24405 raft_consensus.cc:388] T 0d69b1adfa064beb82bdcc16d53717d7 P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:26.654776 24405 raft_consensus.cc:482] T 0d69b1adfa064beb82bdcc16d53717d7 P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:26.654970 24405 raft_consensus.cc:3037] T 0d69b1adfa064beb82bdcc16d53717d7 P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:26.656188 24406 tablet_bootstrap.cc:492] T b2779eae1c08452ca7b5b0874bbac47e P 30d88bcd65b347dabcbac54465e52b21: No bootstrap required, opened a new log
I20240503 08:11:26.656319 24406 ts_tablet_manager.cc:1367] T b2779eae1c08452ca7b5b0874bbac47e P 30d88bcd65b347dabcbac54465e52b21: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:11:26.656832 24406 raft_consensus.cc:348] T b2779eae1c08452ca7b5b0874bbac47e P 30d88bcd65b347dabcbac54465e52b21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } }
I20240503 08:11:26.656867 24405 raft_consensus.cc:504] T 0d69b1adfa064beb82bdcc16d53717d7 P 76e30a81c8b8451898078e9636a1104b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } }
I20240503 08:11:26.656970 24406 raft_consensus.cc:374] T b2779eae1c08452ca7b5b0874bbac47e P 30d88bcd65b347dabcbac54465e52b21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:26.657058 24406 raft_consensus.cc:729] T b2779eae1c08452ca7b5b0874bbac47e P 30d88bcd65b347dabcbac54465e52b21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30d88bcd65b347dabcbac54465e52b21, State: Initialized, Role: FOLLOWER
I20240503 08:11:26.657053 24405 leader_election.cc:302] T 0d69b1adfa064beb82bdcc16d53717d7 P 76e30a81c8b8451898078e9636a1104b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76e30a81c8b8451898078e9636a1104b; no voters:
I20240503 08:11:26.657254 24406 consensus_queue.cc:260] T b2779eae1c08452ca7b5b0874bbac47e P 30d88bcd65b347dabcbac54465e52b21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } }
I20240503 08:11:26.657402 24406 raft_consensus.cc:388] T b2779eae1c08452ca7b5b0874bbac47e P 30d88bcd65b347dabcbac54465e52b21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:26.657442 24405 leader_election.cc:288] T 0d69b1adfa064beb82bdcc16d53717d7 P 76e30a81c8b8451898078e9636a1104b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:26.657474 24406 raft_consensus.cc:482] T b2779eae1c08452ca7b5b0874bbac47e P 30d88bcd65b347dabcbac54465e52b21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:26.657604 24406 raft_consensus.cc:3037] T b2779eae1c08452ca7b5b0874bbac47e P 30d88bcd65b347dabcbac54465e52b21 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:26.657589 24409 raft_consensus.cc:2781] T 0d69b1adfa064beb82bdcc16d53717d7 P 76e30a81c8b8451898078e9636a1104b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:26.658188 24405 ts_tablet_manager.cc:1398] T 0d69b1adfa064beb82bdcc16d53717d7 P 76e30a81c8b8451898078e9636a1104b: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20240503 08:11:26.658488 24405 tablet_bootstrap.cc:492] T b19203edb44242acb3ad7dfbea6459a2 P 76e30a81c8b8451898078e9636a1104b: Bootstrap starting.
I20240503 08:11:26.659523 24406 raft_consensus.cc:504] T b2779eae1c08452ca7b5b0874bbac47e P 30d88bcd65b347dabcbac54465e52b21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } }
I20240503 08:11:26.659726 24406 leader_election.cc:302] T b2779eae1c08452ca7b5b0874bbac47e P 30d88bcd65b347dabcbac54465e52b21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30d88bcd65b347dabcbac54465e52b21; no voters:
I20240503 08:11:26.660007 24405 tablet_bootstrap.cc:654] T b19203edb44242acb3ad7dfbea6459a2 P 76e30a81c8b8451898078e9636a1104b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:26.660091 24406 leader_election.cc:288] T b2779eae1c08452ca7b5b0874bbac47e P 30d88bcd65b347dabcbac54465e52b21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:26.660233 24412 raft_consensus.cc:2781] T b2779eae1c08452ca7b5b0874bbac47e P 30d88bcd65b347dabcbac54465e52b21 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:26.664430 24409 raft_consensus.cc:686] T 0d69b1adfa064beb82bdcc16d53717d7 P 76e30a81c8b8451898078e9636a1104b [term 1 LEADER]: Becoming Leader. State: Replica: 76e30a81c8b8451898078e9636a1104b, State: Running, Role: LEADER
I20240503 08:11:26.664791 24409 consensus_queue.cc:237] T 0d69b1adfa064beb82bdcc16d53717d7 P 76e30a81c8b8451898078e9636a1104b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } }
I20240503 08:11:26.665228 24405 tablet_bootstrap.cc:492] T b19203edb44242acb3ad7dfbea6459a2 P 76e30a81c8b8451898078e9636a1104b: No bootstrap required, opened a new log
I20240503 08:11:26.665433 24405 ts_tablet_manager.cc:1367] T b19203edb44242acb3ad7dfbea6459a2 P 76e30a81c8b8451898078e9636a1104b: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20240503 08:11:26.666250 24405 raft_consensus.cc:348] T b19203edb44242acb3ad7dfbea6459a2 P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } }
I20240503 08:11:26.666491 24405 raft_consensus.cc:374] T b19203edb44242acb3ad7dfbea6459a2 P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:26.666613 24405 raft_consensus.cc:729] T b19203edb44242acb3ad7dfbea6459a2 P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76e30a81c8b8451898078e9636a1104b, State: Initialized, Role: FOLLOWER
I20240503 08:11:26.667099 24412 raft_consensus.cc:686] T b2779eae1c08452ca7b5b0874bbac47e P 30d88bcd65b347dabcbac54465e52b21 [term 1 LEADER]: Becoming Leader. State: Replica: 30d88bcd65b347dabcbac54465e52b21, State: Running, Role: LEADER
I20240503 08:11:26.666915 24405 consensus_queue.cc:260] T b19203edb44242acb3ad7dfbea6459a2 P 76e30a81c8b8451898078e9636a1104b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } }
I20240503 08:11:26.667428 24405 raft_consensus.cc:388] T b19203edb44242acb3ad7dfbea6459a2 P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:26.667578 24405 raft_consensus.cc:482] T b19203edb44242acb3ad7dfbea6459a2 P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:26.667564 24412 consensus_queue.cc:237] T b2779eae1c08452ca7b5b0874bbac47e P 30d88bcd65b347dabcbac54465e52b21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } }
I20240503 08:11:26.667721 24405 raft_consensus.cc:3037] T b19203edb44242acb3ad7dfbea6459a2 P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:26.668882 24406 ts_tablet_manager.cc:1398] T b2779eae1c08452ca7b5b0874bbac47e P 30d88bcd65b347dabcbac54465e52b21: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20240503 08:11:26.670877 24405 raft_consensus.cc:504] T b19203edb44242acb3ad7dfbea6459a2 P 76e30a81c8b8451898078e9636a1104b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } }
I20240503 08:11:26.670850 23281 catalog_manager.cc:5484] T b2779eae1c08452ca7b5b0874bbac47e P 30d88bcd65b347dabcbac54465e52b21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30d88bcd65b347dabcbac54465e52b21 (127.20.230.66). New cstate: current_term: 1 leader_uuid: "30d88bcd65b347dabcbac54465e52b21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } }
I20240503 08:11:26.671051 24405 leader_election.cc:302] T b19203edb44242acb3ad7dfbea6459a2 P 76e30a81c8b8451898078e9636a1104b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76e30a81c8b8451898078e9636a1104b; no voters:
I20240503 08:11:26.671226 24405 leader_election.cc:288] T b19203edb44242acb3ad7dfbea6459a2 P 76e30a81c8b8451898078e9636a1104b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:26.671324 24411 raft_consensus.cc:2781] T b19203edb44242acb3ad7dfbea6459a2 P 76e30a81c8b8451898078e9636a1104b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:26.671471 24411 raft_consensus.cc:686] T b19203edb44242acb3ad7dfbea6459a2 P 76e30a81c8b8451898078e9636a1104b [term 1 LEADER]: Becoming Leader. State: Replica: 76e30a81c8b8451898078e9636a1104b, State: Running, Role: LEADER
I20240503 08:11:26.671617 24405 ts_tablet_manager.cc:1398] T b19203edb44242acb3ad7dfbea6459a2 P 76e30a81c8b8451898078e9636a1104b: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240503 08:11:26.671728 24411 consensus_queue.cc:237] T b19203edb44242acb3ad7dfbea6459a2 P 76e30a81c8b8451898078e9636a1104b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } }
I20240503 08:11:26.674499 23285 catalog_manager.cc:5484] T 0d69b1adfa064beb82bdcc16d53717d7 P 76e30a81c8b8451898078e9636a1104b reported cstate change: term changed from 0 to 1, leader changed from <none> to 76e30a81c8b8451898078e9636a1104b (127.20.230.65). New cstate: current_term: 1 leader_uuid: "76e30a81c8b8451898078e9636a1104b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } }
I20240503 08:11:26.678254 23281 catalog_manager.cc:5484] T b19203edb44242acb3ad7dfbea6459a2 P 76e30a81c8b8451898078e9636a1104b reported cstate change: term changed from 0 to 1, leader changed from <none> to 76e30a81c8b8451898078e9636a1104b (127.20.230.65). New cstate: current_term: 1 leader_uuid: "76e30a81c8b8451898078e9636a1104b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } }
W20240503 08:11:26.980278 24421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:26.980597 24421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:26.987733 24421 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 121.607 ms
time per row: 0.0608035 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 08:11:28.087915 23285 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58436:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:11:28.095957 23536 tablet_service.cc:1467] Processing CreateTablet for tablet 7878470e8ac949bb8971fe8367f7db87 (DEFAULT_TABLE table=rebalance_test_table_3 [id=798c4a054bb5441485fcde0049107bdc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:28.096195 23537 tablet_service.cc:1467] Processing CreateTablet for tablet 8c6217accd494f5182a862c31013b76e (DEFAULT_TABLE table=rebalance_test_table_3 [id=798c4a054bb5441485fcde0049107bdc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:28.096428 23536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7878470e8ac949bb8971fe8367f7db87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:28.096496 23406 tablet_service.cc:1467] Processing CreateTablet for tablet 29dea2294fd74738a55ac1bd5b88121b (DEFAULT_TABLE table=rebalance_test_table_3 [id=798c4a054bb5441485fcde0049107bdc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:28.096910 23406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29dea2294fd74738a55ac1bd5b88121b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:28.097098 23537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c6217accd494f5182a862c31013b76e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:28.101621 24465 tablet_bootstrap.cc:492] T 8c6217accd494f5182a862c31013b76e P 30d88bcd65b347dabcbac54465e52b21: Bootstrap starting.
I20240503 08:11:28.103168 24465 tablet_bootstrap.cc:654] T 8c6217accd494f5182a862c31013b76e P 30d88bcd65b347dabcbac54465e52b21: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:28.103638 24466 tablet_bootstrap.cc:492] T 29dea2294fd74738a55ac1bd5b88121b P 76e30a81c8b8451898078e9636a1104b: Bootstrap starting.
I20240503 08:11:28.104781 24465 tablet_bootstrap.cc:492] T 8c6217accd494f5182a862c31013b76e P 30d88bcd65b347dabcbac54465e52b21: No bootstrap required, opened a new log
I20240503 08:11:28.104913 24465 ts_tablet_manager.cc:1367] T 8c6217accd494f5182a862c31013b76e P 30d88bcd65b347dabcbac54465e52b21: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:11:28.104995 24466 tablet_bootstrap.cc:654] T 29dea2294fd74738a55ac1bd5b88121b P 76e30a81c8b8451898078e9636a1104b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:28.105557 24465 raft_consensus.cc:348] T 8c6217accd494f5182a862c31013b76e P 30d88bcd65b347dabcbac54465e52b21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } }
I20240503 08:11:28.105690 24465 raft_consensus.cc:374] T 8c6217accd494f5182a862c31013b76e P 30d88bcd65b347dabcbac54465e52b21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:28.105830 24465 raft_consensus.cc:729] T 8c6217accd494f5182a862c31013b76e P 30d88bcd65b347dabcbac54465e52b21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30d88bcd65b347dabcbac54465e52b21, State: Initialized, Role: FOLLOWER
I20240503 08:11:28.106040 24465 consensus_queue.cc:260] T 8c6217accd494f5182a862c31013b76e P 30d88bcd65b347dabcbac54465e52b21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } }
I20240503 08:11:28.106182 24465 raft_consensus.cc:388] T 8c6217accd494f5182a862c31013b76e P 30d88bcd65b347dabcbac54465e52b21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:28.106252 24465 raft_consensus.cc:482] T 8c6217accd494f5182a862c31013b76e P 30d88bcd65b347dabcbac54465e52b21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:28.106359 24465 raft_consensus.cc:3037] T 8c6217accd494f5182a862c31013b76e P 30d88bcd65b347dabcbac54465e52b21 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:28.106621 24466 tablet_bootstrap.cc:492] T 29dea2294fd74738a55ac1bd5b88121b P 76e30a81c8b8451898078e9636a1104b: No bootstrap required, opened a new log
I20240503 08:11:28.106732 24466 ts_tablet_manager.cc:1367] T 29dea2294fd74738a55ac1bd5b88121b P 76e30a81c8b8451898078e9636a1104b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:11:28.107264 24466 raft_consensus.cc:348] T 29dea2294fd74738a55ac1bd5b88121b P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } }
I20240503 08:11:28.107388 24466 raft_consensus.cc:374] T 29dea2294fd74738a55ac1bd5b88121b P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:28.107465 24466 raft_consensus.cc:729] T 29dea2294fd74738a55ac1bd5b88121b P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76e30a81c8b8451898078e9636a1104b, State: Initialized, Role: FOLLOWER
I20240503 08:11:28.107645 24466 consensus_queue.cc:260] T 29dea2294fd74738a55ac1bd5b88121b P 76e30a81c8b8451898078e9636a1104b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } }
I20240503 08:11:28.107779 24466 raft_consensus.cc:388] T 29dea2294fd74738a55ac1bd5b88121b P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:28.107847 24466 raft_consensus.cc:482] T 29dea2294fd74738a55ac1bd5b88121b P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:28.107955 24466 raft_consensus.cc:3037] T 29dea2294fd74738a55ac1bd5b88121b P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:28.108251 24465 raft_consensus.cc:504] T 8c6217accd494f5182a862c31013b76e P 30d88bcd65b347dabcbac54465e52b21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } }
I20240503 08:11:28.108471 24465 leader_election.cc:302] T 8c6217accd494f5182a862c31013b76e P 30d88bcd65b347dabcbac54465e52b21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30d88bcd65b347dabcbac54465e52b21; no voters:
I20240503 08:11:28.108899 24465 leader_election.cc:288] T 8c6217accd494f5182a862c31013b76e P 30d88bcd65b347dabcbac54465e52b21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:28.109138 24469 raft_consensus.cc:2781] T 8c6217accd494f5182a862c31013b76e P 30d88bcd65b347dabcbac54465e52b21 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:28.109706 24469 raft_consensus.cc:686] T 8c6217accd494f5182a862c31013b76e P 30d88bcd65b347dabcbac54465e52b21 [term 1 LEADER]: Becoming Leader. State: Replica: 30d88bcd65b347dabcbac54465e52b21, State: Running, Role: LEADER
I20240503 08:11:28.110040 24469 consensus_queue.cc:237] T 8c6217accd494f5182a862c31013b76e P 30d88bcd65b347dabcbac54465e52b21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } }
I20240503 08:11:28.110316 24466 raft_consensus.cc:504] T 29dea2294fd74738a55ac1bd5b88121b P 76e30a81c8b8451898078e9636a1104b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } }
I20240503 08:11:28.110548 24466 leader_election.cc:302] T 29dea2294fd74738a55ac1bd5b88121b P 76e30a81c8b8451898078e9636a1104b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76e30a81c8b8451898078e9636a1104b; no voters:
I20240503 08:11:28.113327 23285 catalog_manager.cc:5484] T 8c6217accd494f5182a862c31013b76e P 30d88bcd65b347dabcbac54465e52b21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30d88bcd65b347dabcbac54465e52b21 (127.20.230.66). New cstate: current_term: 1 leader_uuid: "30d88bcd65b347dabcbac54465e52b21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } }
I20240503 08:11:28.114099 24466 leader_election.cc:288] T 29dea2294fd74738a55ac1bd5b88121b P 76e30a81c8b8451898078e9636a1104b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:28.114243 24473 raft_consensus.cc:2781] T 29dea2294fd74738a55ac1bd5b88121b P 76e30a81c8b8451898078e9636a1104b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:28.115748 24465 ts_tablet_manager.cc:1398] T 8c6217accd494f5182a862c31013b76e P 30d88bcd65b347dabcbac54465e52b21: Time spent starting tablet: real 0.011s user 0.004s sys 0.006s
I20240503 08:11:28.115993 24465 tablet_bootstrap.cc:492] T 7878470e8ac949bb8971fe8367f7db87 P 30d88bcd65b347dabcbac54465e52b21: Bootstrap starting.
I20240503 08:11:28.117427 24465 tablet_bootstrap.cc:654] T 7878470e8ac949bb8971fe8367f7db87 P 30d88bcd65b347dabcbac54465e52b21: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:28.119048 24465 tablet_bootstrap.cc:492] T 7878470e8ac949bb8971fe8367f7db87 P 30d88bcd65b347dabcbac54465e52b21: No bootstrap required, opened a new log
I20240503 08:11:28.119184 24465 ts_tablet_manager.cc:1367] T 7878470e8ac949bb8971fe8367f7db87 P 30d88bcd65b347dabcbac54465e52b21: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:11:28.119720 24465 raft_consensus.cc:348] T 7878470e8ac949bb8971fe8367f7db87 P 30d88bcd65b347dabcbac54465e52b21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } }
I20240503 08:11:28.119850 24466 ts_tablet_manager.cc:1398] T 29dea2294fd74738a55ac1bd5b88121b P 76e30a81c8b8451898078e9636a1104b: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20240503 08:11:28.119868 24465 raft_consensus.cc:374] T 7878470e8ac949bb8971fe8367f7db87 P 30d88bcd65b347dabcbac54465e52b21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:28.119947 24465 raft_consensus.cc:729] T 7878470e8ac949bb8971fe8367f7db87 P 30d88bcd65b347dabcbac54465e52b21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30d88bcd65b347dabcbac54465e52b21, State: Initialized, Role: FOLLOWER
I20240503 08:11:28.120056 24473 raft_consensus.cc:686] T 29dea2294fd74738a55ac1bd5b88121b P 76e30a81c8b8451898078e9636a1104b [term 1 LEADER]: Becoming Leader. State: Replica: 76e30a81c8b8451898078e9636a1104b, State: Running, Role: LEADER
I20240503 08:11:28.120167 24465 consensus_queue.cc:260] T 7878470e8ac949bb8971fe8367f7db87 P 30d88bcd65b347dabcbac54465e52b21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } }
I20240503 08:11:28.120325 24465 raft_consensus.cc:388] T 7878470e8ac949bb8971fe8367f7db87 P 30d88bcd65b347dabcbac54465e52b21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:28.120339 24473 consensus_queue.cc:237] T 29dea2294fd74738a55ac1bd5b88121b P 76e30a81c8b8451898078e9636a1104b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } }
I20240503 08:11:28.120430 24465 raft_consensus.cc:482] T 7878470e8ac949bb8971fe8367f7db87 P 30d88bcd65b347dabcbac54465e52b21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:28.120517 24465 raft_consensus.cc:3037] T 7878470e8ac949bb8971fe8367f7db87 P 30d88bcd65b347dabcbac54465e52b21 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:28.122361 23281 catalog_manager.cc:5484] T 29dea2294fd74738a55ac1bd5b88121b P 76e30a81c8b8451898078e9636a1104b reported cstate change: term changed from 0 to 1, leader changed from <none> to 76e30a81c8b8451898078e9636a1104b (127.20.230.65). New cstate: current_term: 1 leader_uuid: "76e30a81c8b8451898078e9636a1104b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } }
I20240503 08:11:28.122741 24465 raft_consensus.cc:504] T 7878470e8ac949bb8971fe8367f7db87 P 30d88bcd65b347dabcbac54465e52b21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } }
I20240503 08:11:28.122932 24465 leader_election.cc:302] T 7878470e8ac949bb8971fe8367f7db87 P 30d88bcd65b347dabcbac54465e52b21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30d88bcd65b347dabcbac54465e52b21; no voters:
I20240503 08:11:28.123107 24465 leader_election.cc:288] T 7878470e8ac949bb8971fe8367f7db87 P 30d88bcd65b347dabcbac54465e52b21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:28.123234 24469 raft_consensus.cc:2781] T 7878470e8ac949bb8971fe8367f7db87 P 30d88bcd65b347dabcbac54465e52b21 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:28.123384 24469 raft_consensus.cc:686] T 7878470e8ac949bb8971fe8367f7db87 P 30d88bcd65b347dabcbac54465e52b21 [term 1 LEADER]: Becoming Leader. State: Replica: 30d88bcd65b347dabcbac54465e52b21, State: Running, Role: LEADER
I20240503 08:11:28.123596 24469 consensus_queue.cc:237] T 7878470e8ac949bb8971fe8367f7db87 P 30d88bcd65b347dabcbac54465e52b21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } }
I20240503 08:11:28.125047 24465 ts_tablet_manager.cc:1398] T 7878470e8ac949bb8971fe8367f7db87 P 30d88bcd65b347dabcbac54465e52b21: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20240503 08:11:28.127595 23281 catalog_manager.cc:5484] T 7878470e8ac949bb8971fe8367f7db87 P 30d88bcd65b347dabcbac54465e52b21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30d88bcd65b347dabcbac54465e52b21 (127.20.230.66). New cstate: current_term: 1 leader_uuid: "30d88bcd65b347dabcbac54465e52b21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } }
W20240503 08:11:28.434933 24481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:28.435246 24481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:28.442503 24481 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 110.778 ms
time per row: 0.055389 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 08:11:29.533741 23282 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58436:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:11:29.541895 23406 tablet_service.cc:1467] Processing CreateTablet for tablet 85b933bc55b1492fb70055a765a2b09b (DEFAULT_TABLE table=rebalance_test_table_4 [id=a233217db4ea40e88dd7f9aa60cca533]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:29.542091 23537 tablet_service.cc:1467] Processing CreateTablet for tablet 2797c89387af4da5aba9d7d6657eb275 (DEFAULT_TABLE table=rebalance_test_table_4 [id=a233217db4ea40e88dd7f9aa60cca533]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:29.542315 23406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85b933bc55b1492fb70055a765a2b09b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:29.542382 23407 tablet_service.cc:1467] Processing CreateTablet for tablet 98dfd1367b224981ab0fd74678bb5ac2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=a233217db4ea40e88dd7f9aa60cca533]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:29.542484 23537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2797c89387af4da5aba9d7d6657eb275. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:29.542716 23407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98dfd1367b224981ab0fd74678bb5ac2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:29.547978 24525 tablet_bootstrap.cc:492] T 85b933bc55b1492fb70055a765a2b09b P 76e30a81c8b8451898078e9636a1104b: Bootstrap starting.
I20240503 08:11:29.548524 24526 tablet_bootstrap.cc:492] T 2797c89387af4da5aba9d7d6657eb275 P 30d88bcd65b347dabcbac54465e52b21: Bootstrap starting.
I20240503 08:11:29.549597 24525 tablet_bootstrap.cc:654] T 85b933bc55b1492fb70055a765a2b09b P 76e30a81c8b8451898078e9636a1104b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:29.550034 24526 tablet_bootstrap.cc:654] T 2797c89387af4da5aba9d7d6657eb275 P 30d88bcd65b347dabcbac54465e52b21: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:29.551321 24525 tablet_bootstrap.cc:492] T 85b933bc55b1492fb70055a765a2b09b P 76e30a81c8b8451898078e9636a1104b: No bootstrap required, opened a new log
I20240503 08:11:29.551450 24525 ts_tablet_manager.cc:1367] T 85b933bc55b1492fb70055a765a2b09b P 76e30a81c8b8451898078e9636a1104b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 08:11:29.551684 24526 tablet_bootstrap.cc:492] T 2797c89387af4da5aba9d7d6657eb275 P 30d88bcd65b347dabcbac54465e52b21: No bootstrap required, opened a new log
I20240503 08:11:29.551805 24526 ts_tablet_manager.cc:1367] T 2797c89387af4da5aba9d7d6657eb275 P 30d88bcd65b347dabcbac54465e52b21: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:11:29.552157 24525 raft_consensus.cc:348] T 85b933bc55b1492fb70055a765a2b09b P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } }
I20240503 08:11:29.552332 24525 raft_consensus.cc:374] T 85b933bc55b1492fb70055a765a2b09b P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:29.552428 24525 raft_consensus.cc:729] T 85b933bc55b1492fb70055a765a2b09b P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76e30a81c8b8451898078e9636a1104b, State: Initialized, Role: FOLLOWER
I20240503 08:11:29.552487 24526 raft_consensus.cc:348] T 2797c89387af4da5aba9d7d6657eb275 P 30d88bcd65b347dabcbac54465e52b21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } }
I20240503 08:11:29.552639 24526 raft_consensus.cc:374] T 2797c89387af4da5aba9d7d6657eb275 P 30d88bcd65b347dabcbac54465e52b21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:29.552727 24526 raft_consensus.cc:729] T 2797c89387af4da5aba9d7d6657eb275 P 30d88bcd65b347dabcbac54465e52b21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30d88bcd65b347dabcbac54465e52b21, State: Initialized, Role: FOLLOWER
I20240503 08:11:29.552655 24525 consensus_queue.cc:260] T 85b933bc55b1492fb70055a765a2b09b P 76e30a81c8b8451898078e9636a1104b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } }
I20240503 08:11:29.552829 24525 raft_consensus.cc:388] T 85b933bc55b1492fb70055a765a2b09b P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:29.552903 24525 raft_consensus.cc:482] T 85b933bc55b1492fb70055a765a2b09b P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:29.552929 24526 consensus_queue.cc:260] T 2797c89387af4da5aba9d7d6657eb275 P 30d88bcd65b347dabcbac54465e52b21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } }
I20240503 08:11:29.553006 24525 raft_consensus.cc:3037] T 85b933bc55b1492fb70055a765a2b09b P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:29.553098 24526 raft_consensus.cc:388] T 2797c89387af4da5aba9d7d6657eb275 P 30d88bcd65b347dabcbac54465e52b21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:29.553170 24526 raft_consensus.cc:482] T 2797c89387af4da5aba9d7d6657eb275 P 30d88bcd65b347dabcbac54465e52b21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:29.553272 24526 raft_consensus.cc:3037] T 2797c89387af4da5aba9d7d6657eb275 P 30d88bcd65b347dabcbac54465e52b21 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:29.555563 24525 raft_consensus.cc:504] T 85b933bc55b1492fb70055a765a2b09b P 76e30a81c8b8451898078e9636a1104b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } }
I20240503 08:11:29.555708 24526 raft_consensus.cc:504] T 2797c89387af4da5aba9d7d6657eb275 P 30d88bcd65b347dabcbac54465e52b21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } }
I20240503 08:11:29.555824 24525 leader_election.cc:302] T 85b933bc55b1492fb70055a765a2b09b P 76e30a81c8b8451898078e9636a1104b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76e30a81c8b8451898078e9636a1104b; no voters:
I20240503 08:11:29.555922 24526 leader_election.cc:302] T 2797c89387af4da5aba9d7d6657eb275 P 30d88bcd65b347dabcbac54465e52b21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30d88bcd65b347dabcbac54465e52b21; no voters:
I20240503 08:11:29.556244 24525 leader_election.cc:288] T 85b933bc55b1492fb70055a765a2b09b P 76e30a81c8b8451898078e9636a1104b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:29.556375 24529 raft_consensus.cc:2781] T 85b933bc55b1492fb70055a765a2b09b P 76e30a81c8b8451898078e9636a1104b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:29.563024 24529 raft_consensus.cc:686] T 85b933bc55b1492fb70055a765a2b09b P 76e30a81c8b8451898078e9636a1104b [term 1 LEADER]: Becoming Leader. State: Replica: 76e30a81c8b8451898078e9636a1104b, State: Running, Role: LEADER
I20240503 08:11:29.563398 24529 consensus_queue.cc:237] T 85b933bc55b1492fb70055a765a2b09b P 76e30a81c8b8451898078e9636a1104b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } }
I20240503 08:11:29.569797 23281 catalog_manager.cc:5484] T 85b933bc55b1492fb70055a765a2b09b P 76e30a81c8b8451898078e9636a1104b reported cstate change: term changed from 0 to 1, leader changed from <none> to 76e30a81c8b8451898078e9636a1104b (127.20.230.65). New cstate: current_term: 1 leader_uuid: "76e30a81c8b8451898078e9636a1104b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } }
I20240503 08:11:29.581089 24526 leader_election.cc:288] T 2797c89387af4da5aba9d7d6657eb275 P 30d88bcd65b347dabcbac54465e52b21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:29.581158 24530 raft_consensus.cc:2781] T 2797c89387af4da5aba9d7d6657eb275 P 30d88bcd65b347dabcbac54465e52b21 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:29.581609 24530 raft_consensus.cc:686] T 2797c89387af4da5aba9d7d6657eb275 P 30d88bcd65b347dabcbac54465e52b21 [term 1 LEADER]: Becoming Leader. State: Replica: 30d88bcd65b347dabcbac54465e52b21, State: Running, Role: LEADER
I20240503 08:11:29.581750 24525 ts_tablet_manager.cc:1398] T 85b933bc55b1492fb70055a765a2b09b P 76e30a81c8b8451898078e9636a1104b: Time spent starting tablet: real 0.030s user 0.020s sys 0.010s
I20240503 08:11:29.582141 24525 tablet_bootstrap.cc:492] T 98dfd1367b224981ab0fd74678bb5ac2 P 76e30a81c8b8451898078e9636a1104b: Bootstrap starting.
I20240503 08:11:29.582182 24530 consensus_queue.cc:237] T 2797c89387af4da5aba9d7d6657eb275 P 30d88bcd65b347dabcbac54465e52b21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } }
I20240503 08:11:29.583829 24525 tablet_bootstrap.cc:654] T 98dfd1367b224981ab0fd74678bb5ac2 P 76e30a81c8b8451898078e9636a1104b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:29.585986 23281 catalog_manager.cc:5484] T 2797c89387af4da5aba9d7d6657eb275 P 30d88bcd65b347dabcbac54465e52b21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30d88bcd65b347dabcbac54465e52b21 (127.20.230.66). New cstate: current_term: 1 leader_uuid: "30d88bcd65b347dabcbac54465e52b21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } }
I20240503 08:11:29.589388 24526 ts_tablet_manager.cc:1398] T 2797c89387af4da5aba9d7d6657eb275 P 30d88bcd65b347dabcbac54465e52b21: Time spent starting tablet: real 0.037s user 0.009s sys 0.023s
I20240503 08:11:29.590626 24525 tablet_bootstrap.cc:492] T 98dfd1367b224981ab0fd74678bb5ac2 P 76e30a81c8b8451898078e9636a1104b: No bootstrap required, opened a new log
I20240503 08:11:29.590766 24525 ts_tablet_manager.cc:1367] T 98dfd1367b224981ab0fd74678bb5ac2 P 76e30a81c8b8451898078e9636a1104b: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240503 08:11:29.591325 24525 raft_consensus.cc:348] T 98dfd1367b224981ab0fd74678bb5ac2 P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } }
I20240503 08:11:29.591480 24525 raft_consensus.cc:374] T 98dfd1367b224981ab0fd74678bb5ac2 P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:29.591550 24525 raft_consensus.cc:729] T 98dfd1367b224981ab0fd74678bb5ac2 P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76e30a81c8b8451898078e9636a1104b, State: Initialized, Role: FOLLOWER
I20240503 08:11:29.591770 24525 consensus_queue.cc:260] T 98dfd1367b224981ab0fd74678bb5ac2 P 76e30a81c8b8451898078e9636a1104b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } }
I20240503 08:11:29.591946 24525 raft_consensus.cc:388] T 98dfd1367b224981ab0fd74678bb5ac2 P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:29.592063 24525 raft_consensus.cc:482] T 98dfd1367b224981ab0fd74678bb5ac2 P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:29.592159 24525 raft_consensus.cc:3037] T 98dfd1367b224981ab0fd74678bb5ac2 P 76e30a81c8b8451898078e9636a1104b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:29.593958 24525 raft_consensus.cc:504] T 98dfd1367b224981ab0fd74678bb5ac2 P 76e30a81c8b8451898078e9636a1104b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } }
I20240503 08:11:29.594136 24525 leader_election.cc:302] T 98dfd1367b224981ab0fd74678bb5ac2 P 76e30a81c8b8451898078e9636a1104b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76e30a81c8b8451898078e9636a1104b; no voters:
I20240503 08:11:29.594334 24525 leader_election.cc:288] T 98dfd1367b224981ab0fd74678bb5ac2 P 76e30a81c8b8451898078e9636a1104b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:29.594417 24531 raft_consensus.cc:2781] T 98dfd1367b224981ab0fd74678bb5ac2 P 76e30a81c8b8451898078e9636a1104b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:29.594642 24531 raft_consensus.cc:686] T 98dfd1367b224981ab0fd74678bb5ac2 P 76e30a81c8b8451898078e9636a1104b [term 1 LEADER]: Becoming Leader. State: Replica: 76e30a81c8b8451898078e9636a1104b, State: Running, Role: LEADER
I20240503 08:11:29.594709 24525 ts_tablet_manager.cc:1398] T 98dfd1367b224981ab0fd74678bb5ac2 P 76e30a81c8b8451898078e9636a1104b: Time spent starting tablet: real 0.004s user 0.002s sys 0.003s
I20240503 08:11:29.594913 24531 consensus_queue.cc:237] T 98dfd1367b224981ab0fd74678bb5ac2 P 76e30a81c8b8451898078e9636a1104b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } }
I20240503 08:11:29.596834 23281 catalog_manager.cc:5484] T 98dfd1367b224981ab0fd74678bb5ac2 P 76e30a81c8b8451898078e9636a1104b reported cstate change: term changed from 0 to 1, leader changed from <none> to 76e30a81c8b8451898078e9636a1104b (127.20.230.65). New cstate: current_term: 1 leader_uuid: "76e30a81c8b8451898078e9636a1104b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } }
W20240503 08:11:29.912891 24541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:29.913172 24541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:29.920197 24541 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 133.031 ms
time per row: 0.0665153 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 08:11:31.038682 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.67:37877
--local_ip_for_outbound_sockets=127.20.230.67
--tserver_master_addrs=127.20.230.126:38671
--webserver_port=38497
--webserver_interface=127.20.230.67
--builtin_ntp_servers=127.20.230.84:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 08:11:31.348114 24579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:31.348398 24579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:31.348524 24579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:31.355573 24579 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:11:31.355700 24579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:31.355844 24579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.67
I20240503 08:11:31.364223 24579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.67:37877
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.230.67
--webserver_port=38497
--tserver_master_addrs=127.20.230.126:38671
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.67
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:31.365465 24579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:31.367110 24579 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:31.378085 24584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:31.378813 24587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:31.379490 24585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:31.380139 24579 server_base.cc:1034] running on GCE node
I20240503 08:11:31.380643 24579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:31.381446 24579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:31.382629 24579 hybrid_clock.cc:648] HybridClock initialized: now 1714723891382586 us; error 38 us; skew 500 ppm
I20240503 08:11:31.382903 24579 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:31.384862 24579 webserver.cc:460] Webserver started at http://127.20.230.67:38497/ using document root <none> and password file <none>
I20240503 08:11:31.385546 24579 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:31.385663 24579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:31.390372 24579 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 08:11:31.392771 24593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:31.393962 24579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:11:31.394104 24579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "5ea58b5961b54598878bad3f5660913c"
format_stamp: "Formatted at 2024-05-03 08:11:18 on dist-test-slave-6vp8"
I20240503 08:11:31.394824 24579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:31.418637 24579 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:31.419334 24579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:31.419587 24579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:31.420403 24579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:31.421746 24579 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:31.421833 24579 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:31.421948 24579 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:31.422003 24579 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:31.449569 24579 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.67:37877
I20240503 08:11:31.449601 24705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.67:37877 every 8 connection(s)
I20240503 08:11:31.450896 24579 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:11:31.455364 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 24579
I20240503 08:11:31.456053 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.68:44977
--local_ip_for_outbound_sockets=127.20.230.68
--tserver_master_addrs=127.20.230.126:38671
--webserver_port=43945
--webserver_interface=127.20.230.68
--builtin_ntp_servers=127.20.230.84:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:11:31.463271 24706 heartbeater.cc:347] Connected to a master server at 127.20.230.126:38671
I20240503 08:11:31.463521 24706 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:31.464131 24706 heartbeater.cc:511] Master 127.20.230.126:38671 requested a full tablet report, sending...
I20240503 08:11:31.465461 23284 ts_manager.cc:194] Re-registered known tserver with Master: 5ea58b5961b54598878bad3f5660913c (127.20.230.67:37877)
I20240503 08:11:31.466358 23284 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.67:45661
W20240503 08:11:31.756664 24710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:31.756937 24710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:31.757046 24710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:31.764142 24710 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:11:31.764271 24710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:31.764412 24710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.68
I20240503 08:11:31.773079 24710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.68:44977
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.230.68
--webserver_port=43945
--tserver_master_addrs=127.20.230.126:38671
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.68
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:31.774345 24710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:31.775946 24710 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:31.787087 24715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:31.787788 24718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:31.788398 24716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:31.789103 24710 server_base.cc:1034] running on GCE node
I20240503 08:11:31.789631 24710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:31.790462 24710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:31.791652 24710 hybrid_clock.cc:648] HybridClock initialized: now 1714723891791612 us; error 36 us; skew 500 ppm
I20240503 08:11:31.791940 24710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:31.793999 24710 webserver.cc:460] Webserver started at http://127.20.230.68:43945/ using document root <none> and password file <none>
I20240503 08:11:31.794641 24710 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:31.794763 24710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:31.799506 24710 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20240503 08:11:31.802053 24724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:31.803213 24710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20240503 08:11:31.803362 24710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "51d0bc1740db4723a71f1d56b7e6e6b7"
format_stamp: "Formatted at 2024-05-03 08:11:18 on dist-test-slave-6vp8"
I20240503 08:11:31.804107 24710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:31.852720 24710 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:31.853503 24710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:31.853816 24710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:31.854761 24710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:31.856454 24710 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:31.856547 24710 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:31.856669 24710 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:31.856725 24710 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:31.884196 24710 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.68:44977
I20240503 08:11:31.884256 24836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.68:44977 every 8 connection(s)
I20240503 08:11:31.885596 24710 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:11:31.891618 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 24710
I20240503 08:11:31.892184 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.69:41435
--local_ip_for_outbound_sockets=127.20.230.69
--tserver_master_addrs=127.20.230.126:38671
--webserver_port=43303
--webserver_interface=127.20.230.69
--builtin_ntp_servers=127.20.230.84:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:11:31.897616 24837 heartbeater.cc:347] Connected to a master server at 127.20.230.126:38671
I20240503 08:11:31.897929 24837 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:31.898787 24837 heartbeater.cc:511] Master 127.20.230.126:38671 requested a full tablet report, sending...
I20240503 08:11:31.900106 23284 ts_manager.cc:194] Re-registered known tserver with Master: 51d0bc1740db4723a71f1d56b7e6e6b7 (127.20.230.68:44977)
I20240503 08:11:31.900823 23284 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.68:33621
W20240503 08:11:32.190104 24840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:32.190347 24840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:32.190460 24840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:32.197420 24840 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:11:32.197523 24840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:32.197669 24840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.69
I20240503 08:11:32.206104 24840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.69:41435
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.230.69
--webserver_port=43303
--tserver_master_addrs=127.20.230.126:38671
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.69
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:32.207250 24840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:32.209108 24840 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:32.220217 24845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:32.220883 24846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:32.221546 24848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:32.222195 24840 server_base.cc:1034] running on GCE node
I20240503 08:11:32.222710 24840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:32.223531 24840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:32.224716 24840 hybrid_clock.cc:648] HybridClock initialized: now 1714723892224675 us; error 31 us; skew 500 ppm
I20240503 08:11:32.224990 24840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:32.227041 24840 webserver.cc:460] Webserver started at http://127.20.230.69:43303/ using document root <none> and password file <none>
I20240503 08:11:32.227680 24840 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:32.227794 24840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:32.232497 24840 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 08:11:32.235024 24854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:32.236254 24840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:11:32.236394 24840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "509c32504f544f8b90ebf66b6dd951f7"
format_stamp: "Formatted at 2024-05-03 08:11:19 on dist-test-slave-6vp8"
I20240503 08:11:32.237200 24840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:32.252517 24840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:32.253206 24840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:32.253463 24840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:32.254308 24840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:32.255637 24840 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:32.255717 24840 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:32.255838 24840 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:32.255889 24840 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:32.283977 24840 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.69:41435
I20240503 08:11:32.284011 24966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.69:41435 every 8 connection(s)
I20240503 08:11:32.285295 24840 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:11:32.287451 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 24840
I20240503 08:11:32.288049 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.70:36811
--local_ip_for_outbound_sockets=127.20.230.70
--tserver_master_addrs=127.20.230.126:38671
--webserver_port=41101
--webserver_interface=127.20.230.70
--builtin_ntp_servers=127.20.230.84:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:11:32.297394 24967 heartbeater.cc:347] Connected to a master server at 127.20.230.126:38671
I20240503 08:11:32.297672 24967 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:32.298425 24967 heartbeater.cc:511] Master 127.20.230.126:38671 requested a full tablet report, sending...
I20240503 08:11:32.299741 23284 ts_manager.cc:194] Re-registered known tserver with Master: 509c32504f544f8b90ebf66b6dd951f7 (127.20.230.69:41435)
I20240503 08:11:32.300472 23284 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.69:41883
I20240503 08:11:32.468509 24706 heartbeater.cc:503] Master 127.20.230.126:38671 was elected leader, sending a full tablet report...
W20240503 08:11:32.590977 24970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:32.591320 24970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:32.591459 24970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:32.598526 24970 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:11:32.598642 24970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:32.598778 24970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.70
I20240503 08:11:32.608103 24970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.70:36811
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.20.230.70
--webserver_port=41101
--tserver_master_addrs=127.20.230.126:38671
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.70
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:32.609342 24970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:32.611020 24970 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:32.622119 24976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:32.622865 24978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:32.623488 24975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:32.624151 24970 server_base.cc:1034] running on GCE node
I20240503 08:11:32.624656 24970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:32.625439 24970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:32.626616 24970 hybrid_clock.cc:648] HybridClock initialized: now 1714723892626577 us; error 35 us; skew 500 ppm
I20240503 08:11:32.626895 24970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:32.628854 24970 webserver.cc:460] Webserver started at http://127.20.230.70:41101/ using document root <none> and password file <none>
I20240503 08:11:32.629478 24970 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:32.629591 24970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:32.634267 24970 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20240503 08:11:32.636777 24984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:32.638072 24970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:11:32.638211 24970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "c82f2e5e9e13405d9328620179077098"
format_stamp: "Formatted at 2024-05-03 08:11:19 on dist-test-slave-6vp8"
I20240503 08:11:32.638993 24970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:32.656014 24970 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:32.656747 24970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:32.657016 24970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:32.657886 24970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:32.659613 24991 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240503 08:11:32.664428 24970 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 08:11:32.664508 24970 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.001s
I20240503 08:11:32.664636 24970 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 08:11:32.667228 24970 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 08:11:32.667315 24970 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20240503 08:11:32.667629 24991 tablet_bootstrap.cc:492] T 7879251014c84b11a0f49d707d80d85b P c82f2e5e9e13405d9328620179077098: Bootstrap starting.
I20240503 08:11:32.686156 24991 log.cc:824] T 7879251014c84b11a0f49d707d80d85b P c82f2e5e9e13405d9328620179077098: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:32.689334 24991 tablet_bootstrap.cc:492] T 7879251014c84b11a0f49d707d80d85b P c82f2e5e9e13405d9328620179077098: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:32.689987 24991 tablet_bootstrap.cc:492] T 7879251014c84b11a0f49d707d80d85b P c82f2e5e9e13405d9328620179077098: Bootstrap complete.
I20240503 08:11:32.691182 24991 ts_tablet_manager.cc:1367] T 7879251014c84b11a0f49d707d80d85b P c82f2e5e9e13405d9328620179077098: Time spent bootstrapping tablet: real 0.024s user 0.014s sys 0.008s
I20240503 08:11:32.700778 24991 raft_consensus.cc:348] T 7879251014c84b11a0f49d707d80d85b P c82f2e5e9e13405d9328620179077098 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c82f2e5e9e13405d9328620179077098" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 36811 } }
I20240503 08:11:32.701812 24991 raft_consensus.cc:729] T 7879251014c84b11a0f49d707d80d85b P c82f2e5e9e13405d9328620179077098 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c82f2e5e9e13405d9328620179077098, State: Initialized, Role: FOLLOWER
I20240503 08:11:32.702479 24991 consensus_queue.cc:260] T 7879251014c84b11a0f49d707d80d85b P c82f2e5e9e13405d9328620179077098 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c82f2e5e9e13405d9328620179077098" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 36811 } }
I20240503 08:11:32.702679 24991 raft_consensus.cc:388] T 7879251014c84b11a0f49d707d80d85b P c82f2e5e9e13405d9328620179077098 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:32.702800 24991 raft_consensus.cc:482] T 7879251014c84b11a0f49d707d80d85b P c82f2e5e9e13405d9328620179077098 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:32.702973 24991 raft_consensus.cc:3037] T 7879251014c84b11a0f49d707d80d85b P c82f2e5e9e13405d9328620179077098 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:32.706650 24970 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.70:36811
I20240503 08:11:32.706559 24991 raft_consensus.cc:504] T 7879251014c84b11a0f49d707d80d85b P c82f2e5e9e13405d9328620179077098 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c82f2e5e9e13405d9328620179077098" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 36811 } }
I20240503 08:11:32.706758 25098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.70:36811 every 8 connection(s)
I20240503 08:11:32.707093 24991 leader_election.cc:302] T 7879251014c84b11a0f49d707d80d85b P c82f2e5e9e13405d9328620179077098 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c82f2e5e9e13405d9328620179077098; no voters:
I20240503 08:11:32.707674 24991 leader_election.cc:288] T 7879251014c84b11a0f49d707d80d85b P c82f2e5e9e13405d9328620179077098 [CANDIDATE]: Term 2 election: Requested vote from peers
I20240503 08:11:32.707890 25099 raft_consensus.cc:2781] T 7879251014c84b11a0f49d707d80d85b P c82f2e5e9e13405d9328620179077098 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:11:32.708014 24970 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 08:11:32.708567 25099 raft_consensus.cc:686] T 7879251014c84b11a0f49d707d80d85b P c82f2e5e9e13405d9328620179077098 [term 2 LEADER]: Becoming Leader. State: Replica: c82f2e5e9e13405d9328620179077098, State: Running, Role: LEADER
I20240503 08:11:32.709128 25099 consensus_queue.cc:237] T 7879251014c84b11a0f49d707d80d85b P c82f2e5e9e13405d9328620179077098 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c82f2e5e9e13405d9328620179077098" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 36811 } }
I20240503 08:11:32.713156 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 24970
I20240503 08:11:32.713951 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.71:38073
--local_ip_for_outbound_sockets=127.20.230.71
--tserver_master_addrs=127.20.230.126:38671
--webserver_port=45417
--webserver_interface=127.20.230.71
--builtin_ntp_servers=127.20.230.84:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:11:32.715477 24991 ts_tablet_manager.cc:1398] T 7879251014c84b11a0f49d707d80d85b P c82f2e5e9e13405d9328620179077098: Time spent starting tablet: real 0.024s user 0.020s sys 0.004s
I20240503 08:11:32.725718 25102 heartbeater.cc:347] Connected to a master server at 127.20.230.126:38671
I20240503 08:11:32.726042 25102 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:32.726867 25102 heartbeater.cc:511] Master 127.20.230.126:38671 requested a full tablet report, sending...
I20240503 08:11:32.729291 23284 ts_manager.cc:194] Re-registered known tserver with Master: c82f2e5e9e13405d9328620179077098 (127.20.230.70:36811)
I20240503 08:11:32.729647 23284 catalog_manager.cc:5484] T 7879251014c84b11a0f49d707d80d85b P c82f2e5e9e13405d9328620179077098 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "c82f2e5e9e13405d9328620179077098" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c82f2e5e9e13405d9328620179077098" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 36811 } } }
I20240503 08:11:32.734267 23284 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.70:49479
I20240503 08:11:32.736811 25102 heartbeater.cc:503] Master 127.20.230.126:38671 was elected leader, sending a full tablet report...
I20240503 08:11:32.903465 24837 heartbeater.cc:503] Master 127.20.230.126:38671 was elected leader, sending a full tablet report...
W20240503 08:11:33.026317 25107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:33.026579 25107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:33.026696 25107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:33.034222 25107 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:11:33.034333 25107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:33.034482 25107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.71
I20240503 08:11:33.043213 25107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.71:38073
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.20.230.71
--webserver_port=45417
--tserver_master_addrs=127.20.230.126:38671
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.71
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:33.044482 25107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:33.046149 25107 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:33.058132 25116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:33.058699 25119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:33.059355 25117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:33.059975 25107 server_base.cc:1034] running on GCE node
I20240503 08:11:33.060528 25107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:33.061416 25107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:33.062611 25107 hybrid_clock.cc:648] HybridClock initialized: now 1714723893062568 us; error 43 us; skew 500 ppm
I20240503 08:11:33.062898 25107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:33.064955 25107 webserver.cc:460] Webserver started at http://127.20.230.71:45417/ using document root <none> and password file <none>
I20240503 08:11:33.065629 25107 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:33.065744 25107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:33.070483 25107 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20240503 08:11:33.072948 25125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:33.074144 25107 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 08:11:33.074286 25107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "7823cb7f30b045209bc5ccfb110bf199"
format_stamp: "Formatted at 2024-05-03 08:11:20 on dist-test-slave-6vp8"
I20240503 08:11:33.075069 25107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:33.101428 25107 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:33.102185 25107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:33.102444 25107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:33.103286 25107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:33.104682 25107 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:33.104756 25107 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:33.104872 25107 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:33.104923 25107 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:33.135246 25107 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.71:38073
I20240503 08:11:33.135288 25237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.71:38073 every 8 connection(s)
I20240503 08:11:33.136677 25107 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 08:11:33.141199 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 25107
I20240503 08:11:33.148761 25238 heartbeater.cc:347] Connected to a master server at 127.20.230.126:38671
I20240503 08:11:33.149008 25238 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:33.149636 25238 heartbeater.cc:511] Master 127.20.230.126:38671 requested a full tablet report, sending...
I20240503 08:11:33.150923 23284 ts_manager.cc:194] Re-registered known tserver with Master: 7823cb7f30b045209bc5ccfb110bf199 (127.20.230.71:38073)
I20240503 08:11:33.151968 23284 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.71:39265
I20240503 08:11:33.302819 24967 heartbeater.cc:503] Master 127.20.230.126:38671 was elected leader, sending a full tablet report...
I20240503 08:11:33.593384 23537 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:11:33.607393 24902 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:33.623049 24772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:33.635349 24641 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:33.640576 23407 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:11:33.672199 25033 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:33.679971 25173 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:33.870031 23537 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:11:33.880501 24772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:33.883384 23407 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:11:33.883803 24641 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:33.910094 25173 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:33.910111 25033 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:33.922441 24902 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:34.016830 23427 consensus_queue.cc:237] T 85b933bc55b1492fb70055a765a2b09b P 76e30a81c8b8451898078e9636a1104b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "509c32504f544f8b90ebf66b6dd951f7" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 41435 } }
W20240503 08:11:34.026067 23362 consensus_peers.cc:485] T 85b933bc55b1492fb70055a765a2b09b P 76e30a81c8b8451898078e9636a1104b -> Peer 509c32504f544f8b90ebf66b6dd951f7 (127.20.230.69:41435): Couldn't send request to peer 509c32504f544f8b90ebf66b6dd951f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 85b933bc55b1492fb70055a765a2b09b. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:34.154167 25238 heartbeater.cc:503] Master 127.20.230.126:38671 was elected leader, sending a full tablet report...
I20240503 08:11:34.544481 25339 ts_tablet_manager.cc:902] T 85b933bc55b1492fb70055a765a2b09b P 509c32504f544f8b90ebf66b6dd951f7: Initiating tablet copy from peer 76e30a81c8b8451898078e9636a1104b (127.20.230.65:37573)
I20240503 08:11:34.545985 25339 tablet_copy_client.cc:308] T 85b933bc55b1492fb70055a765a2b09b P 509c32504f544f8b90ebf66b6dd951f7: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.65:37573
I20240503 08:11:34.552861 23447 tablet_copy_service.cc:140] P 76e30a81c8b8451898078e9636a1104b: Received BeginTabletCopySession request for tablet 85b933bc55b1492fb70055a765a2b09b from peer 509c32504f544f8b90ebf66b6dd951f7 ({username='slave'} at 127.20.230.69:47793)
I20240503 08:11:34.553094 23447 tablet_copy_service.cc:161] P 76e30a81c8b8451898078e9636a1104b: Beginning new tablet copy session on tablet 85b933bc55b1492fb70055a765a2b09b from peer 509c32504f544f8b90ebf66b6dd951f7 at {username='slave'} at 127.20.230.69:47793: session id = 509c32504f544f8b90ebf66b6dd951f7-85b933bc55b1492fb70055a765a2b09b
I20240503 08:11:34.555680 23447 tablet_copy_source_session.cc:205] T 85b933bc55b1492fb70055a765a2b09b P 76e30a81c8b8451898078e9636a1104b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:34.560411 25339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85b933bc55b1492fb70055a765a2b09b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:34.570459 25339 tablet_copy_client.cc:783] T 85b933bc55b1492fb70055a765a2b09b P 509c32504f544f8b90ebf66b6dd951f7: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:34.570752 25339 tablet_copy_client.cc:647] T 85b933bc55b1492fb70055a765a2b09b P 509c32504f544f8b90ebf66b6dd951f7: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:34.572959 25339 tablet_copy_client.cc:523] T 85b933bc55b1492fb70055a765a2b09b P 509c32504f544f8b90ebf66b6dd951f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:34.575835 25339 tablet_bootstrap.cc:492] T 85b933bc55b1492fb70055a765a2b09b P 509c32504f544f8b90ebf66b6dd951f7: Bootstrap starting.
I20240503 08:11:34.585182 25339 log.cc:824] T 85b933bc55b1492fb70055a765a2b09b P 509c32504f544f8b90ebf66b6dd951f7: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:34.616277 25339 tablet_bootstrap.cc:492] T 85b933bc55b1492fb70055a765a2b09b P 509c32504f544f8b90ebf66b6dd951f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:11:34.616750 25339 tablet_bootstrap.cc:492] T 85b933bc55b1492fb70055a765a2b09b P 509c32504f544f8b90ebf66b6dd951f7: Bootstrap complete.
I20240503 08:11:34.617113 25339 ts_tablet_manager.cc:1367] T 85b933bc55b1492fb70055a765a2b09b P 509c32504f544f8b90ebf66b6dd951f7: Time spent bootstrapping tablet: real 0.041s user 0.034s sys 0.006s
I20240503 08:11:34.621722 25339 raft_consensus.cc:348] T 85b933bc55b1492fb70055a765a2b09b P 509c32504f544f8b90ebf66b6dd951f7 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } }
I20240503 08:11:34.622782 25339 raft_consensus.cc:729] T 85b933bc55b1492fb70055a765a2b09b P 509c32504f544f8b90ebf66b6dd951f7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 509c32504f544f8b90ebf66b6dd951f7, State: Initialized, Role: FOLLOWER
I20240503 08:11:34.623320 25339 consensus_queue.cc:260] T 85b933bc55b1492fb70055a765a2b09b P 509c32504f544f8b90ebf66b6dd951f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "509c32504f544f8b90ebf66b6dd951f7" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 41435 } }
I20240503 08:11:34.625080 25339 ts_tablet_manager.cc:1398] T 85b933bc55b1492fb70055a765a2b09b P 509c32504f544f8b90ebf66b6dd951f7: Time spent starting tablet: real 0.008s user 0.010s sys 0.000s
I20240503 08:11:34.626533 23447 tablet_copy_service.cc:342] P 76e30a81c8b8451898078e9636a1104b: Request end of tablet copy session 509c32504f544f8b90ebf66b6dd951f7-85b933bc55b1492fb70055a765a2b09b received from {username='slave'} at 127.20.230.69:47793
I20240503 08:11:34.626715 23447 tablet_copy_service.cc:433] P 76e30a81c8b8451898078e9636a1104b: ending tablet copy session 509c32504f544f8b90ebf66b6dd951f7-85b933bc55b1492fb70055a765a2b09b on tablet 85b933bc55b1492fb70055a765a2b09b with peer 509c32504f544f8b90ebf66b6dd951f7
I20240503 08:11:35.065948 24922 raft_consensus.cc:1203] T 85b933bc55b1492fb70055a765a2b09b P 509c32504f544f8b90ebf66b6dd951f7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:11:35.067731 25349 raft_consensus.cc:2932] T 85b933bc55b1492fb70055a765a2b09b P 76e30a81c8b8451898078e9636a1104b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 509c32504f544f8b90ebf66b6dd951f7 (127.20.230.69) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "509c32504f544f8b90ebf66b6dd951f7" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 41435 } } }
I20240503 08:11:35.068379 24922 raft_consensus.cc:2932] T 85b933bc55b1492fb70055a765a2b09b P 509c32504f544f8b90ebf66b6dd951f7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 509c32504f544f8b90ebf66b6dd951f7 (127.20.230.69) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "509c32504f544f8b90ebf66b6dd951f7" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 41435 } } }
I20240503 08:11:35.071276 25349 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:39766 (request call id 0) took 1055 ms. Trace:
I20240503 08:11:35.071416 25349 rpcz_store.cc:276] 0503 08:11:34.015688 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 08:11:34.015771 (+ 83us) service_pool.cc:225] Handling call
0503 08:11:34.018075 (+ 2304us) log.cc:842] Serialized 260 byte log entry
0503 08:11:35.071230 (+1053155us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":76,"raft.queue_time_us":386,"raft.run_cpu_time_us":501,"raft.run_wall_time_us":501,"spinlock_wait_cycles":71040,"thread_start_us":477,"threads_started":2,"wal-append.queue_time_us":337,"wal-append.run_cpu_time_us":613,"wal-append.run_wall_time_us":1000694}
I20240503 08:11:35.074143 23280 catalog_manager.cc:5484] T 85b933bc55b1492fb70055a765a2b09b P 76e30a81c8b8451898078e9636a1104b reported cstate change: config changed from index -1 to 4, VOTER 509c32504f544f8b90ebf66b6dd951f7 (127.20.230.69) added. New cstate: current_term: 1 leader_uuid: "76e30a81c8b8451898078e9636a1104b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "509c32504f544f8b90ebf66b6dd951f7" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 41435 } } }
I20240503 08:11:35.108803 23427 consensus_queue.cc:237] T 0d69b1adfa064beb82bdcc16d53717d7 P 76e30a81c8b8451898078e9636a1104b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "5ea58b5961b54598878bad3f5660913c" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 37877 } }
W20240503 08:11:35.116847 23361 consensus_peers.cc:485] T 0d69b1adfa064beb82bdcc16d53717d7 P 76e30a81c8b8451898078e9636a1104b -> Peer 5ea58b5961b54598878bad3f5660913c (127.20.230.67:37877): Couldn't send request to peer 5ea58b5961b54598878bad3f5660913c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0d69b1adfa064beb82bdcc16d53717d7. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:35.590353 25376 ts_tablet_manager.cc:902] T 0d69b1adfa064beb82bdcc16d53717d7 P 5ea58b5961b54598878bad3f5660913c: Initiating tablet copy from peer 76e30a81c8b8451898078e9636a1104b (127.20.230.65:37573)
I20240503 08:11:35.591997 25376 tablet_copy_client.cc:308] T 0d69b1adfa064beb82bdcc16d53717d7 P 5ea58b5961b54598878bad3f5660913c: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.65:37573
I20240503 08:11:35.597579 23447 tablet_copy_service.cc:140] P 76e30a81c8b8451898078e9636a1104b: Received BeginTabletCopySession request for tablet 0d69b1adfa064beb82bdcc16d53717d7 from peer 5ea58b5961b54598878bad3f5660913c ({username='slave'} at 127.20.230.67:38301)
I20240503 08:11:35.597734 23447 tablet_copy_service.cc:161] P 76e30a81c8b8451898078e9636a1104b: Beginning new tablet copy session on tablet 0d69b1adfa064beb82bdcc16d53717d7 from peer 5ea58b5961b54598878bad3f5660913c at {username='slave'} at 127.20.230.67:38301: session id = 5ea58b5961b54598878bad3f5660913c-0d69b1adfa064beb82bdcc16d53717d7
I20240503 08:11:35.600296 23447 tablet_copy_source_session.cc:205] T 0d69b1adfa064beb82bdcc16d53717d7 P 76e30a81c8b8451898078e9636a1104b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:35.604269 25376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d69b1adfa064beb82bdcc16d53717d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:35.613816 25376 tablet_copy_client.cc:783] T 0d69b1adfa064beb82bdcc16d53717d7 P 5ea58b5961b54598878bad3f5660913c: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:35.614068 25376 tablet_copy_client.cc:647] T 0d69b1adfa064beb82bdcc16d53717d7 P 5ea58b5961b54598878bad3f5660913c: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:35.615885 25376 tablet_copy_client.cc:523] T 0d69b1adfa064beb82bdcc16d53717d7 P 5ea58b5961b54598878bad3f5660913c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:35.618568 25376 tablet_bootstrap.cc:492] T 0d69b1adfa064beb82bdcc16d53717d7 P 5ea58b5961b54598878bad3f5660913c: Bootstrap starting.
I20240503 08:11:35.627285 25376 log.cc:824] T 0d69b1adfa064beb82bdcc16d53717d7 P 5ea58b5961b54598878bad3f5660913c: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:35.656947 25376 tablet_bootstrap.cc:492] T 0d69b1adfa064beb82bdcc16d53717d7 P 5ea58b5961b54598878bad3f5660913c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:11:35.657410 25376 tablet_bootstrap.cc:492] T 0d69b1adfa064beb82bdcc16d53717d7 P 5ea58b5961b54598878bad3f5660913c: Bootstrap complete.
I20240503 08:11:35.657779 25376 ts_tablet_manager.cc:1367] T 0d69b1adfa064beb82bdcc16d53717d7 P 5ea58b5961b54598878bad3f5660913c: Time spent bootstrapping tablet: real 0.039s user 0.036s sys 0.000s
I20240503 08:11:35.661720 25376 raft_consensus.cc:348] T 0d69b1adfa064beb82bdcc16d53717d7 P 5ea58b5961b54598878bad3f5660913c [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } }
I20240503 08:11:35.662451 25376 raft_consensus.cc:729] T 0d69b1adfa064beb82bdcc16d53717d7 P 5ea58b5961b54598878bad3f5660913c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ea58b5961b54598878bad3f5660913c, State: Initialized, Role: FOLLOWER
I20240503 08:11:35.662819 25376 consensus_queue.cc:260] T 0d69b1adfa064beb82bdcc16d53717d7 P 5ea58b5961b54598878bad3f5660913c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "5ea58b5961b54598878bad3f5660913c" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 37877 } }
I20240503 08:11:35.664167 25376 ts_tablet_manager.cc:1398] T 0d69b1adfa064beb82bdcc16d53717d7 P 5ea58b5961b54598878bad3f5660913c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240503 08:11:35.665088 23447 tablet_copy_service.cc:342] P 76e30a81c8b8451898078e9636a1104b: Request end of tablet copy session 5ea58b5961b54598878bad3f5660913c-0d69b1adfa064beb82bdcc16d53717d7 received from {username='slave'} at 127.20.230.67:38301
I20240503 08:11:35.665189 23447 tablet_copy_service.cc:433] P 76e30a81c8b8451898078e9636a1104b: ending tablet copy session 5ea58b5961b54598878bad3f5660913c-0d69b1adfa064beb82bdcc16d53717d7 on tablet 0d69b1adfa064beb82bdcc16d53717d7 with peer 5ea58b5961b54598878bad3f5660913c
I20240503 08:11:36.030279 24661 raft_consensus.cc:1203] T 0d69b1adfa064beb82bdcc16d53717d7 P 5ea58b5961b54598878bad3f5660913c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:11:36.031899 25384 raft_consensus.cc:2932] T 0d69b1adfa064beb82bdcc16d53717d7 P 76e30a81c8b8451898078e9636a1104b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5ea58b5961b54598878bad3f5660913c (127.20.230.67) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "5ea58b5961b54598878bad3f5660913c" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 37877 } } }
I20240503 08:11:36.032727 24661 raft_consensus.cc:2932] T 0d69b1adfa064beb82bdcc16d53717d7 P 5ea58b5961b54598878bad3f5660913c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5ea58b5961b54598878bad3f5660913c (127.20.230.67) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "5ea58b5961b54598878bad3f5660913c" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 37877 } } }
I20240503 08:11:36.037186 23284 catalog_manager.cc:5484] T 0d69b1adfa064beb82bdcc16d53717d7 P 76e30a81c8b8451898078e9636a1104b reported cstate change: config changed from index -1 to 4, VOTER 5ea58b5961b54598878bad3f5660913c (127.20.230.67) added. New cstate: current_term: 1 leader_uuid: "76e30a81c8b8451898078e9636a1104b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "5ea58b5961b54598878bad3f5660913c" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 37877 } } }
I20240503 08:11:36.068867 23557 consensus_queue.cc:237] T 2449909504504d1e84cf34f1af673af2 P 30d88bcd65b347dabcbac54465e52b21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "51d0bc1740db4723a71f1d56b7e6e6b7" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 44977 } }
W20240503 08:11:36.078636 23491 consensus_peers.cc:485] T 2449909504504d1e84cf34f1af673af2 P 30d88bcd65b347dabcbac54465e52b21 -> Peer 51d0bc1740db4723a71f1d56b7e6e6b7 (127.20.230.68:44977): Couldn't send request to peer 51d0bc1740db4723a71f1d56b7e6e6b7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2449909504504d1e84cf34f1af673af2. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:36.637178 25411 ts_tablet_manager.cc:902] T 2449909504504d1e84cf34f1af673af2 P 51d0bc1740db4723a71f1d56b7e6e6b7: Initiating tablet copy from peer 30d88bcd65b347dabcbac54465e52b21 (127.20.230.66:33351)
I20240503 08:11:36.638721 25411 tablet_copy_client.cc:308] T 2449909504504d1e84cf34f1af673af2 P 51d0bc1740db4723a71f1d56b7e6e6b7: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.66:33351
I20240503 08:11:36.645327 23577 tablet_copy_service.cc:140] P 30d88bcd65b347dabcbac54465e52b21: Received BeginTabletCopySession request for tablet 2449909504504d1e84cf34f1af673af2 from peer 51d0bc1740db4723a71f1d56b7e6e6b7 ({username='slave'} at 127.20.230.68:54761)
I20240503 08:11:36.645529 23577 tablet_copy_service.cc:161] P 30d88bcd65b347dabcbac54465e52b21: Beginning new tablet copy session on tablet 2449909504504d1e84cf34f1af673af2 from peer 51d0bc1740db4723a71f1d56b7e6e6b7 at {username='slave'} at 127.20.230.68:54761: session id = 51d0bc1740db4723a71f1d56b7e6e6b7-2449909504504d1e84cf34f1af673af2
I20240503 08:11:36.648108 23577 tablet_copy_source_session.cc:205] T 2449909504504d1e84cf34f1af673af2 P 30d88bcd65b347dabcbac54465e52b21: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:36.652333 25411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2449909504504d1e84cf34f1af673af2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:36.661417 25411 tablet_copy_client.cc:783] T 2449909504504d1e84cf34f1af673af2 P 51d0bc1740db4723a71f1d56b7e6e6b7: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:36.661671 25411 tablet_copy_client.cc:647] T 2449909504504d1e84cf34f1af673af2 P 51d0bc1740db4723a71f1d56b7e6e6b7: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:36.664176 25411 tablet_copy_client.cc:523] T 2449909504504d1e84cf34f1af673af2 P 51d0bc1740db4723a71f1d56b7e6e6b7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:36.666795 25411 tablet_bootstrap.cc:492] T 2449909504504d1e84cf34f1af673af2 P 51d0bc1740db4723a71f1d56b7e6e6b7: Bootstrap starting.
I20240503 08:11:36.674726 25411 log.cc:824] T 2449909504504d1e84cf34f1af673af2 P 51d0bc1740db4723a71f1d56b7e6e6b7: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:36.702113 25411 tablet_bootstrap.cc:492] T 2449909504504d1e84cf34f1af673af2 P 51d0bc1740db4723a71f1d56b7e6e6b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:11:36.702576 25411 tablet_bootstrap.cc:492] T 2449909504504d1e84cf34f1af673af2 P 51d0bc1740db4723a71f1d56b7e6e6b7: Bootstrap complete.
I20240503 08:11:36.702906 25411 ts_tablet_manager.cc:1367] T 2449909504504d1e84cf34f1af673af2 P 51d0bc1740db4723a71f1d56b7e6e6b7: Time spent bootstrapping tablet: real 0.036s user 0.028s sys 0.008s
I20240503 08:11:36.707429 25411 raft_consensus.cc:348] T 2449909504504d1e84cf34f1af673af2 P 51d0bc1740db4723a71f1d56b7e6e6b7 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } }
I20240503 08:11:36.708174 25411 raft_consensus.cc:729] T 2449909504504d1e84cf34f1af673af2 P 51d0bc1740db4723a71f1d56b7e6e6b7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51d0bc1740db4723a71f1d56b7e6e6b7, State: Initialized, Role: FOLLOWER
I20240503 08:11:36.708576 25411 consensus_queue.cc:260] T 2449909504504d1e84cf34f1af673af2 P 51d0bc1740db4723a71f1d56b7e6e6b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "51d0bc1740db4723a71f1d56b7e6e6b7" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 44977 } }
I20240503 08:11:36.709911 25411 ts_tablet_manager.cc:1398] T 2449909504504d1e84cf34f1af673af2 P 51d0bc1740db4723a71f1d56b7e6e6b7: Time spent starting tablet: real 0.007s user 0.006s sys 0.002s
I20240503 08:11:36.710935 23577 tablet_copy_service.cc:342] P 30d88bcd65b347dabcbac54465e52b21: Request end of tablet copy session 51d0bc1740db4723a71f1d56b7e6e6b7-2449909504504d1e84cf34f1af673af2 received from {username='slave'} at 127.20.230.68:54761
I20240503 08:11:36.711107 23577 tablet_copy_service.cc:433] P 30d88bcd65b347dabcbac54465e52b21: ending tablet copy session 51d0bc1740db4723a71f1d56b7e6e6b7-2449909504504d1e84cf34f1af673af2 on tablet 2449909504504d1e84cf34f1af673af2 with peer 51d0bc1740db4723a71f1d56b7e6e6b7
I20240503 08:11:37.160884 24792 raft_consensus.cc:1203] T 2449909504504d1e84cf34f1af673af2 P 51d0bc1740db4723a71f1d56b7e6e6b7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:11:37.162464 25421 raft_consensus.cc:2932] T 2449909504504d1e84cf34f1af673af2 P 30d88bcd65b347dabcbac54465e52b21 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 51d0bc1740db4723a71f1d56b7e6e6b7 (127.20.230.68) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "51d0bc1740db4723a71f1d56b7e6e6b7" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 44977 } } }
I20240503 08:11:37.163300 24792 raft_consensus.cc:2932] T 2449909504504d1e84cf34f1af673af2 P 51d0bc1740db4723a71f1d56b7e6e6b7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 51d0bc1740db4723a71f1d56b7e6e6b7 (127.20.230.68) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "51d0bc1740db4723a71f1d56b7e6e6b7" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 44977 } } }
I20240503 08:11:37.166375 25421 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:39672 (request call id 0) took 1098 ms. Trace:
I20240503 08:11:37.166584 25421 rpcz_store.cc:276] 0503 08:11:36.067823 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 08:11:36.067895 (+ 72us) service_pool.cc:225] Handling call
0503 08:11:36.070116 (+ 2221us) log.cc:842] Serialized 260 byte log entry
0503 08:11:37.166329 (+1096213us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":72,"raft.queue_time_us":352,"raft.run_cpu_time_us":526,"raft.run_wall_time_us":526,"spinlock_wait_cycles":215680,"thread_start_us":484,"threads_started":2,"wal-append.queue_time_us":365,"wal-append.run_cpu_time_us":590,"wal-append.run_wall_time_us":1000658}
I20240503 08:11:37.168387 23280 catalog_manager.cc:5484] T 2449909504504d1e84cf34f1af673af2 P 51d0bc1740db4723a71f1d56b7e6e6b7 reported cstate change: config changed from index -1 to 4, VOTER 51d0bc1740db4723a71f1d56b7e6e6b7 (127.20.230.68) added. New cstate: current_term: 1 leader_uuid: "30d88bcd65b347dabcbac54465e52b21" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "51d0bc1740db4723a71f1d56b7e6e6b7" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 44977 } } }
I20240503 08:11:37.196583 23557 consensus_queue.cc:237] T 8c6217accd494f5182a862c31013b76e P 30d88bcd65b347dabcbac54465e52b21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "7823cb7f30b045209bc5ccfb110bf199" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 38073 } }
W20240503 08:11:37.203765 23490 consensus_peers.cc:485] T 8c6217accd494f5182a862c31013b76e P 30d88bcd65b347dabcbac54465e52b21 -> Peer 7823cb7f30b045209bc5ccfb110bf199 (127.20.230.71:38073): Couldn't send request to peer 7823cb7f30b045209bc5ccfb110bf199. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c6217accd494f5182a862c31013b76e. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:37.631135 25448 ts_tablet_manager.cc:902] T 8c6217accd494f5182a862c31013b76e P 7823cb7f30b045209bc5ccfb110bf199: Initiating tablet copy from peer 30d88bcd65b347dabcbac54465e52b21 (127.20.230.66:33351)
I20240503 08:11:37.632756 25448 tablet_copy_client.cc:308] T 8c6217accd494f5182a862c31013b76e P 7823cb7f30b045209bc5ccfb110bf199: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.66:33351
I20240503 08:11:37.639605 23577 tablet_copy_service.cc:140] P 30d88bcd65b347dabcbac54465e52b21: Received BeginTabletCopySession request for tablet 8c6217accd494f5182a862c31013b76e from peer 7823cb7f30b045209bc5ccfb110bf199 ({username='slave'} at 127.20.230.71:44127)
I20240503 08:11:37.639778 23577 tablet_copy_service.cc:161] P 30d88bcd65b347dabcbac54465e52b21: Beginning new tablet copy session on tablet 8c6217accd494f5182a862c31013b76e from peer 7823cb7f30b045209bc5ccfb110bf199 at {username='slave'} at 127.20.230.71:44127: session id = 7823cb7f30b045209bc5ccfb110bf199-8c6217accd494f5182a862c31013b76e
I20240503 08:11:37.641881 23577 tablet_copy_source_session.cc:205] T 8c6217accd494f5182a862c31013b76e P 30d88bcd65b347dabcbac54465e52b21: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:37.646139 25448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c6217accd494f5182a862c31013b76e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:37.655771 25448 tablet_copy_client.cc:783] T 8c6217accd494f5182a862c31013b76e P 7823cb7f30b045209bc5ccfb110bf199: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:37.656059 25448 tablet_copy_client.cc:647] T 8c6217accd494f5182a862c31013b76e P 7823cb7f30b045209bc5ccfb110bf199: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:37.658010 25448 tablet_copy_client.cc:523] T 8c6217accd494f5182a862c31013b76e P 7823cb7f30b045209bc5ccfb110bf199: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:37.661724 25448 tablet_bootstrap.cc:492] T 8c6217accd494f5182a862c31013b76e P 7823cb7f30b045209bc5ccfb110bf199: Bootstrap starting.
I20240503 08:11:37.670435 25448 log.cc:824] T 8c6217accd494f5182a862c31013b76e P 7823cb7f30b045209bc5ccfb110bf199: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:37.699545 25448 tablet_bootstrap.cc:492] T 8c6217accd494f5182a862c31013b76e P 7823cb7f30b045209bc5ccfb110bf199: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:11:37.700055 25448 tablet_bootstrap.cc:492] T 8c6217accd494f5182a862c31013b76e P 7823cb7f30b045209bc5ccfb110bf199: Bootstrap complete.
I20240503 08:11:37.700451 25448 ts_tablet_manager.cc:1367] T 8c6217accd494f5182a862c31013b76e P 7823cb7f30b045209bc5ccfb110bf199: Time spent bootstrapping tablet: real 0.039s user 0.037s sys 0.003s
I20240503 08:11:37.705415 25448 raft_consensus.cc:348] T 8c6217accd494f5182a862c31013b76e P 7823cb7f30b045209bc5ccfb110bf199 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } }
I20240503 08:11:37.706431 25448 raft_consensus.cc:729] T 8c6217accd494f5182a862c31013b76e P 7823cb7f30b045209bc5ccfb110bf199 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7823cb7f30b045209bc5ccfb110bf199, State: Initialized, Role: FOLLOWER
I20240503 08:11:37.706940 25448 consensus_queue.cc:260] T 8c6217accd494f5182a862c31013b76e P 7823cb7f30b045209bc5ccfb110bf199 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "7823cb7f30b045209bc5ccfb110bf199" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 38073 } }
I20240503 08:11:37.708525 25448 ts_tablet_manager.cc:1398] T 8c6217accd494f5182a862c31013b76e P 7823cb7f30b045209bc5ccfb110bf199: Time spent starting tablet: real 0.008s user 0.005s sys 0.004s
I20240503 08:11:37.709539 23577 tablet_copy_service.cc:342] P 30d88bcd65b347dabcbac54465e52b21: Request end of tablet copy session 7823cb7f30b045209bc5ccfb110bf199-8c6217accd494f5182a862c31013b76e received from {username='slave'} at 127.20.230.71:44127
I20240503 08:11:37.709678 23577 tablet_copy_service.cc:433] P 30d88bcd65b347dabcbac54465e52b21: ending tablet copy session 7823cb7f30b045209bc5ccfb110bf199-8c6217accd494f5182a862c31013b76e on tablet 8c6217accd494f5182a862c31013b76e with peer 7823cb7f30b045209bc5ccfb110bf199
I20240503 08:11:38.063223 25193 raft_consensus.cc:1203] T 8c6217accd494f5182a862c31013b76e P 7823cb7f30b045209bc5ccfb110bf199 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:11:38.064720 25456 raft_consensus.cc:2932] T 8c6217accd494f5182a862c31013b76e P 30d88bcd65b347dabcbac54465e52b21 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7823cb7f30b045209bc5ccfb110bf199 (127.20.230.71) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "7823cb7f30b045209bc5ccfb110bf199" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 38073 } } }
I20240503 08:11:38.065486 25193 raft_consensus.cc:2932] T 8c6217accd494f5182a862c31013b76e P 7823cb7f30b045209bc5ccfb110bf199 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7823cb7f30b045209bc5ccfb110bf199 (127.20.230.71) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "7823cb7f30b045209bc5ccfb110bf199" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 38073 } } }
I20240503 08:11:38.069384 23284 catalog_manager.cc:5484] T 8c6217accd494f5182a862c31013b76e P 30d88bcd65b347dabcbac54465e52b21 reported cstate change: config changed from index -1 to 4, VOTER 7823cb7f30b045209bc5ccfb110bf199 (127.20.230.71) added. New cstate: current_term: 1 leader_uuid: "30d88bcd65b347dabcbac54465e52b21" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "7823cb7f30b045209bc5ccfb110bf199" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 38073 } } }
I20240503 08:11:38.095369 23427 consensus_queue.cc:237] T 29dea2294fd74738a55ac1bd5b88121b P 76e30a81c8b8451898078e9636a1104b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "c82f2e5e9e13405d9328620179077098" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 36811 } }
W20240503 08:11:38.102488 23363 consensus_peers.cc:485] T 29dea2294fd74738a55ac1bd5b88121b P 76e30a81c8b8451898078e9636a1104b -> Peer c82f2e5e9e13405d9328620179077098 (127.20.230.70:36811): Couldn't send request to peer c82f2e5e9e13405d9328620179077098. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 29dea2294fd74738a55ac1bd5b88121b. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:38.533149 25481 ts_tablet_manager.cc:902] T 29dea2294fd74738a55ac1bd5b88121b P c82f2e5e9e13405d9328620179077098: Initiating tablet copy from peer 76e30a81c8b8451898078e9636a1104b (127.20.230.65:37573)
I20240503 08:11:38.535239 25481 tablet_copy_client.cc:308] T 29dea2294fd74738a55ac1bd5b88121b P c82f2e5e9e13405d9328620179077098: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.65:37573
I20240503 08:11:38.540966 23447 tablet_copy_service.cc:140] P 76e30a81c8b8451898078e9636a1104b: Received BeginTabletCopySession request for tablet 29dea2294fd74738a55ac1bd5b88121b from peer c82f2e5e9e13405d9328620179077098 ({username='slave'} at 127.20.230.70:56587)
I20240503 08:11:38.541121 23447 tablet_copy_service.cc:161] P 76e30a81c8b8451898078e9636a1104b: Beginning new tablet copy session on tablet 29dea2294fd74738a55ac1bd5b88121b from peer c82f2e5e9e13405d9328620179077098 at {username='slave'} at 127.20.230.70:56587: session id = c82f2e5e9e13405d9328620179077098-29dea2294fd74738a55ac1bd5b88121b
I20240503 08:11:38.543195 23447 tablet_copy_source_session.cc:205] T 29dea2294fd74738a55ac1bd5b88121b P 76e30a81c8b8451898078e9636a1104b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:38.545327 25481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29dea2294fd74738a55ac1bd5b88121b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:38.550750 25481 tablet_copy_client.cc:783] T 29dea2294fd74738a55ac1bd5b88121b P c82f2e5e9e13405d9328620179077098: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:38.550994 25481 tablet_copy_client.cc:647] T 29dea2294fd74738a55ac1bd5b88121b P c82f2e5e9e13405d9328620179077098: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:38.552665 25481 tablet_copy_client.cc:523] T 29dea2294fd74738a55ac1bd5b88121b P c82f2e5e9e13405d9328620179077098: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:38.554950 25481 tablet_bootstrap.cc:492] T 29dea2294fd74738a55ac1bd5b88121b P c82f2e5e9e13405d9328620179077098: Bootstrap starting.
I20240503 08:11:38.587762 25481 tablet_bootstrap.cc:492] T 29dea2294fd74738a55ac1bd5b88121b P c82f2e5e9e13405d9328620179077098: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:11:38.588150 25481 tablet_bootstrap.cc:492] T 29dea2294fd74738a55ac1bd5b88121b P c82f2e5e9e13405d9328620179077098: Bootstrap complete.
I20240503 08:11:38.588382 25481 ts_tablet_manager.cc:1367] T 29dea2294fd74738a55ac1bd5b88121b P c82f2e5e9e13405d9328620179077098: Time spent bootstrapping tablet: real 0.033s user 0.026s sys 0.005s
I20240503 08:11:38.589102 25481 raft_consensus.cc:348] T 29dea2294fd74738a55ac1bd5b88121b P c82f2e5e9e13405d9328620179077098 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } }
I20240503 08:11:38.589491 25481 raft_consensus.cc:729] T 29dea2294fd74738a55ac1bd5b88121b P c82f2e5e9e13405d9328620179077098 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c82f2e5e9e13405d9328620179077098, State: Initialized, Role: FOLLOWER
I20240503 08:11:38.589653 25481 consensus_queue.cc:260] T 29dea2294fd74738a55ac1bd5b88121b P c82f2e5e9e13405d9328620179077098 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "c82f2e5e9e13405d9328620179077098" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 36811 } }
I20240503 08:11:38.590348 25481 ts_tablet_manager.cc:1398] T 29dea2294fd74738a55ac1bd5b88121b P c82f2e5e9e13405d9328620179077098: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 08:11:38.591164 23447 tablet_copy_service.cc:342] P 76e30a81c8b8451898078e9636a1104b: Request end of tablet copy session c82f2e5e9e13405d9328620179077098-29dea2294fd74738a55ac1bd5b88121b received from {username='slave'} at 127.20.230.70:56587
I20240503 08:11:38.591284 23447 tablet_copy_service.cc:433] P 76e30a81c8b8451898078e9636a1104b: ending tablet copy session c82f2e5e9e13405d9328620179077098-29dea2294fd74738a55ac1bd5b88121b on tablet 29dea2294fd74738a55ac1bd5b88121b with peer c82f2e5e9e13405d9328620179077098
I20240503 08:11:39.064150 25053 raft_consensus.cc:1203] T 29dea2294fd74738a55ac1bd5b88121b P c82f2e5e9e13405d9328620179077098 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:11:39.065603 25489 raft_consensus.cc:2932] T 29dea2294fd74738a55ac1bd5b88121b P 76e30a81c8b8451898078e9636a1104b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c82f2e5e9e13405d9328620179077098 (127.20.230.70) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "c82f2e5e9e13405d9328620179077098" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 36811 } } }
I20240503 08:11:39.066254 25053 raft_consensus.cc:2932] T 29dea2294fd74738a55ac1bd5b88121b P c82f2e5e9e13405d9328620179077098 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c82f2e5e9e13405d9328620179077098 (127.20.230.70) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "c82f2e5e9e13405d9328620179077098" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 36811 } } }
I20240503 08:11:39.069883 23284 catalog_manager.cc:5484] T 29dea2294fd74738a55ac1bd5b88121b P 76e30a81c8b8451898078e9636a1104b reported cstate change: config changed from index -1 to 4, VOTER c82f2e5e9e13405d9328620179077098 (127.20.230.70) added. New cstate: current_term: 1 leader_uuid: "76e30a81c8b8451898078e9636a1104b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "c82f2e5e9e13405d9328620179077098" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 36811 } } }
I20240503 08:11:39.101578 23427 consensus_queue.cc:237] T 78b4f0eae3d14b97b8f74cbc24389044 P 76e30a81c8b8451898078e9636a1104b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "509c32504f544f8b90ebf66b6dd951f7" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 41435 } }
W20240503 08:11:39.103325 23362 consensus_peers.cc:485] T 78b4f0eae3d14b97b8f74cbc24389044 P 76e30a81c8b8451898078e9636a1104b -> Peer 509c32504f544f8b90ebf66b6dd951f7 (127.20.230.69:41435): Couldn't send request to peer 509c32504f544f8b90ebf66b6dd951f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 78b4f0eae3d14b97b8f74cbc24389044. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:39.623383 25512 ts_tablet_manager.cc:902] T 78b4f0eae3d14b97b8f74cbc24389044 P 509c32504f544f8b90ebf66b6dd951f7: Initiating tablet copy from peer 76e30a81c8b8451898078e9636a1104b (127.20.230.65:37573)
I20240503 08:11:39.624518 25512 tablet_copy_client.cc:308] T 78b4f0eae3d14b97b8f74cbc24389044 P 509c32504f544f8b90ebf66b6dd951f7: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.65:37573
I20240503 08:11:39.625177 23447 tablet_copy_service.cc:140] P 76e30a81c8b8451898078e9636a1104b: Received BeginTabletCopySession request for tablet 78b4f0eae3d14b97b8f74cbc24389044 from peer 509c32504f544f8b90ebf66b6dd951f7 ({username='slave'} at 127.20.230.69:47793)
I20240503 08:11:39.625366 23447 tablet_copy_service.cc:161] P 76e30a81c8b8451898078e9636a1104b: Beginning new tablet copy session on tablet 78b4f0eae3d14b97b8f74cbc24389044 from peer 509c32504f544f8b90ebf66b6dd951f7 at {username='slave'} at 127.20.230.69:47793: session id = 509c32504f544f8b90ebf66b6dd951f7-78b4f0eae3d14b97b8f74cbc24389044
I20240503 08:11:39.627305 23447 tablet_copy_source_session.cc:205] T 78b4f0eae3d14b97b8f74cbc24389044 P 76e30a81c8b8451898078e9636a1104b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:39.628499 25512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78b4f0eae3d14b97b8f74cbc24389044. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:39.632771 25512 tablet_copy_client.cc:783] T 78b4f0eae3d14b97b8f74cbc24389044 P 509c32504f544f8b90ebf66b6dd951f7: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:39.632994 25512 tablet_copy_client.cc:647] T 78b4f0eae3d14b97b8f74cbc24389044 P 509c32504f544f8b90ebf66b6dd951f7: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:39.634680 25512 tablet_copy_client.cc:523] T 78b4f0eae3d14b97b8f74cbc24389044 P 509c32504f544f8b90ebf66b6dd951f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:39.637037 25512 tablet_bootstrap.cc:492] T 78b4f0eae3d14b97b8f74cbc24389044 P 509c32504f544f8b90ebf66b6dd951f7: Bootstrap starting.
I20240503 08:11:39.666277 25512 tablet_bootstrap.cc:492] T 78b4f0eae3d14b97b8f74cbc24389044 P 509c32504f544f8b90ebf66b6dd951f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:11:39.666666 25512 tablet_bootstrap.cc:492] T 78b4f0eae3d14b97b8f74cbc24389044 P 509c32504f544f8b90ebf66b6dd951f7: Bootstrap complete.
I20240503 08:11:39.666920 25512 ts_tablet_manager.cc:1367] T 78b4f0eae3d14b97b8f74cbc24389044 P 509c32504f544f8b90ebf66b6dd951f7: Time spent bootstrapping tablet: real 0.030s user 0.027s sys 0.004s
I20240503 08:11:39.667711 25512 raft_consensus.cc:348] T 78b4f0eae3d14b97b8f74cbc24389044 P 509c32504f544f8b90ebf66b6dd951f7 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } }
I20240503 08:11:39.668068 25512 raft_consensus.cc:729] T 78b4f0eae3d14b97b8f74cbc24389044 P 509c32504f544f8b90ebf66b6dd951f7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 509c32504f544f8b90ebf66b6dd951f7, State: Initialized, Role: FOLLOWER
I20240503 08:11:39.668246 25512 consensus_queue.cc:260] T 78b4f0eae3d14b97b8f74cbc24389044 P 509c32504f544f8b90ebf66b6dd951f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "509c32504f544f8b90ebf66b6dd951f7" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 41435 } }
I20240503 08:11:39.668956 25512 ts_tablet_manager.cc:1398] T 78b4f0eae3d14b97b8f74cbc24389044 P 509c32504f544f8b90ebf66b6dd951f7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 08:11:39.669867 23447 tablet_copy_service.cc:342] P 76e30a81c8b8451898078e9636a1104b: Request end of tablet copy session 509c32504f544f8b90ebf66b6dd951f7-78b4f0eae3d14b97b8f74cbc24389044 received from {username='slave'} at 127.20.230.69:47793
I20240503 08:11:39.670012 23447 tablet_copy_service.cc:433] P 76e30a81c8b8451898078e9636a1104b: ending tablet copy session 509c32504f544f8b90ebf66b6dd951f7-78b4f0eae3d14b97b8f74cbc24389044 on tablet 78b4f0eae3d14b97b8f74cbc24389044 with peer 509c32504f544f8b90ebf66b6dd951f7
I20240503 08:11:40.075034 24922 raft_consensus.cc:1203] T 78b4f0eae3d14b97b8f74cbc24389044 P 509c32504f544f8b90ebf66b6dd951f7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:11:40.076292 25519 raft_consensus.cc:2932] T 78b4f0eae3d14b97b8f74cbc24389044 P 76e30a81c8b8451898078e9636a1104b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 509c32504f544f8b90ebf66b6dd951f7 (127.20.230.69) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "509c32504f544f8b90ebf66b6dd951f7" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 41435 } } }
I20240503 08:11:40.076929 24922 raft_consensus.cc:2932] T 78b4f0eae3d14b97b8f74cbc24389044 P 509c32504f544f8b90ebf66b6dd951f7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 509c32504f544f8b90ebf66b6dd951f7 (127.20.230.69) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "509c32504f544f8b90ebf66b6dd951f7" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 41435 } } }
I20240503 08:11:40.086001 23286 catalog_manager.cc:5484] T 78b4f0eae3d14b97b8f74cbc24389044 P 76e30a81c8b8451898078e9636a1104b reported cstate change: config changed from index -1 to 4, VOTER 509c32504f544f8b90ebf66b6dd951f7 (127.20.230.69) added. New cstate: current_term: 1 leader_uuid: "76e30a81c8b8451898078e9636a1104b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "509c32504f544f8b90ebf66b6dd951f7" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 41435 } } }
I20240503 08:11:40.114929 23427 consensus_queue.cc:237] T 98dfd1367b224981ab0fd74678bb5ac2 P 76e30a81c8b8451898078e9636a1104b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "5ea58b5961b54598878bad3f5660913c" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 37877 } }
W20240503 08:11:40.117110 23361 consensus_peers.cc:485] T 98dfd1367b224981ab0fd74678bb5ac2 P 76e30a81c8b8451898078e9636a1104b -> Peer 5ea58b5961b54598878bad3f5660913c (127.20.230.67:37877): Couldn't send request to peer 5ea58b5961b54598878bad3f5660913c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 98dfd1367b224981ab0fd74678bb5ac2. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:40.705687 25543 ts_tablet_manager.cc:902] T 98dfd1367b224981ab0fd74678bb5ac2 P 5ea58b5961b54598878bad3f5660913c: Initiating tablet copy from peer 76e30a81c8b8451898078e9636a1104b (127.20.230.65:37573)
I20240503 08:11:40.706779 25543 tablet_copy_client.cc:308] T 98dfd1367b224981ab0fd74678bb5ac2 P 5ea58b5961b54598878bad3f5660913c: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.65:37573
I20240503 08:11:40.707445 23447 tablet_copy_service.cc:140] P 76e30a81c8b8451898078e9636a1104b: Received BeginTabletCopySession request for tablet 98dfd1367b224981ab0fd74678bb5ac2 from peer 5ea58b5961b54598878bad3f5660913c ({username='slave'} at 127.20.230.67:38301)
I20240503 08:11:40.707597 23447 tablet_copy_service.cc:161] P 76e30a81c8b8451898078e9636a1104b: Beginning new tablet copy session on tablet 98dfd1367b224981ab0fd74678bb5ac2 from peer 5ea58b5961b54598878bad3f5660913c at {username='slave'} at 127.20.230.67:38301: session id = 5ea58b5961b54598878bad3f5660913c-98dfd1367b224981ab0fd74678bb5ac2
I20240503 08:11:40.709635 23447 tablet_copy_source_session.cc:205] T 98dfd1367b224981ab0fd74678bb5ac2 P 76e30a81c8b8451898078e9636a1104b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:40.710764 25543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98dfd1367b224981ab0fd74678bb5ac2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:40.715348 25543 tablet_copy_client.cc:783] T 98dfd1367b224981ab0fd74678bb5ac2 P 5ea58b5961b54598878bad3f5660913c: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:40.715576 25543 tablet_copy_client.cc:647] T 98dfd1367b224981ab0fd74678bb5ac2 P 5ea58b5961b54598878bad3f5660913c: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:40.717005 25543 tablet_copy_client.cc:523] T 98dfd1367b224981ab0fd74678bb5ac2 P 5ea58b5961b54598878bad3f5660913c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:40.719230 25543 tablet_bootstrap.cc:492] T 98dfd1367b224981ab0fd74678bb5ac2 P 5ea58b5961b54598878bad3f5660913c: Bootstrap starting.
I20240503 08:11:40.747931 25543 tablet_bootstrap.cc:492] T 98dfd1367b224981ab0fd74678bb5ac2 P 5ea58b5961b54598878bad3f5660913c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:11:40.748270 25543 tablet_bootstrap.cc:492] T 98dfd1367b224981ab0fd74678bb5ac2 P 5ea58b5961b54598878bad3f5660913c: Bootstrap complete.
I20240503 08:11:40.748499 25543 ts_tablet_manager.cc:1367] T 98dfd1367b224981ab0fd74678bb5ac2 P 5ea58b5961b54598878bad3f5660913c: Time spent bootstrapping tablet: real 0.029s user 0.024s sys 0.004s
I20240503 08:11:40.749230 25543 raft_consensus.cc:348] T 98dfd1367b224981ab0fd74678bb5ac2 P 5ea58b5961b54598878bad3f5660913c [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } }
I20240503 08:11:40.749570 25543 raft_consensus.cc:729] T 98dfd1367b224981ab0fd74678bb5ac2 P 5ea58b5961b54598878bad3f5660913c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ea58b5961b54598878bad3f5660913c, State: Initialized, Role: FOLLOWER
I20240503 08:11:40.749734 25543 consensus_queue.cc:260] T 98dfd1367b224981ab0fd74678bb5ac2 P 5ea58b5961b54598878bad3f5660913c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "5ea58b5961b54598878bad3f5660913c" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 37877 } }
I20240503 08:11:40.750399 25543 ts_tablet_manager.cc:1398] T 98dfd1367b224981ab0fd74678bb5ac2 P 5ea58b5961b54598878bad3f5660913c: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 08:11:40.751339 23447 tablet_copy_service.cc:342] P 76e30a81c8b8451898078e9636a1104b: Request end of tablet copy session 5ea58b5961b54598878bad3f5660913c-98dfd1367b224981ab0fd74678bb5ac2 received from {username='slave'} at 127.20.230.67:38301
I20240503 08:11:40.751502 23447 tablet_copy_service.cc:433] P 76e30a81c8b8451898078e9636a1104b: ending tablet copy session 5ea58b5961b54598878bad3f5660913c-98dfd1367b224981ab0fd74678bb5ac2 on tablet 98dfd1367b224981ab0fd74678bb5ac2 with peer 5ea58b5961b54598878bad3f5660913c
I20240503 08:11:41.278811 24661 raft_consensus.cc:1203] T 98dfd1367b224981ab0fd74678bb5ac2 P 5ea58b5961b54598878bad3f5660913c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:11:41.280231 25551 raft_consensus.cc:2932] T 98dfd1367b224981ab0fd74678bb5ac2 P 76e30a81c8b8451898078e9636a1104b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5ea58b5961b54598878bad3f5660913c (127.20.230.67) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "5ea58b5961b54598878bad3f5660913c" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 37877 } } }
I20240503 08:11:41.280834 24661 raft_consensus.cc:2932] T 98dfd1367b224981ab0fd74678bb5ac2 P 5ea58b5961b54598878bad3f5660913c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5ea58b5961b54598878bad3f5660913c (127.20.230.67) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "5ea58b5961b54598878bad3f5660913c" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 37877 } } }
I20240503 08:11:41.283284 25551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:38068 (request call id 0) took 1168 ms. Trace:
I20240503 08:11:41.283465 25551 rpcz_store.cc:276] 0503 08:11:40.114477 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 08:11:40.114538 (+ 61us) service_pool.cc:225] Handling call
0503 08:11:40.115413 (+ 875us) log.cc:842] Serialized 260 byte log entry
0503 08:11:41.283235 (+1167822us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":42,"raft.run_cpu_time_us":196,"raft.run_wall_time_us":196,"spinlock_wait_cycles":19840,"thread_start_us":621,"threads_started":1,"wal-append.queue_time_us":715,"wal-append.run_cpu_time_us":431,"wal-append.run_wall_time_us":1000492}
I20240503 08:11:41.285398 23286 catalog_manager.cc:5484] T 98dfd1367b224981ab0fd74678bb5ac2 P 76e30a81c8b8451898078e9636a1104b reported cstate change: config changed from index -1 to 4, VOTER 5ea58b5961b54598878bad3f5660913c (127.20.230.67) added. New cstate: current_term: 1 leader_uuid: "76e30a81c8b8451898078e9636a1104b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "5ea58b5961b54598878bad3f5660913c" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 37877 } } }
I20240503 08:11:41.321573 23557 consensus_queue.cc:237] T b2779eae1c08452ca7b5b0874bbac47e P 30d88bcd65b347dabcbac54465e52b21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "51d0bc1740db4723a71f1d56b7e6e6b7" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 44977 } }
W20240503 08:11:41.323460 23491 consensus_peers.cc:485] T b2779eae1c08452ca7b5b0874bbac47e P 30d88bcd65b347dabcbac54465e52b21 -> Peer 51d0bc1740db4723a71f1d56b7e6e6b7 (127.20.230.68:44977): Couldn't send request to peer 51d0bc1740db4723a71f1d56b7e6e6b7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b2779eae1c08452ca7b5b0874bbac47e. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:41.821971 25576 ts_tablet_manager.cc:902] T b2779eae1c08452ca7b5b0874bbac47e P 51d0bc1740db4723a71f1d56b7e6e6b7: Initiating tablet copy from peer 30d88bcd65b347dabcbac54465e52b21 (127.20.230.66:33351)
I20240503 08:11:41.822983 25576 tablet_copy_client.cc:308] T b2779eae1c08452ca7b5b0874bbac47e P 51d0bc1740db4723a71f1d56b7e6e6b7: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.66:33351
I20240503 08:11:41.823652 23577 tablet_copy_service.cc:140] P 30d88bcd65b347dabcbac54465e52b21: Received BeginTabletCopySession request for tablet b2779eae1c08452ca7b5b0874bbac47e from peer 51d0bc1740db4723a71f1d56b7e6e6b7 ({username='slave'} at 127.20.230.68:54761)
I20240503 08:11:41.823801 23577 tablet_copy_service.cc:161] P 30d88bcd65b347dabcbac54465e52b21: Beginning new tablet copy session on tablet b2779eae1c08452ca7b5b0874bbac47e from peer 51d0bc1740db4723a71f1d56b7e6e6b7 at {username='slave'} at 127.20.230.68:54761: session id = 51d0bc1740db4723a71f1d56b7e6e6b7-b2779eae1c08452ca7b5b0874bbac47e
I20240503 08:11:41.825659 23577 tablet_copy_source_session.cc:205] T b2779eae1c08452ca7b5b0874bbac47e P 30d88bcd65b347dabcbac54465e52b21: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:41.826754 25576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2779eae1c08452ca7b5b0874bbac47e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:41.830775 25576 tablet_copy_client.cc:783] T b2779eae1c08452ca7b5b0874bbac47e P 51d0bc1740db4723a71f1d56b7e6e6b7: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:41.830977 25576 tablet_copy_client.cc:647] T b2779eae1c08452ca7b5b0874bbac47e P 51d0bc1740db4723a71f1d56b7e6e6b7: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:41.832403 25576 tablet_copy_client.cc:523] T b2779eae1c08452ca7b5b0874bbac47e P 51d0bc1740db4723a71f1d56b7e6e6b7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:41.834887 25576 tablet_bootstrap.cc:492] T b2779eae1c08452ca7b5b0874bbac47e P 51d0bc1740db4723a71f1d56b7e6e6b7: Bootstrap starting.
I20240503 08:11:41.864403 25576 tablet_bootstrap.cc:492] T b2779eae1c08452ca7b5b0874bbac47e P 51d0bc1740db4723a71f1d56b7e6e6b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:11:41.864744 25576 tablet_bootstrap.cc:492] T b2779eae1c08452ca7b5b0874bbac47e P 51d0bc1740db4723a71f1d56b7e6e6b7: Bootstrap complete.
I20240503 08:11:41.864960 25576 ts_tablet_manager.cc:1367] T b2779eae1c08452ca7b5b0874bbac47e P 51d0bc1740db4723a71f1d56b7e6e6b7: Time spent bootstrapping tablet: real 0.030s user 0.019s sys 0.009s
I20240503 08:11:41.865691 25576 raft_consensus.cc:348] T b2779eae1c08452ca7b5b0874bbac47e P 51d0bc1740db4723a71f1d56b7e6e6b7 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } }
I20240503 08:11:41.866067 25576 raft_consensus.cc:729] T b2779eae1c08452ca7b5b0874bbac47e P 51d0bc1740db4723a71f1d56b7e6e6b7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51d0bc1740db4723a71f1d56b7e6e6b7, State: Initialized, Role: FOLLOWER
I20240503 08:11:41.866238 25576 consensus_queue.cc:260] T b2779eae1c08452ca7b5b0874bbac47e P 51d0bc1740db4723a71f1d56b7e6e6b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "51d0bc1740db4723a71f1d56b7e6e6b7" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 44977 } }
I20240503 08:11:41.866855 25576 ts_tablet_manager.cc:1398] T b2779eae1c08452ca7b5b0874bbac47e P 51d0bc1740db4723a71f1d56b7e6e6b7: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 08:11:41.867524 23577 tablet_copy_service.cc:342] P 30d88bcd65b347dabcbac54465e52b21: Request end of tablet copy session 51d0bc1740db4723a71f1d56b7e6e6b7-b2779eae1c08452ca7b5b0874bbac47e received from {username='slave'} at 127.20.230.68:54761
I20240503 08:11:41.867651 23577 tablet_copy_service.cc:433] P 30d88bcd65b347dabcbac54465e52b21: ending tablet copy session 51d0bc1740db4723a71f1d56b7e6e6b7-b2779eae1c08452ca7b5b0874bbac47e on tablet b2779eae1c08452ca7b5b0874bbac47e with peer 51d0bc1740db4723a71f1d56b7e6e6b7
I20240503 08:11:42.281793 24792 raft_consensus.cc:1203] T b2779eae1c08452ca7b5b0874bbac47e P 51d0bc1740db4723a71f1d56b7e6e6b7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:11:42.283068 25583 raft_consensus.cc:2932] T b2779eae1c08452ca7b5b0874bbac47e P 30d88bcd65b347dabcbac54465e52b21 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 51d0bc1740db4723a71f1d56b7e6e6b7 (127.20.230.68) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "51d0bc1740db4723a71f1d56b7e6e6b7" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 44977 } } }
I20240503 08:11:42.283730 24792 raft_consensus.cc:2932] T b2779eae1c08452ca7b5b0874bbac47e P 51d0bc1740db4723a71f1d56b7e6e6b7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 51d0bc1740db4723a71f1d56b7e6e6b7 (127.20.230.68) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "51d0bc1740db4723a71f1d56b7e6e6b7" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 44977 } } }
I20240503 08:11:42.289737 23280 catalog_manager.cc:5484] T b2779eae1c08452ca7b5b0874bbac47e P 30d88bcd65b347dabcbac54465e52b21 reported cstate change: config changed from index -1 to 4, VOTER 51d0bc1740db4723a71f1d56b7e6e6b7 (127.20.230.68) added. New cstate: current_term: 1 leader_uuid: "30d88bcd65b347dabcbac54465e52b21" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "51d0bc1740db4723a71f1d56b7e6e6b7" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 44977 } } }
I20240503 08:11:42.322170 23557 consensus_queue.cc:237] T 43fb21a411474f62afad731b47bad5a9 P 30d88bcd65b347dabcbac54465e52b21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "7823cb7f30b045209bc5ccfb110bf199" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 38073 } }
W20240503 08:11:42.323984 23490 consensus_peers.cc:485] T 43fb21a411474f62afad731b47bad5a9 P 30d88bcd65b347dabcbac54465e52b21 -> Peer 7823cb7f30b045209bc5ccfb110bf199 (127.20.230.71:38073): Couldn't send request to peer 7823cb7f30b045209bc5ccfb110bf199. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 43fb21a411474f62afad731b47bad5a9. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:42.819070 25606 ts_tablet_manager.cc:902] T 43fb21a411474f62afad731b47bad5a9 P 7823cb7f30b045209bc5ccfb110bf199: Initiating tablet copy from peer 30d88bcd65b347dabcbac54465e52b21 (127.20.230.66:33351)
I20240503 08:11:42.820068 25606 tablet_copy_client.cc:308] T 43fb21a411474f62afad731b47bad5a9 P 7823cb7f30b045209bc5ccfb110bf199: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.66:33351
I20240503 08:11:42.820678 23577 tablet_copy_service.cc:140] P 30d88bcd65b347dabcbac54465e52b21: Received BeginTabletCopySession request for tablet 43fb21a411474f62afad731b47bad5a9 from peer 7823cb7f30b045209bc5ccfb110bf199 ({username='slave'} at 127.20.230.71:44127)
I20240503 08:11:42.820812 23577 tablet_copy_service.cc:161] P 30d88bcd65b347dabcbac54465e52b21: Beginning new tablet copy session on tablet 43fb21a411474f62afad731b47bad5a9 from peer 7823cb7f30b045209bc5ccfb110bf199 at {username='slave'} at 127.20.230.71:44127: session id = 7823cb7f30b045209bc5ccfb110bf199-43fb21a411474f62afad731b47bad5a9
I20240503 08:11:42.822829 23577 tablet_copy_source_session.cc:205] T 43fb21a411474f62afad731b47bad5a9 P 30d88bcd65b347dabcbac54465e52b21: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:42.823887 25606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43fb21a411474f62afad731b47bad5a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:42.827873 25606 tablet_copy_client.cc:783] T 43fb21a411474f62afad731b47bad5a9 P 7823cb7f30b045209bc5ccfb110bf199: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:42.828068 25606 tablet_copy_client.cc:647] T 43fb21a411474f62afad731b47bad5a9 P 7823cb7f30b045209bc5ccfb110bf199: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:42.829542 25606 tablet_copy_client.cc:523] T 43fb21a411474f62afad731b47bad5a9 P 7823cb7f30b045209bc5ccfb110bf199: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:42.831696 25606 tablet_bootstrap.cc:492] T 43fb21a411474f62afad731b47bad5a9 P 7823cb7f30b045209bc5ccfb110bf199: Bootstrap starting.
I20240503 08:11:42.861196 25606 tablet_bootstrap.cc:492] T 43fb21a411474f62afad731b47bad5a9 P 7823cb7f30b045209bc5ccfb110bf199: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:11:42.861550 25606 tablet_bootstrap.cc:492] T 43fb21a411474f62afad731b47bad5a9 P 7823cb7f30b045209bc5ccfb110bf199: Bootstrap complete.
I20240503 08:11:42.861814 25606 ts_tablet_manager.cc:1367] T 43fb21a411474f62afad731b47bad5a9 P 7823cb7f30b045209bc5ccfb110bf199: Time spent bootstrapping tablet: real 0.030s user 0.019s sys 0.009s
I20240503 08:11:42.862622 25606 raft_consensus.cc:348] T 43fb21a411474f62afad731b47bad5a9 P 7823cb7f30b045209bc5ccfb110bf199 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } }
I20240503 08:11:42.862963 25606 raft_consensus.cc:729] T 43fb21a411474f62afad731b47bad5a9 P 7823cb7f30b045209bc5ccfb110bf199 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7823cb7f30b045209bc5ccfb110bf199, State: Initialized, Role: FOLLOWER
I20240503 08:11:42.863130 25606 consensus_queue.cc:260] T 43fb21a411474f62afad731b47bad5a9 P 7823cb7f30b045209bc5ccfb110bf199 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "7823cb7f30b045209bc5ccfb110bf199" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 38073 } }
I20240503 08:11:42.864243 25606 ts_tablet_manager.cc:1398] T 43fb21a411474f62afad731b47bad5a9 P 7823cb7f30b045209bc5ccfb110bf199: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 08:11:42.865128 23577 tablet_copy_service.cc:342] P 30d88bcd65b347dabcbac54465e52b21: Request end of tablet copy session 7823cb7f30b045209bc5ccfb110bf199-43fb21a411474f62afad731b47bad5a9 received from {username='slave'} at 127.20.230.71:44127
I20240503 08:11:42.865294 23577 tablet_copy_service.cc:433] P 30d88bcd65b347dabcbac54465e52b21: ending tablet copy session 7823cb7f30b045209bc5ccfb110bf199-43fb21a411474f62afad731b47bad5a9 on tablet 43fb21a411474f62afad731b47bad5a9 with peer 7823cb7f30b045209bc5ccfb110bf199
I20240503 08:11:43.257694 25193 raft_consensus.cc:1203] T 43fb21a411474f62afad731b47bad5a9 P 7823cb7f30b045209bc5ccfb110bf199 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:11:43.258997 25613 raft_consensus.cc:2932] T 43fb21a411474f62afad731b47bad5a9 P 30d88bcd65b347dabcbac54465e52b21 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7823cb7f30b045209bc5ccfb110bf199 (127.20.230.71) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "7823cb7f30b045209bc5ccfb110bf199" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 38073 } } }
I20240503 08:11:43.259567 25193 raft_consensus.cc:2932] T 43fb21a411474f62afad731b47bad5a9 P 7823cb7f30b045209bc5ccfb110bf199 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7823cb7f30b045209bc5ccfb110bf199 (127.20.230.71) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "7823cb7f30b045209bc5ccfb110bf199" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 38073 } } }
I20240503 08:11:43.263412 23286 catalog_manager.cc:5484] T 43fb21a411474f62afad731b47bad5a9 P 30d88bcd65b347dabcbac54465e52b21 reported cstate change: config changed from index -1 to 4, VOTER 7823cb7f30b045209bc5ccfb110bf199 (127.20.230.71) added. New cstate: current_term: 1 leader_uuid: "30d88bcd65b347dabcbac54465e52b21" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "7823cb7f30b045209bc5ccfb110bf199" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 38073 } } }
I20240503 08:11:43.362015 23537 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:11:43.394596 24772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:43.397526 24902 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:43.409813 25173 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:43.411438 24641 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:43.418764 25033 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:43.435824 23407 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:11:43.492036 23557 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "43fb21a411474f62afad731b47bad5a9"
dest_uuid: "30d88bcd65b347dabcbac54465e52b21"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39754
I20240503 08:11:43.492295 23557 raft_consensus.cc:595] T 43fb21a411474f62afad731b47bad5a9 P 30d88bcd65b347dabcbac54465e52b21 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:11:43.582592 23537 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:11:43.598120 24902 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:43.609601 24772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:43.612430 24641 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:43.627939 23407 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:11:43.636340 25173 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:43.639923 25456 raft_consensus.cc:979] T 43fb21a411474f62afad731b47bad5a9 P 30d88bcd65b347dabcbac54465e52b21: : Instructing follower 7823cb7f30b045209bc5ccfb110bf199 to start an election
I20240503 08:11:43.640108 25456 raft_consensus.cc:1067] T 43fb21a411474f62afad731b47bad5a9 P 30d88bcd65b347dabcbac54465e52b21 [term 1 LEADER]: Signalling peer 7823cb7f30b045209bc5ccfb110bf199 to start an election
I20240503 08:11:43.641369 25193 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "43fb21a411474f62afad731b47bad5a9"
dest_uuid: "7823cb7f30b045209bc5ccfb110bf199"
from {username='slave'} at 127.20.230.66:42981
I20240503 08:11:43.641664 25193 raft_consensus.cc:482] T 43fb21a411474f62afad731b47bad5a9 P 7823cb7f30b045209bc5ccfb110bf199 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:11:43.641865 25193 raft_consensus.cc:3037] T 43fb21a411474f62afad731b47bad5a9 P 7823cb7f30b045209bc5ccfb110bf199 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:43.644912 25193 raft_consensus.cc:504] T 43fb21a411474f62afad731b47bad5a9 P 7823cb7f30b045209bc5ccfb110bf199 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "7823cb7f30b045209bc5ccfb110bf199" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 38073 } }
I20240503 08:11:43.650210 23557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43fb21a411474f62afad731b47bad5a9" candidate_uuid: "7823cb7f30b045209bc5ccfb110bf199" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "30d88bcd65b347dabcbac54465e52b21"
I20240503 08:11:43.650471 25033 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:43.650498 23557 raft_consensus.cc:3032] T 43fb21a411474f62afad731b47bad5a9 P 30d88bcd65b347dabcbac54465e52b21 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:11:43.650596 23557 raft_consensus.cc:729] T 43fb21a411474f62afad731b47bad5a9 P 30d88bcd65b347dabcbac54465e52b21 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30d88bcd65b347dabcbac54465e52b21, State: Running, Role: LEADER
I20240503 08:11:43.650856 23557 consensus_queue.cc:260] T 43fb21a411474f62afad731b47bad5a9 P 30d88bcd65b347dabcbac54465e52b21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "7823cb7f30b045209bc5ccfb110bf199" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 38073 } }
I20240503 08:11:43.651096 23557 raft_consensus.cc:3037] T 43fb21a411474f62afad731b47bad5a9 P 30d88bcd65b347dabcbac54465e52b21 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:43.654079 25193 leader_election.cc:288] T 43fb21a411474f62afad731b47bad5a9 P 7823cb7f30b045209bc5ccfb110bf199 [CANDIDATE]: Term 2 election: Requested vote from peers 30d88bcd65b347dabcbac54465e52b21 (127.20.230.66:33351)
I20240503 08:11:43.655107 23557 raft_consensus.cc:2454] T 43fb21a411474f62afad731b47bad5a9 P 30d88bcd65b347dabcbac54465e52b21 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7823cb7f30b045209bc5ccfb110bf199 in term 2.
I20240503 08:11:43.655761 25129 leader_election.cc:302] T 43fb21a411474f62afad731b47bad5a9 P 7823cb7f30b045209bc5ccfb110bf199 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 30d88bcd65b347dabcbac54465e52b21, 7823cb7f30b045209bc5ccfb110bf199; no voters:
I20240503 08:11:43.656354 25612 raft_consensus.cc:2781] T 43fb21a411474f62afad731b47bad5a9 P 7823cb7f30b045209bc5ccfb110bf199 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:11:43.674161 25612 raft_consensus.cc:686] T 43fb21a411474f62afad731b47bad5a9 P 7823cb7f30b045209bc5ccfb110bf199 [term 2 LEADER]: Becoming Leader. State: Replica: 7823cb7f30b045209bc5ccfb110bf199, State: Running, Role: LEADER
I20240503 08:11:43.675069 25612 consensus_queue.cc:237] T 43fb21a411474f62afad731b47bad5a9 P 7823cb7f30b045209bc5ccfb110bf199 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "7823cb7f30b045209bc5ccfb110bf199" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 38073 } }
I20240503 08:11:43.679515 23280 catalog_manager.cc:5484] T 43fb21a411474f62afad731b47bad5a9 P 7823cb7f30b045209bc5ccfb110bf199 reported cstate change: term changed from 1 to 2, leader changed from 30d88bcd65b347dabcbac54465e52b21 (127.20.230.66) to 7823cb7f30b045209bc5ccfb110bf199 (127.20.230.71). New cstate: current_term: 2 leader_uuid: "7823cb7f30b045209bc5ccfb110bf199" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "7823cb7f30b045209bc5ccfb110bf199" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 38073 } } }
I20240503 08:11:43.810271 23537 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:11:43.823547 24641 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:43.824128 24902 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:43.829357 24772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:43.857090 25173 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:43.862224 23407 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:11:43.882802 25033 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:43.933594 23557 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b2779eae1c08452ca7b5b0874bbac47e"
dest_uuid: "30d88bcd65b347dabcbac54465e52b21"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39788
I20240503 08:11:43.933831 23557 raft_consensus.cc:595] T b2779eae1c08452ca7b5b0874bbac47e P 30d88bcd65b347dabcbac54465e52b21 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:11:44.029314 23537 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:11:44.046105 24902 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:44.051067 25173 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:44.068639 24641 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:44.071238 23407 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:11:44.072574 24772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:44.095794 25033 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:44.166483 23557 raft_consensus.cc:1261] T 43fb21a411474f62afad731b47bad5a9 P 30d88bcd65b347dabcbac54465e52b21 [term 2 FOLLOWER]: Refusing update from remote peer 7823cb7f30b045209bc5ccfb110bf199: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 08:11:44.167270 25612 consensus_queue.cc:1035] T 43fb21a411474f62afad731b47bad5a9 P 7823cb7f30b045209bc5ccfb110bf199 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:11:44.246016 23537 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:11:44.256099 24902 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:44.256700 24641 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:44.260200 24772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:44.267755 23407 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:11:44.271543 25173 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:44.281102 25033 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:44.362506 23427 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "98dfd1367b224981ab0fd74678bb5ac2"
dest_uuid: "76e30a81c8b8451898078e9636a1104b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38116
I20240503 08:11:44.362777 23427 raft_consensus.cc:595] T 98dfd1367b224981ab0fd74678bb5ac2 P 76e30a81c8b8451898078e9636a1104b [term 1 LEADER]: Received request to transfer leadership
I20240503 08:11:44.409916 25836 raft_consensus.cc:979] T b2779eae1c08452ca7b5b0874bbac47e P 30d88bcd65b347dabcbac54465e52b21: : Instructing follower 51d0bc1740db4723a71f1d56b7e6e6b7 to start an election
I20240503 08:11:44.410099 25613 raft_consensus.cc:1067] T b2779eae1c08452ca7b5b0874bbac47e P 30d88bcd65b347dabcbac54465e52b21 [term 1 LEADER]: Signalling peer 51d0bc1740db4723a71f1d56b7e6e6b7 to start an election
I20240503 08:11:44.410768 24792 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b2779eae1c08452ca7b5b0874bbac47e"
dest_uuid: "51d0bc1740db4723a71f1d56b7e6e6b7"
from {username='slave'} at 127.20.230.66:40991
I20240503 08:11:44.411033 24792 raft_consensus.cc:482] T b2779eae1c08452ca7b5b0874bbac47e P 51d0bc1740db4723a71f1d56b7e6e6b7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:11:44.411222 24792 raft_consensus.cc:3037] T b2779eae1c08452ca7b5b0874bbac47e P 51d0bc1740db4723a71f1d56b7e6e6b7 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:44.413455 24792 raft_consensus.cc:504] T b2779eae1c08452ca7b5b0874bbac47e P 51d0bc1740db4723a71f1d56b7e6e6b7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "51d0bc1740db4723a71f1d56b7e6e6b7" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 44977 } }
I20240503 08:11:44.414884 24792 leader_election.cc:288] T b2779eae1c08452ca7b5b0874bbac47e P 51d0bc1740db4723a71f1d56b7e6e6b7 [CANDIDATE]: Term 2 election: Requested vote from peers 30d88bcd65b347dabcbac54465e52b21 (127.20.230.66:33351)
I20240503 08:11:44.415336 23557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2779eae1c08452ca7b5b0874bbac47e" candidate_uuid: "51d0bc1740db4723a71f1d56b7e6e6b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "30d88bcd65b347dabcbac54465e52b21"
I20240503 08:11:44.415555 23557 raft_consensus.cc:3032] T b2779eae1c08452ca7b5b0874bbac47e P 30d88bcd65b347dabcbac54465e52b21 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:11:44.415645 23557 raft_consensus.cc:729] T b2779eae1c08452ca7b5b0874bbac47e P 30d88bcd65b347dabcbac54465e52b21 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30d88bcd65b347dabcbac54465e52b21, State: Running, Role: LEADER
I20240503 08:11:44.415875 23557 consensus_queue.cc:260] T b2779eae1c08452ca7b5b0874bbac47e P 30d88bcd65b347dabcbac54465e52b21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "51d0bc1740db4723a71f1d56b7e6e6b7" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 44977 } }
I20240503 08:11:44.416028 23557 raft_consensus.cc:3037] T b2779eae1c08452ca7b5b0874bbac47e P 30d88bcd65b347dabcbac54465e52b21 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:44.417784 23557 raft_consensus.cc:2454] T b2779eae1c08452ca7b5b0874bbac47e P 30d88bcd65b347dabcbac54465e52b21 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51d0bc1740db4723a71f1d56b7e6e6b7 in term 2.
I20240503 08:11:44.418277 24728 leader_election.cc:302] T b2779eae1c08452ca7b5b0874bbac47e P 51d0bc1740db4723a71f1d56b7e6e6b7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 30d88bcd65b347dabcbac54465e52b21, 51d0bc1740db4723a71f1d56b7e6e6b7; no voters:
I20240503 08:11:44.418875 25842 raft_consensus.cc:2781] T b2779eae1c08452ca7b5b0874bbac47e P 51d0bc1740db4723a71f1d56b7e6e6b7 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:11:44.419314 25842 raft_consensus.cc:686] T b2779eae1c08452ca7b5b0874bbac47e P 51d0bc1740db4723a71f1d56b7e6e6b7 [term 2 LEADER]: Becoming Leader. State: Replica: 51d0bc1740db4723a71f1d56b7e6e6b7, State: Running, Role: LEADER
I20240503 08:11:44.420403 25842 consensus_queue.cc:237] T b2779eae1c08452ca7b5b0874bbac47e P 51d0bc1740db4723a71f1d56b7e6e6b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "51d0bc1740db4723a71f1d56b7e6e6b7" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 44977 } }
I20240503 08:11:44.423825 23280 catalog_manager.cc:5484] T b2779eae1c08452ca7b5b0874bbac47e P 51d0bc1740db4723a71f1d56b7e6e6b7 reported cstate change: term changed from 1 to 2, leader changed from 30d88bcd65b347dabcbac54465e52b21 (127.20.230.66) to 51d0bc1740db4723a71f1d56b7e6e6b7 (127.20.230.68). New cstate: current_term: 2 leader_uuid: "51d0bc1740db4723a71f1d56b7e6e6b7" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "51d0bc1740db4723a71f1d56b7e6e6b7" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 44977 } } }
I20240503 08:11:44.461838 23537 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:11:44.470517 24772 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:44.486446 24641 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:44.490447 23407 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:11:44.501150 25173 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:44.508832 24902 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:44.533743 25033 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:44.538393 25864 raft_consensus.cc:979] T 98dfd1367b224981ab0fd74678bb5ac2 P 76e30a81c8b8451898078e9636a1104b: : Instructing follower 5ea58b5961b54598878bad3f5660913c to start an election
I20240503 08:11:44.538566 25864 raft_consensus.cc:1067] T 98dfd1367b224981ab0fd74678bb5ac2 P 76e30a81c8b8451898078e9636a1104b [term 1 LEADER]: Signalling peer 5ea58b5961b54598878bad3f5660913c to start an election
I20240503 08:11:44.539693 24661 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "98dfd1367b224981ab0fd74678bb5ac2"
dest_uuid: "5ea58b5961b54598878bad3f5660913c"
from {username='slave'} at 127.20.230.65:37691
I20240503 08:11:44.539978 24661 raft_consensus.cc:482] T 98dfd1367b224981ab0fd74678bb5ac2 P 5ea58b5961b54598878bad3f5660913c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:11:44.540156 24661 raft_consensus.cc:3037] T 98dfd1367b224981ab0fd74678bb5ac2 P 5ea58b5961b54598878bad3f5660913c [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:44.543609 24661 raft_consensus.cc:504] T 98dfd1367b224981ab0fd74678bb5ac2 P 5ea58b5961b54598878bad3f5660913c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "5ea58b5961b54598878bad3f5660913c" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 37877 } }
I20240503 08:11:44.545670 24661 leader_election.cc:288] T 98dfd1367b224981ab0fd74678bb5ac2 P 5ea58b5961b54598878bad3f5660913c [CANDIDATE]: Term 2 election: Requested vote from peers 76e30a81c8b8451898078e9636a1104b (127.20.230.65:37573)
I20240503 08:11:44.545701 23427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98dfd1367b224981ab0fd74678bb5ac2" candidate_uuid: "5ea58b5961b54598878bad3f5660913c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "76e30a81c8b8451898078e9636a1104b"
I20240503 08:11:44.546059 23427 raft_consensus.cc:3032] T 98dfd1367b224981ab0fd74678bb5ac2 P 76e30a81c8b8451898078e9636a1104b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:11:44.546147 23427 raft_consensus.cc:729] T 98dfd1367b224981ab0fd74678bb5ac2 P 76e30a81c8b8451898078e9636a1104b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 76e30a81c8b8451898078e9636a1104b, State: Running, Role: LEADER
I20240503 08:11:44.546448 23427 consensus_queue.cc:260] T 98dfd1367b224981ab0fd74678bb5ac2 P 76e30a81c8b8451898078e9636a1104b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "5ea58b5961b54598878bad3f5660913c" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 37877 } }
I20240503 08:11:44.546870 23427 raft_consensus.cc:3037] T 98dfd1367b224981ab0fd74678bb5ac2 P 76e30a81c8b8451898078e9636a1104b [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:44.550341 23427 raft_consensus.cc:2454] T 98dfd1367b224981ab0fd74678bb5ac2 P 76e30a81c8b8451898078e9636a1104b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ea58b5961b54598878bad3f5660913c in term 2.
I20240503 08:11:44.550984 24594 leader_election.cc:302] T 98dfd1367b224981ab0fd74678bb5ac2 P 5ea58b5961b54598878bad3f5660913c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5ea58b5961b54598878bad3f5660913c, 76e30a81c8b8451898078e9636a1104b; no voters:
I20240503 08:11:44.582234 25867 raft_consensus.cc:2781] T 98dfd1367b224981ab0fd74678bb5ac2 P 5ea58b5961b54598878bad3f5660913c [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:11:44.588900 25867 raft_consensus.cc:686] T 98dfd1367b224981ab0fd74678bb5ac2 P 5ea58b5961b54598878bad3f5660913c [term 2 LEADER]: Becoming Leader. State: Replica: 5ea58b5961b54598878bad3f5660913c, State: Running, Role: LEADER
I20240503 08:11:44.589691 25867 consensus_queue.cc:237] T 98dfd1367b224981ab0fd74678bb5ac2 P 5ea58b5961b54598878bad3f5660913c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "5ea58b5961b54598878bad3f5660913c" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 37877 } }
I20240503 08:11:44.593704 23285 catalog_manager.cc:5484] T 98dfd1367b224981ab0fd74678bb5ac2 P 5ea58b5961b54598878bad3f5660913c reported cstate change: term changed from 1 to 2, leader changed from 76e30a81c8b8451898078e9636a1104b (127.20.230.65) to 5ea58b5961b54598878bad3f5660913c (127.20.230.67). New cstate: current_term: 2 leader_uuid: "5ea58b5961b54598878bad3f5660913c" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "5ea58b5961b54598878bad3f5660913c" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 37877 } } }
I20240503 08:11:44.702210 23537 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:11:44.704435 24902 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:44.716284 24772 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:44.725090 25173 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:44.728298 23407 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:11:44.729593 24641 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:44.773092 25033 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:44.814657 23427 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "85b933bc55b1492fb70055a765a2b09b"
dest_uuid: "76e30a81c8b8451898078e9636a1104b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38146
I20240503 08:11:44.814867 23427 raft_consensus.cc:595] T 85b933bc55b1492fb70055a765a2b09b P 76e30a81c8b8451898078e9636a1104b [term 1 LEADER]: Received request to transfer leadership
I20240503 08:11:44.907296 24902 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:44.913337 24772 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:44.921213 24641 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:44.924172 23407 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:11:44.939427 23537 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:11:44.939899 25173 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:44.957283 25551 raft_consensus.cc:979] T 85b933bc55b1492fb70055a765a2b09b P 76e30a81c8b8451898078e9636a1104b: : Instructing follower 509c32504f544f8b90ebf66b6dd951f7 to start an election
I20240503 08:11:44.957475 25551 raft_consensus.cc:1067] T 85b933bc55b1492fb70055a765a2b09b P 76e30a81c8b8451898078e9636a1104b [term 1 LEADER]: Signalling peer 509c32504f544f8b90ebf66b6dd951f7 to start an election
I20240503 08:11:44.958245 24922 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "85b933bc55b1492fb70055a765a2b09b"
dest_uuid: "509c32504f544f8b90ebf66b6dd951f7"
from {username='slave'} at 127.20.230.65:37329
I20240503 08:11:44.958535 24922 raft_consensus.cc:482] T 85b933bc55b1492fb70055a765a2b09b P 509c32504f544f8b90ebf66b6dd951f7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:11:44.958716 24922 raft_consensus.cc:3037] T 85b933bc55b1492fb70055a765a2b09b P 509c32504f544f8b90ebf66b6dd951f7 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:44.961460 24922 raft_consensus.cc:504] T 85b933bc55b1492fb70055a765a2b09b P 509c32504f544f8b90ebf66b6dd951f7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "509c32504f544f8b90ebf66b6dd951f7" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 41435 } }
I20240503 08:11:44.963339 23427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b933bc55b1492fb70055a765a2b09b" candidate_uuid: "509c32504f544f8b90ebf66b6dd951f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "76e30a81c8b8451898078e9636a1104b"
I20240503 08:11:44.963569 23427 raft_consensus.cc:3032] T 85b933bc55b1492fb70055a765a2b09b P 76e30a81c8b8451898078e9636a1104b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:11:44.963647 23427 raft_consensus.cc:729] T 85b933bc55b1492fb70055a765a2b09b P 76e30a81c8b8451898078e9636a1104b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 76e30a81c8b8451898078e9636a1104b, State: Running, Role: LEADER
I20240503 08:11:44.963904 23427 consensus_queue.cc:260] T 85b933bc55b1492fb70055a765a2b09b P 76e30a81c8b8451898078e9636a1104b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "509c32504f544f8b90ebf66b6dd951f7" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 41435 } }
I20240503 08:11:44.964102 23427 raft_consensus.cc:3037] T 85b933bc55b1492fb70055a765a2b09b P 76e30a81c8b8451898078e9636a1104b [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:44.964641 24922 leader_election.cc:288] T 85b933bc55b1492fb70055a765a2b09b P 509c32504f544f8b90ebf66b6dd951f7 [CANDIDATE]: Term 2 election: Requested vote from peers 76e30a81c8b8451898078e9636a1104b (127.20.230.65:37573)
I20240503 08:11:44.967268 23427 raft_consensus.cc:2454] T 85b933bc55b1492fb70055a765a2b09b P 76e30a81c8b8451898078e9636a1104b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 509c32504f544f8b90ebf66b6dd951f7 in term 2.
I20240503 08:11:44.967738 24855 leader_election.cc:302] T 85b933bc55b1492fb70055a765a2b09b P 509c32504f544f8b90ebf66b6dd951f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 509c32504f544f8b90ebf66b6dd951f7, 76e30a81c8b8451898078e9636a1104b; no voters:
I20240503 08:11:44.968621 25033 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:44.978520 25954 raft_consensus.cc:2781] T 85b933bc55b1492fb70055a765a2b09b P 509c32504f544f8b90ebf66b6dd951f7 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:11:44.986155 25954 raft_consensus.cc:686] T 85b933bc55b1492fb70055a765a2b09b P 509c32504f544f8b90ebf66b6dd951f7 [term 2 LEADER]: Becoming Leader. State: Replica: 509c32504f544f8b90ebf66b6dd951f7, State: Running, Role: LEADER
I20240503 08:11:44.986738 25954 consensus_queue.cc:237] T 85b933bc55b1492fb70055a765a2b09b P 509c32504f544f8b90ebf66b6dd951f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "509c32504f544f8b90ebf66b6dd951f7" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 41435 } }
I20240503 08:11:44.998121 23286 catalog_manager.cc:5484] T 85b933bc55b1492fb70055a765a2b09b P 509c32504f544f8b90ebf66b6dd951f7 reported cstate change: term changed from 1 to 2, leader changed from 76e30a81c8b8451898078e9636a1104b (127.20.230.65) to 509c32504f544f8b90ebf66b6dd951f7 (127.20.230.69). New cstate: current_term: 2 leader_uuid: "509c32504f544f8b90ebf66b6dd951f7" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "509c32504f544f8b90ebf66b6dd951f7" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 41435 } } }
I20240503 08:11:45.013998 23557 raft_consensus.cc:1261] T b2779eae1c08452ca7b5b0874bbac47e P 30d88bcd65b347dabcbac54465e52b21 [term 2 FOLLOWER]: Refusing update from remote peer 51d0bc1740db4723a71f1d56b7e6e6b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 08:11:45.014748 25957 consensus_queue.cc:1035] T b2779eae1c08452ca7b5b0874bbac47e P 51d0bc1740db4723a71f1d56b7e6e6b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:11:45.076876 23427 raft_consensus.cc:1261] T 98dfd1367b224981ab0fd74678bb5ac2 P 76e30a81c8b8451898078e9636a1104b [term 2 FOLLOWER]: Refusing update from remote peer 5ea58b5961b54598878bad3f5660913c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 08:11:45.077602 25867 consensus_queue.cc:1035] T 98dfd1367b224981ab0fd74678bb5ac2 P 5ea58b5961b54598878bad3f5660913c [LEADER]: Connected to new peer: Peer: permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:11:45.139640 23537 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:11:45.149333 24902 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:45.151185 24641 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:45.168911 24772 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:45.182641 23407 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:11:45.187969 25033 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:45.205250 25173 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:45.258431 23557 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2449909504504d1e84cf34f1af673af2"
dest_uuid: "30d88bcd65b347dabcbac54465e52b21"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39864
I20240503 08:11:45.258622 23557 raft_consensus.cc:595] T 2449909504504d1e84cf34f1af673af2 P 30d88bcd65b347dabcbac54465e52b21 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:11:45.346015 24902 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:45.346686 23537 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:11:45.370108 24772 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:45.373562 24641 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:45.374951 25173 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:45.377606 23407 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:11:45.388358 25033 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:45.466614 25613 raft_consensus.cc:979] T 2449909504504d1e84cf34f1af673af2 P 30d88bcd65b347dabcbac54465e52b21: : Instructing follower 51d0bc1740db4723a71f1d56b7e6e6b7 to start an election
I20240503 08:11:45.466748 25613 raft_consensus.cc:1067] T 2449909504504d1e84cf34f1af673af2 P 30d88bcd65b347dabcbac54465e52b21 [term 1 LEADER]: Signalling peer 51d0bc1740db4723a71f1d56b7e6e6b7 to start an election
I20240503 08:11:45.467353 24792 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2449909504504d1e84cf34f1af673af2"
dest_uuid: "51d0bc1740db4723a71f1d56b7e6e6b7"
from {username='slave'} at 127.20.230.66:40991
I20240503 08:11:45.467564 24792 raft_consensus.cc:482] T 2449909504504d1e84cf34f1af673af2 P 51d0bc1740db4723a71f1d56b7e6e6b7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:11:45.467654 24792 raft_consensus.cc:3037] T 2449909504504d1e84cf34f1af673af2 P 51d0bc1740db4723a71f1d56b7e6e6b7 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:45.470849 24792 raft_consensus.cc:504] T 2449909504504d1e84cf34f1af673af2 P 51d0bc1740db4723a71f1d56b7e6e6b7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "51d0bc1740db4723a71f1d56b7e6e6b7" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 44977 } }
I20240503 08:11:45.471500 24792 leader_election.cc:288] T 2449909504504d1e84cf34f1af673af2 P 51d0bc1740db4723a71f1d56b7e6e6b7 [CANDIDATE]: Term 2 election: Requested vote from peers 30d88bcd65b347dabcbac54465e52b21 (127.20.230.66:33351)
I20240503 08:11:45.471927 23557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2449909504504d1e84cf34f1af673af2" candidate_uuid: "51d0bc1740db4723a71f1d56b7e6e6b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "30d88bcd65b347dabcbac54465e52b21"
I20240503 08:11:45.472134 23557 raft_consensus.cc:3032] T 2449909504504d1e84cf34f1af673af2 P 30d88bcd65b347dabcbac54465e52b21 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:11:45.472241 23557 raft_consensus.cc:729] T 2449909504504d1e84cf34f1af673af2 P 30d88bcd65b347dabcbac54465e52b21 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30d88bcd65b347dabcbac54465e52b21, State: Running, Role: LEADER
I20240503 08:11:45.472510 23557 consensus_queue.cc:260] T 2449909504504d1e84cf34f1af673af2 P 30d88bcd65b347dabcbac54465e52b21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "51d0bc1740db4723a71f1d56b7e6e6b7" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 44977 } }
I20240503 08:11:45.472708 23557 raft_consensus.cc:3037] T 2449909504504d1e84cf34f1af673af2 P 30d88bcd65b347dabcbac54465e52b21 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:45.475013 23557 raft_consensus.cc:2454] T 2449909504504d1e84cf34f1af673af2 P 30d88bcd65b347dabcbac54465e52b21 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51d0bc1740db4723a71f1d56b7e6e6b7 in term 2.
I20240503 08:11:45.475419 24728 leader_election.cc:302] T 2449909504504d1e84cf34f1af673af2 P 51d0bc1740db4723a71f1d56b7e6e6b7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 30d88bcd65b347dabcbac54465e52b21, 51d0bc1740db4723a71f1d56b7e6e6b7; no voters:
I20240503 08:11:45.475766 25957 raft_consensus.cc:2781] T 2449909504504d1e84cf34f1af673af2 P 51d0bc1740db4723a71f1d56b7e6e6b7 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:11:45.475929 25957 raft_consensus.cc:686] T 2449909504504d1e84cf34f1af673af2 P 51d0bc1740db4723a71f1d56b7e6e6b7 [term 2 LEADER]: Becoming Leader. State: Replica: 51d0bc1740db4723a71f1d56b7e6e6b7, State: Running, Role: LEADER
I20240503 08:11:45.476194 25957 consensus_queue.cc:237] T 2449909504504d1e84cf34f1af673af2 P 51d0bc1740db4723a71f1d56b7e6e6b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "51d0bc1740db4723a71f1d56b7e6e6b7" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 44977 } }
I20240503 08:11:45.478572 23285 catalog_manager.cc:5484] T 2449909504504d1e84cf34f1af673af2 P 51d0bc1740db4723a71f1d56b7e6e6b7 reported cstate change: term changed from 1 to 2, leader changed from 30d88bcd65b347dabcbac54465e52b21 (127.20.230.66) to 51d0bc1740db4723a71f1d56b7e6e6b7 (127.20.230.68). New cstate: current_term: 2 leader_uuid: "51d0bc1740db4723a71f1d56b7e6e6b7" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "51d0bc1740db4723a71f1d56b7e6e6b7" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 44977 } } }
I20240503 08:11:45.520375 23427 raft_consensus.cc:1261] T 85b933bc55b1492fb70055a765a2b09b P 76e30a81c8b8451898078e9636a1104b [term 2 FOLLOWER]: Refusing update from remote peer 509c32504f544f8b90ebf66b6dd951f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 08:11:45.521051 26070 consensus_queue.cc:1035] T 85b933bc55b1492fb70055a765a2b09b P 509c32504f544f8b90ebf66b6dd951f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:11:45.564409 23537 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:11:45.569468 24772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:45.579457 24902 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:45.582368 24641 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:45.617269 23407 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:11:45.633731 25173 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:45.652573 25033 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:45.690927 23427 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0d69b1adfa064beb82bdcc16d53717d7"
dest_uuid: "76e30a81c8b8451898078e9636a1104b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38196
I20240503 08:11:45.691143 23427 raft_consensus.cc:595] T 0d69b1adfa064beb82bdcc16d53717d7 P 76e30a81c8b8451898078e9636a1104b [term 1 LEADER]: Received request to transfer leadership
I20240503 08:11:45.778384 23537 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:11:45.794620 24902 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:45.804013 25173 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:45.806041 24772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:45.816534 24641 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:45.817425 23407 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:11:45.821388 25033 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:45.974443 23537 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:11:45.979130 24902 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:45.999097 24772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:46.014904 24641 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:46.049280 23407 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:11:46.058264 25033 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:46.059018 25173 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:46.087574 23557 raft_consensus.cc:1261] T 2449909504504d1e84cf34f1af673af2 P 30d88bcd65b347dabcbac54465e52b21 [term 2 FOLLOWER]: Refusing update from remote peer 51d0bc1740db4723a71f1d56b7e6e6b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 08:11:46.089857 26165 consensus_queue.cc:1035] T 2449909504504d1e84cf34f1af673af2 P 51d0bc1740db4723a71f1d56b7e6e6b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:11:46.110193 26167 raft_consensus.cc:979] T 0d69b1adfa064beb82bdcc16d53717d7 P 76e30a81c8b8451898078e9636a1104b: : Instructing follower 5ea58b5961b54598878bad3f5660913c to start an election
I20240503 08:11:46.111132 25865 raft_consensus.cc:1067] T 0d69b1adfa064beb82bdcc16d53717d7 P 76e30a81c8b8451898078e9636a1104b [term 1 LEADER]: Signalling peer 5ea58b5961b54598878bad3f5660913c to start an election
I20240503 08:11:46.113289 24661 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0d69b1adfa064beb82bdcc16d53717d7"
dest_uuid: "5ea58b5961b54598878bad3f5660913c"
from {username='slave'} at 127.20.230.65:37691
I20240503 08:11:46.113555 24661 raft_consensus.cc:482] T 0d69b1adfa064beb82bdcc16d53717d7 P 5ea58b5961b54598878bad3f5660913c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:11:46.113682 24661 raft_consensus.cc:3037] T 0d69b1adfa064beb82bdcc16d53717d7 P 5ea58b5961b54598878bad3f5660913c [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:46.116708 24661 raft_consensus.cc:504] T 0d69b1adfa064beb82bdcc16d53717d7 P 5ea58b5961b54598878bad3f5660913c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "5ea58b5961b54598878bad3f5660913c" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 37877 } }
I20240503 08:11:46.117336 24661 leader_election.cc:288] T 0d69b1adfa064beb82bdcc16d53717d7 P 5ea58b5961b54598878bad3f5660913c [CANDIDATE]: Term 2 election: Requested vote from peers 76e30a81c8b8451898078e9636a1104b (127.20.230.65:37573)
I20240503 08:11:46.117937 23427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d69b1adfa064beb82bdcc16d53717d7" candidate_uuid: "5ea58b5961b54598878bad3f5660913c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "76e30a81c8b8451898078e9636a1104b"
I20240503 08:11:46.118191 23427 raft_consensus.cc:3032] T 0d69b1adfa064beb82bdcc16d53717d7 P 76e30a81c8b8451898078e9636a1104b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:11:46.118299 23427 raft_consensus.cc:729] T 0d69b1adfa064beb82bdcc16d53717d7 P 76e30a81c8b8451898078e9636a1104b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 76e30a81c8b8451898078e9636a1104b, State: Running, Role: LEADER
I20240503 08:11:46.118597 23427 consensus_queue.cc:260] T 0d69b1adfa064beb82bdcc16d53717d7 P 76e30a81c8b8451898078e9636a1104b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "5ea58b5961b54598878bad3f5660913c" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 37877 } }
I20240503 08:11:46.118817 23427 raft_consensus.cc:3037] T 0d69b1adfa064beb82bdcc16d53717d7 P 76e30a81c8b8451898078e9636a1104b [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:46.121686 23427 raft_consensus.cc:2454] T 0d69b1adfa064beb82bdcc16d53717d7 P 76e30a81c8b8451898078e9636a1104b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ea58b5961b54598878bad3f5660913c in term 2.
I20240503 08:11:46.122301 24594 leader_election.cc:302] T 0d69b1adfa064beb82bdcc16d53717d7 P 5ea58b5961b54598878bad3f5660913c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5ea58b5961b54598878bad3f5660913c, 76e30a81c8b8451898078e9636a1104b; no voters:
I20240503 08:11:46.122718 26161 raft_consensus.cc:2781] T 0d69b1adfa064beb82bdcc16d53717d7 P 5ea58b5961b54598878bad3f5660913c [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:11:46.122865 26161 raft_consensus.cc:686] T 0d69b1adfa064beb82bdcc16d53717d7 P 5ea58b5961b54598878bad3f5660913c [term 2 LEADER]: Becoming Leader. State: Replica: 5ea58b5961b54598878bad3f5660913c, State: Running, Role: LEADER
I20240503 08:11:46.123144 26161 consensus_queue.cc:237] T 0d69b1adfa064beb82bdcc16d53717d7 P 5ea58b5961b54598878bad3f5660913c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "5ea58b5961b54598878bad3f5660913c" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 37877 } }
I20240503 08:11:46.129348 23286 catalog_manager.cc:5484] T 0d69b1adfa064beb82bdcc16d53717d7 P 5ea58b5961b54598878bad3f5660913c reported cstate change: term changed from 1 to 2, leader changed from 76e30a81c8b8451898078e9636a1104b (127.20.230.65) to 5ea58b5961b54598878bad3f5660913c (127.20.230.67). New cstate: current_term: 2 leader_uuid: "5ea58b5961b54598878bad3f5660913c" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "5ea58b5961b54598878bad3f5660913c" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 37877 } } }
I20240503 08:11:46.145565 23557 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8c6217accd494f5182a862c31013b76e"
dest_uuid: "30d88bcd65b347dabcbac54465e52b21"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55768
I20240503 08:11:46.145977 23557 raft_consensus.cc:595] T 8c6217accd494f5182a862c31013b76e P 30d88bcd65b347dabcbac54465e52b21 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:11:46.224408 23537 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:11:46.234632 24902 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:46.248577 24641 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:46.248868 24772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:46.275144 25033 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:46.289018 23407 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:11:46.298655 25173 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:46.384141 26222 raft_consensus.cc:979] T 8c6217accd494f5182a862c31013b76e P 30d88bcd65b347dabcbac54465e52b21: : Instructing follower 7823cb7f30b045209bc5ccfb110bf199 to start an election
I20240503 08:11:46.384330 26052 raft_consensus.cc:1067] T 8c6217accd494f5182a862c31013b76e P 30d88bcd65b347dabcbac54465e52b21 [term 1 LEADER]: Signalling peer 7823cb7f30b045209bc5ccfb110bf199 to start an election
I20240503 08:11:46.384936 25193 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c6217accd494f5182a862c31013b76e"
dest_uuid: "7823cb7f30b045209bc5ccfb110bf199"
from {username='slave'} at 127.20.230.66:42981
I20240503 08:11:46.385134 25193 raft_consensus.cc:482] T 8c6217accd494f5182a862c31013b76e P 7823cb7f30b045209bc5ccfb110bf199 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:11:46.385241 25193 raft_consensus.cc:3037] T 8c6217accd494f5182a862c31013b76e P 7823cb7f30b045209bc5ccfb110bf199 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:46.387250 25193 raft_consensus.cc:504] T 8c6217accd494f5182a862c31013b76e P 7823cb7f30b045209bc5ccfb110bf199 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "7823cb7f30b045209bc5ccfb110bf199" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 38073 } }
I20240503 08:11:46.387698 25193 leader_election.cc:288] T 8c6217accd494f5182a862c31013b76e P 7823cb7f30b045209bc5ccfb110bf199 [CANDIDATE]: Term 2 election: Requested vote from peers 30d88bcd65b347dabcbac54465e52b21 (127.20.230.66:33351)
I20240503 08:11:46.388193 23557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c6217accd494f5182a862c31013b76e" candidate_uuid: "7823cb7f30b045209bc5ccfb110bf199" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "30d88bcd65b347dabcbac54465e52b21"
I20240503 08:11:46.388410 23557 raft_consensus.cc:3032] T 8c6217accd494f5182a862c31013b76e P 30d88bcd65b347dabcbac54465e52b21 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:11:46.388482 23557 raft_consensus.cc:729] T 8c6217accd494f5182a862c31013b76e P 30d88bcd65b347dabcbac54465e52b21 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30d88bcd65b347dabcbac54465e52b21, State: Running, Role: LEADER
I20240503 08:11:46.388707 23557 consensus_queue.cc:260] T 8c6217accd494f5182a862c31013b76e P 30d88bcd65b347dabcbac54465e52b21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "7823cb7f30b045209bc5ccfb110bf199" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 38073 } }
I20240503 08:11:46.388957 23557 raft_consensus.cc:3037] T 8c6217accd494f5182a862c31013b76e P 30d88bcd65b347dabcbac54465e52b21 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:46.390746 23557 raft_consensus.cc:2454] T 8c6217accd494f5182a862c31013b76e P 30d88bcd65b347dabcbac54465e52b21 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7823cb7f30b045209bc5ccfb110bf199 in term 2.
I20240503 08:11:46.391345 25129 leader_election.cc:302] T 8c6217accd494f5182a862c31013b76e P 7823cb7f30b045209bc5ccfb110bf199 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 30d88bcd65b347dabcbac54465e52b21, 7823cb7f30b045209bc5ccfb110bf199; no voters:
I20240503 08:11:46.392191 25700 raft_consensus.cc:2781] T 8c6217accd494f5182a862c31013b76e P 7823cb7f30b045209bc5ccfb110bf199 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:11:46.392537 25700 raft_consensus.cc:686] T 8c6217accd494f5182a862c31013b76e P 7823cb7f30b045209bc5ccfb110bf199 [term 2 LEADER]: Becoming Leader. State: Replica: 7823cb7f30b045209bc5ccfb110bf199, State: Running, Role: LEADER
I20240503 08:11:46.392851 25700 consensus_queue.cc:237] T 8c6217accd494f5182a862c31013b76e P 7823cb7f30b045209bc5ccfb110bf199 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "7823cb7f30b045209bc5ccfb110bf199" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 38073 } }
I20240503 08:11:46.395123 23284 catalog_manager.cc:5484] T 8c6217accd494f5182a862c31013b76e P 7823cb7f30b045209bc5ccfb110bf199 reported cstate change: term changed from 1 to 2, leader changed from 30d88bcd65b347dabcbac54465e52b21 (127.20.230.66) to 7823cb7f30b045209bc5ccfb110bf199 (127.20.230.71). New cstate: current_term: 2 leader_uuid: "7823cb7f30b045209bc5ccfb110bf199" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "7823cb7f30b045209bc5ccfb110bf199" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 38073 } } }
I20240503 08:11:46.440758 23537 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:46.457624 24772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:46.458004 24902 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:46.464701 24641 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:46.497778 25173 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:46.499967 23407 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:11:46.508232 25033 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:46.546994 23427 raft_consensus.cc:1261] T 0d69b1adfa064beb82bdcc16d53717d7 P 76e30a81c8b8451898078e9636a1104b [term 2 FOLLOWER]: Refusing update from remote peer 5ea58b5961b54598878bad3f5660913c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 08:11:46.547642 26161 consensus_queue.cc:1035] T 0d69b1adfa064beb82bdcc16d53717d7 P 5ea58b5961b54598878bad3f5660913c [LEADER]: Connected to new peer: Peer: permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:11:46.582444 23427 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "29dea2294fd74738a55ac1bd5b88121b"
dest_uuid: "76e30a81c8b8451898078e9636a1104b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55300
I20240503 08:11:46.582612 23427 raft_consensus.cc:595] T 29dea2294fd74738a55ac1bd5b88121b P 76e30a81c8b8451898078e9636a1104b [term 1 LEADER]: Received request to transfer leadership
I20240503 08:11:46.659251 23537 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:46.665474 24902 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:46.679360 23407 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:11:46.683403 25173 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:46.690182 25033 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:46.717379 24772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:46.717952 24641 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:46.846498 23557 raft_consensus.cc:1261] T 8c6217accd494f5182a862c31013b76e P 30d88bcd65b347dabcbac54465e52b21 [term 2 FOLLOWER]: Refusing update from remote peer 7823cb7f30b045209bc5ccfb110bf199: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 08:11:46.847124 25700 consensus_queue.cc:1035] T 8c6217accd494f5182a862c31013b76e P 7823cb7f30b045209bc5ccfb110bf199 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:11:46.858409 24772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:46.868796 23537 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:46.874426 23407 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:11:46.878718 24641 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:46.891505 24902 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:46.906656 25173 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:46.920612 25033 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:46.940789 26329 raft_consensus.cc:979] T 29dea2294fd74738a55ac1bd5b88121b P 76e30a81c8b8451898078e9636a1104b: : Instructing follower c82f2e5e9e13405d9328620179077098 to start an election
I20240503 08:11:46.941061 26167 raft_consensus.cc:1067] T 29dea2294fd74738a55ac1bd5b88121b P 76e30a81c8b8451898078e9636a1104b [term 1 LEADER]: Signalling peer c82f2e5e9e13405d9328620179077098 to start an election
I20240503 08:11:46.941862 25053 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "29dea2294fd74738a55ac1bd5b88121b"
dest_uuid: "c82f2e5e9e13405d9328620179077098"
from {username='slave'} at 127.20.230.65:59847
I20240503 08:11:46.942307 25053 raft_consensus.cc:482] T 29dea2294fd74738a55ac1bd5b88121b P c82f2e5e9e13405d9328620179077098 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:11:46.942440 25053 raft_consensus.cc:3037] T 29dea2294fd74738a55ac1bd5b88121b P c82f2e5e9e13405d9328620179077098 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:46.945261 25053 raft_consensus.cc:504] T 29dea2294fd74738a55ac1bd5b88121b P c82f2e5e9e13405d9328620179077098 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "c82f2e5e9e13405d9328620179077098" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 36811 } }
I20240503 08:11:46.946980 25053 leader_election.cc:288] T 29dea2294fd74738a55ac1bd5b88121b P c82f2e5e9e13405d9328620179077098 [CANDIDATE]: Term 2 election: Requested vote from peers 76e30a81c8b8451898078e9636a1104b (127.20.230.65:37573)
I20240503 08:11:46.947427 23427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29dea2294fd74738a55ac1bd5b88121b" candidate_uuid: "c82f2e5e9e13405d9328620179077098" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "76e30a81c8b8451898078e9636a1104b"
I20240503 08:11:46.947623 23427 raft_consensus.cc:3032] T 29dea2294fd74738a55ac1bd5b88121b P 76e30a81c8b8451898078e9636a1104b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:11:46.947710 23427 raft_consensus.cc:729] T 29dea2294fd74738a55ac1bd5b88121b P 76e30a81c8b8451898078e9636a1104b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 76e30a81c8b8451898078e9636a1104b, State: Running, Role: LEADER
I20240503 08:11:46.948122 23427 consensus_queue.cc:260] T 29dea2294fd74738a55ac1bd5b88121b P 76e30a81c8b8451898078e9636a1104b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "c82f2e5e9e13405d9328620179077098" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 36811 } }
I20240503 08:11:46.948383 23427 raft_consensus.cc:3037] T 29dea2294fd74738a55ac1bd5b88121b P 76e30a81c8b8451898078e9636a1104b [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:46.951179 23427 raft_consensus.cc:2454] T 29dea2294fd74738a55ac1bd5b88121b P 76e30a81c8b8451898078e9636a1104b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c82f2e5e9e13405d9328620179077098 in term 2.
I20240503 08:11:46.951721 24985 leader_election.cc:302] T 29dea2294fd74738a55ac1bd5b88121b P c82f2e5e9e13405d9328620179077098 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 76e30a81c8b8451898078e9636a1104b, c82f2e5e9e13405d9328620179077098; no voters:
I20240503 08:11:46.963320 26330 raft_consensus.cc:2781] T 29dea2294fd74738a55ac1bd5b88121b P c82f2e5e9e13405d9328620179077098 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:11:46.964789 26330 raft_consensus.cc:686] T 29dea2294fd74738a55ac1bd5b88121b P c82f2e5e9e13405d9328620179077098 [term 2 LEADER]: Becoming Leader. State: Replica: c82f2e5e9e13405d9328620179077098, State: Running, Role: LEADER
I20240503 08:11:46.965116 26330 consensus_queue.cc:237] T 29dea2294fd74738a55ac1bd5b88121b P c82f2e5e9e13405d9328620179077098 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "c82f2e5e9e13405d9328620179077098" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 36811 } }
I20240503 08:11:46.975730 23282 catalog_manager.cc:5484] T 29dea2294fd74738a55ac1bd5b88121b P c82f2e5e9e13405d9328620179077098 reported cstate change: term changed from 1 to 2, leader changed from 76e30a81c8b8451898078e9636a1104b (127.20.230.65) to c82f2e5e9e13405d9328620179077098 (127.20.230.70). New cstate: current_term: 2 leader_uuid: "c82f2e5e9e13405d9328620179077098" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "c82f2e5e9e13405d9328620179077098" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 36811 } } }
I20240503 08:11:46.990705 23427 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "78b4f0eae3d14b97b8f74cbc24389044"
dest_uuid: "76e30a81c8b8451898078e9636a1104b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55324
I20240503 08:11:46.990885 23427 raft_consensus.cc:595] T 78b4f0eae3d14b97b8f74cbc24389044 P 76e30a81c8b8451898078e9636a1104b [term 1 LEADER]: Received request to transfer leadership
I20240503 08:11:47.088418 26329 raft_consensus.cc:979] T 78b4f0eae3d14b97b8f74cbc24389044 P 76e30a81c8b8451898078e9636a1104b: : Instructing follower 509c32504f544f8b90ebf66b6dd951f7 to start an election
I20240503 08:11:47.088600 26329 raft_consensus.cc:1067] T 78b4f0eae3d14b97b8f74cbc24389044 P 76e30a81c8b8451898078e9636a1104b [term 1 LEADER]: Signalling peer 509c32504f544f8b90ebf66b6dd951f7 to start an election
I20240503 08:11:47.089398 24922 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "78b4f0eae3d14b97b8f74cbc24389044"
dest_uuid: "509c32504f544f8b90ebf66b6dd951f7"
from {username='slave'} at 127.20.230.65:37329
I20240503 08:11:47.089637 24922 raft_consensus.cc:482] T 78b4f0eae3d14b97b8f74cbc24389044 P 509c32504f544f8b90ebf66b6dd951f7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:11:47.089756 24922 raft_consensus.cc:3037] T 78b4f0eae3d14b97b8f74cbc24389044 P 509c32504f544f8b90ebf66b6dd951f7 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:47.094259 24922 raft_consensus.cc:504] T 78b4f0eae3d14b97b8f74cbc24389044 P 509c32504f544f8b90ebf66b6dd951f7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "509c32504f544f8b90ebf66b6dd951f7" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 41435 } }
I20240503 08:11:47.097640 24922 leader_election.cc:288] T 78b4f0eae3d14b97b8f74cbc24389044 P 509c32504f544f8b90ebf66b6dd951f7 [CANDIDATE]: Term 2 election: Requested vote from peers 76e30a81c8b8451898078e9636a1104b (127.20.230.65:37573)
I20240503 08:11:47.097671 23537 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:47.098161 23427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78b4f0eae3d14b97b8f74cbc24389044" candidate_uuid: "509c32504f544f8b90ebf66b6dd951f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "76e30a81c8b8451898078e9636a1104b"
I20240503 08:11:47.098398 23427 raft_consensus.cc:3032] T 78b4f0eae3d14b97b8f74cbc24389044 P 76e30a81c8b8451898078e9636a1104b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:11:47.098516 23427 raft_consensus.cc:729] T 78b4f0eae3d14b97b8f74cbc24389044 P 76e30a81c8b8451898078e9636a1104b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 76e30a81c8b8451898078e9636a1104b, State: Running, Role: LEADER
I20240503 08:11:47.098781 23427 consensus_queue.cc:260] T 78b4f0eae3d14b97b8f74cbc24389044 P 76e30a81c8b8451898078e9636a1104b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "509c32504f544f8b90ebf66b6dd951f7" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 41435 } }
I20240503 08:11:47.099028 23427 raft_consensus.cc:3037] T 78b4f0eae3d14b97b8f74cbc24389044 P 76e30a81c8b8451898078e9636a1104b [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:47.103601 23427 raft_consensus.cc:2454] T 78b4f0eae3d14b97b8f74cbc24389044 P 76e30a81c8b8451898078e9636a1104b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 509c32504f544f8b90ebf66b6dd951f7 in term 2.
I20240503 08:11:47.104429 24855 leader_election.cc:302] T 78b4f0eae3d14b97b8f74cbc24389044 P 509c32504f544f8b90ebf66b6dd951f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 509c32504f544f8b90ebf66b6dd951f7, 76e30a81c8b8451898078e9636a1104b; no voters:
I20240503 08:11:47.106977 26075 raft_consensus.cc:2781] T 78b4f0eae3d14b97b8f74cbc24389044 P 509c32504f544f8b90ebf66b6dd951f7 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:11:47.115676 24641 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:47.117637 23407 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:47.123303 26075 raft_consensus.cc:686] T 78b4f0eae3d14b97b8f74cbc24389044 P 509c32504f544f8b90ebf66b6dd951f7 [term 2 LEADER]: Becoming Leader. State: Replica: 509c32504f544f8b90ebf66b6dd951f7, State: Running, Role: LEADER
I20240503 08:11:47.123692 26075 consensus_queue.cc:237] T 78b4f0eae3d14b97b8f74cbc24389044 P 509c32504f544f8b90ebf66b6dd951f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "509c32504f544f8b90ebf66b6dd951f7" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 41435 } }
I20240503 08:11:47.127977 25173 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:47.131958 24902 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:47.134464 23285 catalog_manager.cc:5484] T 78b4f0eae3d14b97b8f74cbc24389044 P 509c32504f544f8b90ebf66b6dd951f7 reported cstate change: term changed from 1 to 2, leader changed from 76e30a81c8b8451898078e9636a1104b (127.20.230.65) to 509c32504f544f8b90ebf66b6dd951f7 (127.20.230.69). New cstate: current_term: 2 leader_uuid: "509c32504f544f8b90ebf66b6dd951f7" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "509c32504f544f8b90ebf66b6dd951f7" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 41435 } } }
I20240503 08:11:47.151605 25033 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:47.166599 24772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:47.419756 23427 raft_consensus.cc:1261] T 29dea2294fd74738a55ac1bd5b88121b P 76e30a81c8b8451898078e9636a1104b [term 2 FOLLOWER]: Refusing update from remote peer c82f2e5e9e13405d9328620179077098: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 08:11:47.420514 26330 consensus_queue.cc:1035] T 29dea2294fd74738a55ac1bd5b88121b P c82f2e5e9e13405d9328620179077098 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:11:47.507753 23537 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:47.514384 24641 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:47.527226 24902 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:47.530069 24772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:47.546118 23407 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:47.552680 25033 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:47.578912 25173 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:47.650650 25193 consensus_queue.cc:237] T 43fb21a411474f62afad731b47bad5a9 P 7823cb7f30b045209bc5ccfb110bf199 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7823cb7f30b045209bc5ccfb110bf199" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 38073 } }
I20240503 08:11:47.651970 26228 raft_consensus.cc:2932] T 43fb21a411474f62afad731b47bad5a9 P 7823cb7f30b045209bc5ccfb110bf199 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 30d88bcd65b347dabcbac54465e52b21 (127.20.230.66) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7823cb7f30b045209bc5ccfb110bf199" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 38073 } } }
I20240503 08:11:47.656567 23282 catalog_manager.cc:5484] T 43fb21a411474f62afad731b47bad5a9 P 7823cb7f30b045209bc5ccfb110bf199 reported cstate change: config changed from index 4 to 6, VOTER 30d88bcd65b347dabcbac54465e52b21 (127.20.230.66) evicted. New cstate: current_term: 2 leader_uuid: "7823cb7f30b045209bc5ccfb110bf199" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7823cb7f30b045209bc5ccfb110bf199" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 38073 } } }
I20240503 08:11:47.661935 23537 tablet_service.cc:1514] Processing DeleteTablet for tablet 43fb21a411474f62afad731b47bad5a9 with delete_type TABLET_DATA_TOMBSTONED (TS 30d88bcd65b347dabcbac54465e52b21 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35234
I20240503 08:11:47.662775 26439 tablet_replica.cc:330] T 43fb21a411474f62afad731b47bad5a9 P 30d88bcd65b347dabcbac54465e52b21: stopping tablet replica
I20240503 08:11:47.663195 26439 raft_consensus.cc:2229] T 43fb21a411474f62afad731b47bad5a9 P 30d88bcd65b347dabcbac54465e52b21 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:47.663431 26439 raft_consensus.cc:2258] T 43fb21a411474f62afad731b47bad5a9 P 30d88bcd65b347dabcbac54465e52b21 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:47.664589 26439 ts_tablet_manager.cc:1875] T 43fb21a411474f62afad731b47bad5a9 P 30d88bcd65b347dabcbac54465e52b21: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:11:47.670794 26439 ts_tablet_manager.cc:1888] T 43fb21a411474f62afad731b47bad5a9 P 30d88bcd65b347dabcbac54465e52b21: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 08:11:47.670969 26439 log.cc:1196] T 43fb21a411474f62afad731b47bad5a9 P 30d88bcd65b347dabcbac54465e52b21: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/43fb21a411474f62afad731b47bad5a9
I20240503 08:11:47.671881 23273 catalog_manager.cc:4830] TS 30d88bcd65b347dabcbac54465e52b21 (127.20.230.66:33351): tablet 43fb21a411474f62afad731b47bad5a9 (table rebalance_test_table_1 [id=be175f2340dc4b2a947a393d8a2bd2c3]) successfully deleted
I20240503 08:11:47.741547 23427 raft_consensus.cc:1261] T 78b4f0eae3d14b97b8f74cbc24389044 P 76e30a81c8b8451898078e9636a1104b [term 2 FOLLOWER]: Refusing update from remote peer 509c32504f544f8b90ebf66b6dd951f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 08:11:47.742259 26075 consensus_queue.cc:1035] T 78b4f0eae3d14b97b8f74cbc24389044 P 509c32504f544f8b90ebf66b6dd951f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:11:47.759248 23537 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:47.762225 24902 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:47.770992 24772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:47.778844 23407 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:47.781730 24641 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:47.804402 25173 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:47.826594 25033 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:47.912158 24792 consensus_queue.cc:237] T b2779eae1c08452ca7b5b0874bbac47e P 51d0bc1740db4723a71f1d56b7e6e6b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "51d0bc1740db4723a71f1d56b7e6e6b7" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 44977 } }
I20240503 08:11:47.913309 26165 raft_consensus.cc:2932] T b2779eae1c08452ca7b5b0874bbac47e P 51d0bc1740db4723a71f1d56b7e6e6b7 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 30d88bcd65b347dabcbac54465e52b21 (127.20.230.66) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "51d0bc1740db4723a71f1d56b7e6e6b7" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 44977 } } }
I20240503 08:11:47.918578 23280 catalog_manager.cc:5484] T b2779eae1c08452ca7b5b0874bbac47e P 51d0bc1740db4723a71f1d56b7e6e6b7 reported cstate change: config changed from index 4 to 6, VOTER 30d88bcd65b347dabcbac54465e52b21 (127.20.230.66) evicted. New cstate: current_term: 2 leader_uuid: "51d0bc1740db4723a71f1d56b7e6e6b7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "51d0bc1740db4723a71f1d56b7e6e6b7" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 44977 } } }
I20240503 08:11:47.921942 23537 tablet_service.cc:1514] Processing DeleteTablet for tablet b2779eae1c08452ca7b5b0874bbac47e with delete_type TABLET_DATA_TOMBSTONED (TS 30d88bcd65b347dabcbac54465e52b21 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35234
I20240503 08:11:47.922250 26439 tablet_replica.cc:330] T b2779eae1c08452ca7b5b0874bbac47e P 30d88bcd65b347dabcbac54465e52b21: stopping tablet replica
I20240503 08:11:47.922415 26439 raft_consensus.cc:2229] T b2779eae1c08452ca7b5b0874bbac47e P 30d88bcd65b347dabcbac54465e52b21 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:47.922586 26439 raft_consensus.cc:2258] T b2779eae1c08452ca7b5b0874bbac47e P 30d88bcd65b347dabcbac54465e52b21 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:47.923142 26439 ts_tablet_manager.cc:1875] T b2779eae1c08452ca7b5b0874bbac47e P 30d88bcd65b347dabcbac54465e52b21: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:11:47.927389 26439 ts_tablet_manager.cc:1888] T b2779eae1c08452ca7b5b0874bbac47e P 30d88bcd65b347dabcbac54465e52b21: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 08:11:47.927492 26439 log.cc:1196] T b2779eae1c08452ca7b5b0874bbac47e P 30d88bcd65b347dabcbac54465e52b21: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/b2779eae1c08452ca7b5b0874bbac47e
I20240503 08:11:47.928087 23273 catalog_manager.cc:4830] TS 30d88bcd65b347dabcbac54465e52b21 (127.20.230.66:33351): tablet b2779eae1c08452ca7b5b0874bbac47e (table rebalance_test_table_2 [id=9e0052fa5a20450fad9ea538bf678fcd]) successfully deleted
I20240503 08:11:48.002720 23537 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:48.006742 24902 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:48.025804 24641 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:48.031865 24772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:48.034363 23407 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:48.060580 25033 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:48.068506 25173 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:48.145182 24661 consensus_queue.cc:237] T 98dfd1367b224981ab0fd74678bb5ac2 P 5ea58b5961b54598878bad3f5660913c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5ea58b5961b54598878bad3f5660913c" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 37877 } }
I20240503 08:11:48.146526 26161 raft_consensus.cc:2932] T 98dfd1367b224981ab0fd74678bb5ac2 P 5ea58b5961b54598878bad3f5660913c [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 76e30a81c8b8451898078e9636a1104b (127.20.230.65) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5ea58b5961b54598878bad3f5660913c" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 37877 } } }
I20240503 08:11:48.150720 23283 catalog_manager.cc:5484] T 98dfd1367b224981ab0fd74678bb5ac2 P 5ea58b5961b54598878bad3f5660913c reported cstate change: config changed from index 4 to 6, VOTER 76e30a81c8b8451898078e9636a1104b (127.20.230.65) evicted. New cstate: current_term: 2 leader_uuid: "5ea58b5961b54598878bad3f5660913c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5ea58b5961b54598878bad3f5660913c" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 37877 } } }
I20240503 08:11:48.154124 23407 tablet_service.cc:1514] Processing DeleteTablet for tablet 98dfd1367b224981ab0fd74678bb5ac2 with delete_type TABLET_DATA_TOMBSTONED (TS 76e30a81c8b8451898078e9636a1104b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41888
I20240503 08:11:48.162397 26549 tablet_replica.cc:330] T 98dfd1367b224981ab0fd74678bb5ac2 P 76e30a81c8b8451898078e9636a1104b: stopping tablet replica
I20240503 08:11:48.162879 26549 raft_consensus.cc:2229] T 98dfd1367b224981ab0fd74678bb5ac2 P 76e30a81c8b8451898078e9636a1104b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:48.163583 26549 raft_consensus.cc:2258] T 98dfd1367b224981ab0fd74678bb5ac2 P 76e30a81c8b8451898078e9636a1104b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:48.164669 26549 ts_tablet_manager.cc:1875] T 98dfd1367b224981ab0fd74678bb5ac2 P 76e30a81c8b8451898078e9636a1104b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:11:48.169412 26549 ts_tablet_manager.cc:1888] T 98dfd1367b224981ab0fd74678bb5ac2 P 76e30a81c8b8451898078e9636a1104b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 08:11:48.169548 26549 log.cc:1196] T 98dfd1367b224981ab0fd74678bb5ac2 P 76e30a81c8b8451898078e9636a1104b: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/98dfd1367b224981ab0fd74678bb5ac2
I20240503 08:11:48.170393 23270 catalog_manager.cc:4830] TS 76e30a81c8b8451898078e9636a1104b (127.20.230.65:37573): tablet 98dfd1367b224981ab0fd74678bb5ac2 (table rebalance_test_table_4 [id=a233217db4ea40e88dd7f9aa60cca533]) successfully deleted
I20240503 08:11:48.247874 23537 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:48.254859 24902 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:48.262151 24641 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:48.264739 23407 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:48.266129 24772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:48.289456 25173 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:48.303972 25033 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:48.379426 24922 consensus_queue.cc:237] T 85b933bc55b1492fb70055a765a2b09b P 509c32504f544f8b90ebf66b6dd951f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "509c32504f544f8b90ebf66b6dd951f7" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 41435 } }
I20240503 08:11:48.380698 26075 raft_consensus.cc:2932] T 85b933bc55b1492fb70055a765a2b09b P 509c32504f544f8b90ebf66b6dd951f7 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 76e30a81c8b8451898078e9636a1104b (127.20.230.65) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "509c32504f544f8b90ebf66b6dd951f7" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 41435 } } }
I20240503 08:11:48.387246 23280 catalog_manager.cc:5484] T 85b933bc55b1492fb70055a765a2b09b P 509c32504f544f8b90ebf66b6dd951f7 reported cstate change: config changed from index 4 to 6, VOTER 76e30a81c8b8451898078e9636a1104b (127.20.230.65) evicted. New cstate: current_term: 2 leader_uuid: "509c32504f544f8b90ebf66b6dd951f7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "509c32504f544f8b90ebf66b6dd951f7" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 41435 } } }
I20240503 08:11:48.390677 23407 tablet_service.cc:1514] Processing DeleteTablet for tablet 85b933bc55b1492fb70055a765a2b09b with delete_type TABLET_DATA_TOMBSTONED (TS 76e30a81c8b8451898078e9636a1104b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41888
I20240503 08:11:48.390987 26549 tablet_replica.cc:330] T 85b933bc55b1492fb70055a765a2b09b P 76e30a81c8b8451898078e9636a1104b: stopping tablet replica
I20240503 08:11:48.391163 26549 raft_consensus.cc:2229] T 85b933bc55b1492fb70055a765a2b09b P 76e30a81c8b8451898078e9636a1104b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:48.391331 26549 raft_consensus.cc:2258] T 85b933bc55b1492fb70055a765a2b09b P 76e30a81c8b8451898078e9636a1104b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:48.391896 26549 ts_tablet_manager.cc:1875] T 85b933bc55b1492fb70055a765a2b09b P 76e30a81c8b8451898078e9636a1104b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:11:48.398938 26549 ts_tablet_manager.cc:1888] T 85b933bc55b1492fb70055a765a2b09b P 76e30a81c8b8451898078e9636a1104b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 08:11:48.399044 26549 log.cc:1196] T 85b933bc55b1492fb70055a765a2b09b P 76e30a81c8b8451898078e9636a1104b: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/85b933bc55b1492fb70055a765a2b09b
I20240503 08:11:48.399696 23270 catalog_manager.cc:4830] TS 76e30a81c8b8451898078e9636a1104b (127.20.230.65:37573): tablet 85b933bc55b1492fb70055a765a2b09b (table rebalance_test_table_4 [id=a233217db4ea40e88dd7f9aa60cca533]) successfully deleted
I20240503 08:11:48.488580 23537 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:48.497473 24772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:48.498498 24902 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:48.502480 24641 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:48.514811 23407 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:48.528721 25033 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:48.539549 25173 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:48.619920 24792 consensus_queue.cc:237] T 2449909504504d1e84cf34f1af673af2 P 51d0bc1740db4723a71f1d56b7e6e6b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "51d0bc1740db4723a71f1d56b7e6e6b7" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 44977 } }
I20240503 08:11:48.621027 26165 raft_consensus.cc:2932] T 2449909504504d1e84cf34f1af673af2 P 51d0bc1740db4723a71f1d56b7e6e6b7 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 30d88bcd65b347dabcbac54465e52b21 (127.20.230.66) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "51d0bc1740db4723a71f1d56b7e6e6b7" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 44977 } } }
I20240503 08:11:48.624897 23280 catalog_manager.cc:5484] T 2449909504504d1e84cf34f1af673af2 P 51d0bc1740db4723a71f1d56b7e6e6b7 reported cstate change: config changed from index 4 to 6, VOTER 30d88bcd65b347dabcbac54465e52b21 (127.20.230.66) evicted. New cstate: current_term: 2 leader_uuid: "51d0bc1740db4723a71f1d56b7e6e6b7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "51d0bc1740db4723a71f1d56b7e6e6b7" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 44977 } } }
I20240503 08:11:48.628242 23537 tablet_service.cc:1514] Processing DeleteTablet for tablet 2449909504504d1e84cf34f1af673af2 with delete_type TABLET_DATA_TOMBSTONED (TS 30d88bcd65b347dabcbac54465e52b21 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35234
I20240503 08:11:48.628849 26655 tablet_replica.cc:330] T 2449909504504d1e84cf34f1af673af2 P 30d88bcd65b347dabcbac54465e52b21: stopping tablet replica
I20240503 08:11:48.629015 26655 raft_consensus.cc:2229] T 2449909504504d1e84cf34f1af673af2 P 30d88bcd65b347dabcbac54465e52b21 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:48.629212 26655 raft_consensus.cc:2258] T 2449909504504d1e84cf34f1af673af2 P 30d88bcd65b347dabcbac54465e52b21 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:48.629948 26655 ts_tablet_manager.cc:1875] T 2449909504504d1e84cf34f1af673af2 P 30d88bcd65b347dabcbac54465e52b21: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:11:48.635251 26655 ts_tablet_manager.cc:1888] T 2449909504504d1e84cf34f1af673af2 P 30d88bcd65b347dabcbac54465e52b21: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 08:11:48.635354 26655 log.cc:1196] T 2449909504504d1e84cf34f1af673af2 P 30d88bcd65b347dabcbac54465e52b21: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/2449909504504d1e84cf34f1af673af2
I20240503 08:11:48.635990 23273 catalog_manager.cc:4830] TS 30d88bcd65b347dabcbac54465e52b21 (127.20.230.66:33351): tablet 2449909504504d1e84cf34f1af673af2 (table rebalance_test_table_1 [id=be175f2340dc4b2a947a393d8a2bd2c3]) successfully deleted
I20240503 08:11:48.742990 24902 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:48.744679 23537 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:48.745715 23407 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:48.755363 24772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:48.766029 25173 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:48.778779 24641 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:48.795758 25033 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:48.861886 24661 consensus_queue.cc:237] T 0d69b1adfa064beb82bdcc16d53717d7 P 5ea58b5961b54598878bad3f5660913c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5ea58b5961b54598878bad3f5660913c" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 37877 } }
I20240503 08:11:48.862967 26687 raft_consensus.cc:2932] T 0d69b1adfa064beb82bdcc16d53717d7 P 5ea58b5961b54598878bad3f5660913c [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 76e30a81c8b8451898078e9636a1104b (127.20.230.65) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5ea58b5961b54598878bad3f5660913c" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 37877 } } }
I20240503 08:11:48.866966 23282 catalog_manager.cc:5484] T 0d69b1adfa064beb82bdcc16d53717d7 P 5ea58b5961b54598878bad3f5660913c reported cstate change: config changed from index 4 to 6, VOTER 76e30a81c8b8451898078e9636a1104b (127.20.230.65) evicted. New cstate: current_term: 2 leader_uuid: "5ea58b5961b54598878bad3f5660913c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5ea58b5961b54598878bad3f5660913c" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 37877 } } }
I20240503 08:11:48.870342 23407 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d69b1adfa064beb82bdcc16d53717d7 with delete_type TABLET_DATA_TOMBSTONED (TS 76e30a81c8b8451898078e9636a1104b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41888
I20240503 08:11:48.870668 26549 tablet_replica.cc:330] T 0d69b1adfa064beb82bdcc16d53717d7 P 76e30a81c8b8451898078e9636a1104b: stopping tablet replica
I20240503 08:11:48.870800 26549 raft_consensus.cc:2229] T 0d69b1adfa064beb82bdcc16d53717d7 P 76e30a81c8b8451898078e9636a1104b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:48.870939 26549 raft_consensus.cc:2258] T 0d69b1adfa064beb82bdcc16d53717d7 P 76e30a81c8b8451898078e9636a1104b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:48.871448 26549 ts_tablet_manager.cc:1875] T 0d69b1adfa064beb82bdcc16d53717d7 P 76e30a81c8b8451898078e9636a1104b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:11:48.875833 26549 ts_tablet_manager.cc:1888] T 0d69b1adfa064beb82bdcc16d53717d7 P 76e30a81c8b8451898078e9636a1104b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 08:11:48.875967 26549 log.cc:1196] T 0d69b1adfa064beb82bdcc16d53717d7 P 76e30a81c8b8451898078e9636a1104b: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/0d69b1adfa064beb82bdcc16d53717d7
I20240503 08:11:48.876691 23270 catalog_manager.cc:4830] TS 76e30a81c8b8451898078e9636a1104b (127.20.230.65:37573): tablet 0d69b1adfa064beb82bdcc16d53717d7 (table rebalance_test_table_2 [id=9e0052fa5a20450fad9ea538bf678fcd]) successfully deleted
I20240503 08:11:48.959110 23537 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:48.961441 24902 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:48.976856 24641 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:48.977558 24772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:48.979146 23407 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:49.009611 25173 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:49.036516 25033 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:49.118880 25193 consensus_queue.cc:237] T 8c6217accd494f5182a862c31013b76e P 7823cb7f30b045209bc5ccfb110bf199 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7823cb7f30b045209bc5ccfb110bf199" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 38073 } }
I20240503 08:11:49.120069 26610 raft_consensus.cc:2932] T 8c6217accd494f5182a862c31013b76e P 7823cb7f30b045209bc5ccfb110bf199 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 30d88bcd65b347dabcbac54465e52b21 (127.20.230.66) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7823cb7f30b045209bc5ccfb110bf199" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 38073 } } }
I20240503 08:11:49.125142 23285 catalog_manager.cc:5484] T 8c6217accd494f5182a862c31013b76e P 7823cb7f30b045209bc5ccfb110bf199 reported cstate change: config changed from index 4 to 6, VOTER 30d88bcd65b347dabcbac54465e52b21 (127.20.230.66) evicted. New cstate: current_term: 2 leader_uuid: "7823cb7f30b045209bc5ccfb110bf199" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7823cb7f30b045209bc5ccfb110bf199" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 38073 } } }
I20240503 08:11:49.128719 23537 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c6217accd494f5182a862c31013b76e with delete_type TABLET_DATA_TOMBSTONED (TS 30d88bcd65b347dabcbac54465e52b21 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35234
I20240503 08:11:49.129031 26655 tablet_replica.cc:330] T 8c6217accd494f5182a862c31013b76e P 30d88bcd65b347dabcbac54465e52b21: stopping tablet replica
I20240503 08:11:49.129205 26655 raft_consensus.cc:2229] T 8c6217accd494f5182a862c31013b76e P 30d88bcd65b347dabcbac54465e52b21 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:49.129381 26655 raft_consensus.cc:2258] T 8c6217accd494f5182a862c31013b76e P 30d88bcd65b347dabcbac54465e52b21 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:49.129979 26655 ts_tablet_manager.cc:1875] T 8c6217accd494f5182a862c31013b76e P 30d88bcd65b347dabcbac54465e52b21: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:11:49.134510 26655 ts_tablet_manager.cc:1888] T 8c6217accd494f5182a862c31013b76e P 30d88bcd65b347dabcbac54465e52b21: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 08:11:49.134617 26655 log.cc:1196] T 8c6217accd494f5182a862c31013b76e P 30d88bcd65b347dabcbac54465e52b21: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/8c6217accd494f5182a862c31013b76e
I20240503 08:11:49.135246 23273 catalog_manager.cc:4830] TS 30d88bcd65b347dabcbac54465e52b21 (127.20.230.66:33351): tablet 8c6217accd494f5182a862c31013b76e (table rebalance_test_table_3 [id=798c4a054bb5441485fcde0049107bdc]) successfully deleted
I20240503 08:11:49.212689 23537 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:49.222041 24902 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:49.251391 24772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:49.251663 24641 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:49.265844 23407 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:49.276724 25033 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:49.283620 25173 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:49.362629 25053 consensus_queue.cc:237] T 29dea2294fd74738a55ac1bd5b88121b P c82f2e5e9e13405d9328620179077098 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c82f2e5e9e13405d9328620179077098" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 36811 } }
I20240503 08:11:49.363718 26803 raft_consensus.cc:2932] T 29dea2294fd74738a55ac1bd5b88121b P c82f2e5e9e13405d9328620179077098 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 76e30a81c8b8451898078e9636a1104b (127.20.230.65) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c82f2e5e9e13405d9328620179077098" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 36811 } } }
I20240503 08:11:49.368149 23283 catalog_manager.cc:5484] T 29dea2294fd74738a55ac1bd5b88121b P c82f2e5e9e13405d9328620179077098 reported cstate change: config changed from index 4 to 6, VOTER 76e30a81c8b8451898078e9636a1104b (127.20.230.65) evicted. New cstate: current_term: 2 leader_uuid: "c82f2e5e9e13405d9328620179077098" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c82f2e5e9e13405d9328620179077098" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 36811 } } }
I20240503 08:11:49.371762 23407 tablet_service.cc:1514] Processing DeleteTablet for tablet 29dea2294fd74738a55ac1bd5b88121b with delete_type TABLET_DATA_TOMBSTONED (TS 76e30a81c8b8451898078e9636a1104b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41888
I20240503 08:11:49.372143 26549 tablet_replica.cc:330] T 29dea2294fd74738a55ac1bd5b88121b P 76e30a81c8b8451898078e9636a1104b: stopping tablet replica
I20240503 08:11:49.372370 26549 raft_consensus.cc:2229] T 29dea2294fd74738a55ac1bd5b88121b P 76e30a81c8b8451898078e9636a1104b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:49.372562 26549 raft_consensus.cc:2258] T 29dea2294fd74738a55ac1bd5b88121b P 76e30a81c8b8451898078e9636a1104b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:49.373257 26549 ts_tablet_manager.cc:1875] T 29dea2294fd74738a55ac1bd5b88121b P 76e30a81c8b8451898078e9636a1104b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:11:49.378859 26549 ts_tablet_manager.cc:1888] T 29dea2294fd74738a55ac1bd5b88121b P 76e30a81c8b8451898078e9636a1104b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 08:11:49.379043 26549 log.cc:1196] T 29dea2294fd74738a55ac1bd5b88121b P 76e30a81c8b8451898078e9636a1104b: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/29dea2294fd74738a55ac1bd5b88121b
I20240503 08:11:49.379773 23270 catalog_manager.cc:4830] TS 76e30a81c8b8451898078e9636a1104b (127.20.230.65:37573): tablet 29dea2294fd74738a55ac1bd5b88121b (table rebalance_test_table_3 [id=798c4a054bb5441485fcde0049107bdc]) successfully deleted
I20240503 08:11:49.458956 23537 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:49.471906 24902 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:49.479880 24772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:49.486282 24641 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:49.491669 23407 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:49.501109 25033 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:49.503018 25173 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:49.606138 24922 consensus_queue.cc:237] T 78b4f0eae3d14b97b8f74cbc24389044 P 509c32504f544f8b90ebf66b6dd951f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "509c32504f544f8b90ebf66b6dd951f7" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 41435 } }
I20240503 08:11:49.607506 26868 raft_consensus.cc:2932] T 78b4f0eae3d14b97b8f74cbc24389044 P 509c32504f544f8b90ebf66b6dd951f7 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 76e30a81c8b8451898078e9636a1104b (127.20.230.65) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "509c32504f544f8b90ebf66b6dd951f7" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 41435 } } }
I20240503 08:11:49.611850 23283 catalog_manager.cc:5484] T 78b4f0eae3d14b97b8f74cbc24389044 P 509c32504f544f8b90ebf66b6dd951f7 reported cstate change: config changed from index 4 to 6, VOTER 76e30a81c8b8451898078e9636a1104b (127.20.230.65) evicted. New cstate: current_term: 2 leader_uuid: "509c32504f544f8b90ebf66b6dd951f7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "509c32504f544f8b90ebf66b6dd951f7" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 41435 } } }
I20240503 08:11:49.615409 23407 tablet_service.cc:1514] Processing DeleteTablet for tablet 78b4f0eae3d14b97b8f74cbc24389044 with delete_type TABLET_DATA_TOMBSTONED (TS 76e30a81c8b8451898078e9636a1104b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41888
I20240503 08:11:49.615759 26549 tablet_replica.cc:330] T 78b4f0eae3d14b97b8f74cbc24389044 P 76e30a81c8b8451898078e9636a1104b: stopping tablet replica
I20240503 08:11:49.615965 26549 raft_consensus.cc:2229] T 78b4f0eae3d14b97b8f74cbc24389044 P 76e30a81c8b8451898078e9636a1104b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:49.616150 26549 raft_consensus.cc:2258] T 78b4f0eae3d14b97b8f74cbc24389044 P 76e30a81c8b8451898078e9636a1104b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:49.616881 26549 ts_tablet_manager.cc:1875] T 78b4f0eae3d14b97b8f74cbc24389044 P 76e30a81c8b8451898078e9636a1104b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:11:49.623462 26549 ts_tablet_manager.cc:1888] T 78b4f0eae3d14b97b8f74cbc24389044 P 76e30a81c8b8451898078e9636a1104b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 08:11:49.623595 26549 log.cc:1196] T 78b4f0eae3d14b97b8f74cbc24389044 P 76e30a81c8b8451898078e9636a1104b: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/78b4f0eae3d14b97b8f74cbc24389044
I20240503 08:11:49.624377 23270 catalog_manager.cc:4830] TS 76e30a81c8b8451898078e9636a1104b (127.20.230.65:37573): tablet 78b4f0eae3d14b97b8f74cbc24389044 (table rebalance_test_table_0 [id=e75f9b41b7294dddba472b9ede90ddcb]) successfully deleted
I20240503 08:11:49.999679 24772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:50.009253 23537 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:50.012573 24902 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:50.019359 25173 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:50.034494 24641 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:50.040117 23407 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:50.058509 25033 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:50.214823 23537 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:50.230993 24772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:50.232000 24902 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:50.247429 25173 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:50.251375 25033 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:50.257959 24641 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:50.270045 23407 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:50.971949 23537 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:50.972268 24902 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:50.975656 24772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:51.012881 23407 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:51.033114 24641 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:51.060529 25173 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:51.094100 25033 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:51.234445 23537 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:51.246155 24772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:51.254134 23407 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:51.256095 24641 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:51.281502 24902 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:51.291946 25033 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:51.298449 25173 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:51.554584 23537 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:51.558809 24902 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:51.566015 24772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:51.569171 24641 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:51.597133 25173 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:51.600039 25033 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:51.603865 23407 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
0842171f0fd24fa69ee8e4099dc61148 | 127.20.230.126:38671 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.20.230.84:40151 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
30d88bcd65b347dabcbac54465e52b21 | 127.20.230.66:33351 | HEALTHY | <none> | 3 | 0
509c32504f544f8b90ebf66b6dd951f7 | 127.20.230.69:41435 | HEALTHY | <none> | 2 | 0
51d0bc1740db4723a71f1d56b7e6e6b7 | 127.20.230.68:44977 | HEALTHY | <none> | 2 | 0
5ea58b5961b54598878bad3f5660913c | 127.20.230.67:37877 | HEALTHY | <none> | 2 | 0
76e30a81c8b8451898078e9636a1104b | 127.20.230.65:37573 | HEALTHY | <none> | 3 | 0
7823cb7f30b045209bc5ccfb110bf199 | 127.20.230.71:38073 | HEALTHY | <none> | 2 | 0
c82f2e5e9e13405d9328620179077098 | 127.20.230.70:36811 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.20.230.65 | experimental | 127.20.230.65:37573
local_ip_for_outbound_sockets | 127.20.230.66 | experimental | 127.20.230.66:33351
local_ip_for_outbound_sockets | 127.20.230.67 | experimental | 127.20.230.67:37877
local_ip_for_outbound_sockets | 127.20.230.68 | experimental | 127.20.230.68:44977
local_ip_for_outbound_sockets | 127.20.230.69 | experimental | 127.20.230.69:41435
local_ip_for_outbound_sockets | 127.20.230.70 | experimental | 127.20.230.70:36811
local_ip_for_outbound_sockets | 127.20.230.71 | experimental | 127.20.230.71:38073
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.20.230.65:37573
server_dump_info_path | /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.20.230.66:33351
server_dump_info_path | /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.20.230.67:37877
server_dump_info_path | /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.20.230.68:44977
server_dump_info_path | /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.20.230.69:41435
server_dump_info_path | /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.20.230.70:36811
server_dump_info_path | /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.20.230.71:38073
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.20.230.84:40151 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 08:11:51.670604 21401 log_verifier.cc:126] Checking tablet 0d69b1adfa064beb82bdcc16d53717d7
I20240503 08:11:51.685415 21401 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 0d69b1adfa064beb82bdcc16d53717d7
I20240503 08:11:51.685588 21401 log_verifier.cc:126] Checking tablet 2449909504504d1e84cf34f1af673af2
I20240503 08:11:51.696903 21401 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 2449909504504d1e84cf34f1af673af2
I20240503 08:11:51.696995 21401 log_verifier.cc:126] Checking tablet 2797c89387af4da5aba9d7d6657eb275
I20240503 08:11:51.708628 21401 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 2797c89387af4da5aba9d7d6657eb275
I20240503 08:11:51.708717 21401 log_verifier.cc:126] Checking tablet 29dea2294fd74738a55ac1bd5b88121b
I20240503 08:11:51.720400 21401 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 29dea2294fd74738a55ac1bd5b88121b
I20240503 08:11:51.720499 21401 log_verifier.cc:126] Checking tablet 43fb21a411474f62afad731b47bad5a9
I20240503 08:11:51.731972 21401 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 43fb21a411474f62afad731b47bad5a9
I20240503 08:11:51.732069 21401 log_verifier.cc:126] Checking tablet 494f9374df1d45f5a0a8f7705e72557a
I20240503 08:11:51.743544 21401 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 494f9374df1d45f5a0a8f7705e72557a
I20240503 08:11:51.743631 21401 log_verifier.cc:126] Checking tablet 6bae8d67e9894646aae3f6230eda07c7
I20240503 08:11:51.754815 21401 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 6bae8d67e9894646aae3f6230eda07c7
I20240503 08:11:51.754904 21401 log_verifier.cc:126] Checking tablet 6bb8e518a6b6496fb52f41f0b6914ff4
I20240503 08:11:51.765818 21401 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 6bb8e518a6b6496fb52f41f0b6914ff4
I20240503 08:11:51.765901 21401 log_verifier.cc:126] Checking tablet 7878470e8ac949bb8971fe8367f7db87
I20240503 08:11:51.776932 21401 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 7878470e8ac949bb8971fe8367f7db87
I20240503 08:11:51.777021 21401 log_verifier.cc:126] Checking tablet 7879251014c84b11a0f49d707d80d85b
I20240503 08:11:51.784046 21401 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 7879251014c84b11a0f49d707d80d85b
I20240503 08:11:51.784137 21401 log_verifier.cc:126] Checking tablet 78b4f0eae3d14b97b8f74cbc24389044
I20240503 08:11:51.795585 21401 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 78b4f0eae3d14b97b8f74cbc24389044
I20240503 08:11:51.795676 21401 log_verifier.cc:126] Checking tablet 85b933bc55b1492fb70055a765a2b09b
I20240503 08:11:51.807642 21401 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 85b933bc55b1492fb70055a765a2b09b
I20240503 08:11:51.807754 21401 log_verifier.cc:126] Checking tablet 8c6217accd494f5182a862c31013b76e
I20240503 08:11:51.819360 21401 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 8c6217accd494f5182a862c31013b76e
I20240503 08:11:51.819456 21401 log_verifier.cc:126] Checking tablet 98dfd1367b224981ab0fd74678bb5ac2
I20240503 08:11:51.831408 21401 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 98dfd1367b224981ab0fd74678bb5ac2
I20240503 08:11:51.831508 21401 log_verifier.cc:126] Checking tablet b19203edb44242acb3ad7dfbea6459a2
I20240503 08:11:51.842679 21401 log_verifier.cc:177] Verified matching terms for 3 ops in tablet b19203edb44242acb3ad7dfbea6459a2
I20240503 08:11:51.842765 21401 log_verifier.cc:126] Checking tablet b2779eae1c08452ca7b5b0874bbac47e
I20240503 08:11:51.855894 21401 log_verifier.cc:177] Verified matching terms for 6 ops in tablet b2779eae1c08452ca7b5b0874bbac47e
I20240503 08:11:51.859757 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.72:0
--local_ip_for_outbound_sockets=127.20.230.72
--webserver_interface=127.20.230.72
--webserver_port=0
--tserver_master_addrs=127.20.230.126:38671
--builtin_ntp_servers=127.20.230.84:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 08:11:52.189091 27093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:52.189414 27093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:52.189538 27093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:52.197017 27093 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:11:52.197664 27093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:52.197868 27093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.72
I20240503 08:11:52.206959 27093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.20.230.72
--webserver_port=0
--tserver_master_addrs=127.20.230.126:38671
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.72
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:52.208317 27093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:52.210182 27093 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:52.222664 27098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:52.223364 27099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:52.223898 27101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:52.224669 27093 server_base.cc:1034] running on GCE node
I20240503 08:11:52.225260 27093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:52.226253 27093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:52.227445 27093 hybrid_clock.cc:648] HybridClock initialized: now 1714723912227383 us; error 54 us; skew 500 ppm
I20240503 08:11:52.227777 27093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:52.229984 27093 webserver.cc:460] Webserver started at http://127.20.230.72:39839/ using document root <none> and password file <none>
I20240503 08:11:52.230731 27093 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:52.230836 27093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:52.231148 27093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:52.234329 27093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "0188f247aeb8461bade4be31a5251288"
format_stamp: "Formatted at 2024-05-03 08:11:52 on dist-test-slave-6vp8"
I20240503 08:11:52.235118 27093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "0188f247aeb8461bade4be31a5251288"
format_stamp: "Formatted at 2024-05-03 08:11:52 on dist-test-slave-6vp8"
I20240503 08:11:52.240554 27093 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 08:11:52.244120 27107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:52.245481 27093 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240503 08:11:52.245627 27093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "0188f247aeb8461bade4be31a5251288"
format_stamp: "Formatted at 2024-05-03 08:11:52 on dist-test-slave-6vp8"
I20240503 08:11:52.245864 27093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:52.278317 27093 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:52.279233 27093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:52.279481 27093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:52.280488 27093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:52.282003 27093 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:52.282089 27093 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:52.282167 27093 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:52.282217 27093 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:52.315702 27093 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.72:38171
I20240503 08:11:52.315755 27219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.72:38171 every 8 connection(s)
I20240503 08:11:52.316964 27093 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 08:11:52.326773 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 27093
I20240503 08:11:52.327013 21401 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 08:11:52.329629 27220 heartbeater.cc:347] Connected to a master server at 127.20.230.126:38671
I20240503 08:11:52.329998 27220 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:52.330899 27220 heartbeater.cc:511] Master 127.20.230.126:38671 requested a full tablet report, sending...
I20240503 08:11:52.332325 23285 ts_manager.cc:194] Registered new tserver with Master: 0188f247aeb8461bade4be31a5251288 (127.20.230.72:38171)
I20240503 08:11:52.333110 23285 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.72:46473
I20240503 08:11:52.818194 27155 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:52.822288 24902 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:52.848531 23537 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:52.853358 23407 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:52.861785 24772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:52.862936 24641 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:52.889916 25173 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:52.913275 25033 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:53.092636 27155 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:53.099470 24902 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:53.115561 23537 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:53.116840 24772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:53.130532 24641 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:53.162887 25033 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:53.165326 25173 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:53.165921 23407 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:53.264994 23557 consensus_queue.cc:237] T 2797c89387af4da5aba9d7d6657eb275 P 30d88bcd65b347dabcbac54465e52b21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "0188f247aeb8461bade4be31a5251288" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 38171 } }
W20240503 08:11:53.273232 23491 consensus_peers.cc:485] T 2797c89387af4da5aba9d7d6657eb275 P 30d88bcd65b347dabcbac54465e52b21 -> Peer 0188f247aeb8461bade4be31a5251288 (127.20.230.72:38171): Couldn't send request to peer 0188f247aeb8461bade4be31a5251288. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2797c89387af4da5aba9d7d6657eb275. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:53.335439 27220 heartbeater.cc:503] Master 127.20.230.126:38671 was elected leader, sending a full tablet report...
I20240503 08:11:53.661721 27325 ts_tablet_manager.cc:902] T 2797c89387af4da5aba9d7d6657eb275 P 0188f247aeb8461bade4be31a5251288: Initiating tablet copy from peer 30d88bcd65b347dabcbac54465e52b21 (127.20.230.66:33351)
I20240503 08:11:53.663414 27325 tablet_copy_client.cc:308] T 2797c89387af4da5aba9d7d6657eb275 P 0188f247aeb8461bade4be31a5251288: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.66:33351
I20240503 08:11:53.669284 23577 tablet_copy_service.cc:140] P 30d88bcd65b347dabcbac54465e52b21: Received BeginTabletCopySession request for tablet 2797c89387af4da5aba9d7d6657eb275 from peer 0188f247aeb8461bade4be31a5251288 ({username='slave'} at 127.20.230.72:38825)
I20240503 08:11:53.669422 23577 tablet_copy_service.cc:161] P 30d88bcd65b347dabcbac54465e52b21: Beginning new tablet copy session on tablet 2797c89387af4da5aba9d7d6657eb275 from peer 0188f247aeb8461bade4be31a5251288 at {username='slave'} at 127.20.230.72:38825: session id = 0188f247aeb8461bade4be31a5251288-2797c89387af4da5aba9d7d6657eb275
I20240503 08:11:53.671419 23577 tablet_copy_source_session.cc:205] T 2797c89387af4da5aba9d7d6657eb275 P 30d88bcd65b347dabcbac54465e52b21: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:53.675526 27325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2797c89387af4da5aba9d7d6657eb275. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:53.684988 27325 tablet_copy_client.cc:783] T 2797c89387af4da5aba9d7d6657eb275 P 0188f247aeb8461bade4be31a5251288: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:53.685326 27325 tablet_copy_client.cc:647] T 2797c89387af4da5aba9d7d6657eb275 P 0188f247aeb8461bade4be31a5251288: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:53.687306 27325 tablet_copy_client.cc:523] T 2797c89387af4da5aba9d7d6657eb275 P 0188f247aeb8461bade4be31a5251288: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:53.690207 27325 tablet_bootstrap.cc:492] T 2797c89387af4da5aba9d7d6657eb275 P 0188f247aeb8461bade4be31a5251288: Bootstrap starting.
I20240503 08:11:53.699097 27325 log.cc:824] T 2797c89387af4da5aba9d7d6657eb275 P 0188f247aeb8461bade4be31a5251288: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:53.727808 27325 tablet_bootstrap.cc:492] T 2797c89387af4da5aba9d7d6657eb275 P 0188f247aeb8461bade4be31a5251288: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:11:53.728317 27325 tablet_bootstrap.cc:492] T 2797c89387af4da5aba9d7d6657eb275 P 0188f247aeb8461bade4be31a5251288: Bootstrap complete.
I20240503 08:11:53.728694 27325 ts_tablet_manager.cc:1367] T 2797c89387af4da5aba9d7d6657eb275 P 0188f247aeb8461bade4be31a5251288: Time spent bootstrapping tablet: real 0.039s user 0.028s sys 0.008s
I20240503 08:11:53.732925 27325 raft_consensus.cc:348] T 2797c89387af4da5aba9d7d6657eb275 P 0188f247aeb8461bade4be31a5251288 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } }
I20240503 08:11:53.733712 27325 raft_consensus.cc:729] T 2797c89387af4da5aba9d7d6657eb275 P 0188f247aeb8461bade4be31a5251288 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0188f247aeb8461bade4be31a5251288, State: Initialized, Role: FOLLOWER
I20240503 08:11:53.734136 27325 consensus_queue.cc:260] T 2797c89387af4da5aba9d7d6657eb275 P 0188f247aeb8461bade4be31a5251288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "0188f247aeb8461bade4be31a5251288" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 38171 } }
I20240503 08:11:53.735605 27325 ts_tablet_manager.cc:1398] T 2797c89387af4da5aba9d7d6657eb275 P 0188f247aeb8461bade4be31a5251288: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
I20240503 08:11:53.736603 23577 tablet_copy_service.cc:342] P 30d88bcd65b347dabcbac54465e52b21: Request end of tablet copy session 0188f247aeb8461bade4be31a5251288-2797c89387af4da5aba9d7d6657eb275 received from {username='slave'} at 127.20.230.72:38825
I20240503 08:11:53.736747 23577 tablet_copy_service.cc:433] P 30d88bcd65b347dabcbac54465e52b21: ending tablet copy session 0188f247aeb8461bade4be31a5251288-2797c89387af4da5aba9d7d6657eb275 on tablet 2797c89387af4da5aba9d7d6657eb275 with peer 0188f247aeb8461bade4be31a5251288
I20240503 08:11:54.188439 27175 raft_consensus.cc:1203] T 2797c89387af4da5aba9d7d6657eb275 P 0188f247aeb8461bade4be31a5251288 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:11:54.190050 27334 raft_consensus.cc:2932] T 2797c89387af4da5aba9d7d6657eb275 P 30d88bcd65b347dabcbac54465e52b21 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0188f247aeb8461bade4be31a5251288 (127.20.230.72) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "0188f247aeb8461bade4be31a5251288" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 38171 } } }
I20240503 08:11:54.190693 27175 raft_consensus.cc:2932] T 2797c89387af4da5aba9d7d6657eb275 P 0188f247aeb8461bade4be31a5251288 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0188f247aeb8461bade4be31a5251288 (127.20.230.72) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "0188f247aeb8461bade4be31a5251288" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 38171 } } }
I20240503 08:11:54.195642 23283 catalog_manager.cc:5484] T 2797c89387af4da5aba9d7d6657eb275 P 0188f247aeb8461bade4be31a5251288 reported cstate change: config changed from index -1 to 4, VOTER 0188f247aeb8461bade4be31a5251288 (127.20.230.72) added. New cstate: current_term: 1 leader_uuid: "30d88bcd65b347dabcbac54465e52b21" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "0188f247aeb8461bade4be31a5251288" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 38171 } } }
I20240503 08:11:54.227730 23427 consensus_queue.cc:237] T b19203edb44242acb3ad7dfbea6459a2 P 76e30a81c8b8451898078e9636a1104b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "0188f247aeb8461bade4be31a5251288" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 38171 } }
W20240503 08:11:54.235683 23361 consensus_peers.cc:485] T b19203edb44242acb3ad7dfbea6459a2 P 76e30a81c8b8451898078e9636a1104b -> Peer 0188f247aeb8461bade4be31a5251288 (127.20.230.72:38171): Couldn't send request to peer 0188f247aeb8461bade4be31a5251288. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b19203edb44242acb3ad7dfbea6459a2. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:54.710600 27361 ts_tablet_manager.cc:902] T b19203edb44242acb3ad7dfbea6459a2 P 0188f247aeb8461bade4be31a5251288: Initiating tablet copy from peer 76e30a81c8b8451898078e9636a1104b (127.20.230.65:37573)
I20240503 08:11:54.711604 27361 tablet_copy_client.cc:308] T b19203edb44242acb3ad7dfbea6459a2 P 0188f247aeb8461bade4be31a5251288: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.65:37573
I20240503 08:11:54.717082 23447 tablet_copy_service.cc:140] P 76e30a81c8b8451898078e9636a1104b: Received BeginTabletCopySession request for tablet b19203edb44242acb3ad7dfbea6459a2 from peer 0188f247aeb8461bade4be31a5251288 ({username='slave'} at 127.20.230.72:49487)
I20240503 08:11:54.717245 23447 tablet_copy_service.cc:161] P 76e30a81c8b8451898078e9636a1104b: Beginning new tablet copy session on tablet b19203edb44242acb3ad7dfbea6459a2 from peer 0188f247aeb8461bade4be31a5251288 at {username='slave'} at 127.20.230.72:49487: session id = 0188f247aeb8461bade4be31a5251288-b19203edb44242acb3ad7dfbea6459a2
I20240503 08:11:54.719318 23447 tablet_copy_source_session.cc:205] T b19203edb44242acb3ad7dfbea6459a2 P 76e30a81c8b8451898078e9636a1104b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:54.720499 27361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b19203edb44242acb3ad7dfbea6459a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:54.725037 27361 tablet_copy_client.cc:783] T b19203edb44242acb3ad7dfbea6459a2 P 0188f247aeb8461bade4be31a5251288: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:54.725250 27361 tablet_copy_client.cc:647] T b19203edb44242acb3ad7dfbea6459a2 P 0188f247aeb8461bade4be31a5251288: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:54.726688 27361 tablet_copy_client.cc:523] T b19203edb44242acb3ad7dfbea6459a2 P 0188f247aeb8461bade4be31a5251288: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:54.728839 27361 tablet_bootstrap.cc:492] T b19203edb44242acb3ad7dfbea6459a2 P 0188f247aeb8461bade4be31a5251288: Bootstrap starting.
I20240503 08:11:54.757712 27361 tablet_bootstrap.cc:492] T b19203edb44242acb3ad7dfbea6459a2 P 0188f247aeb8461bade4be31a5251288: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:11:54.758075 27361 tablet_bootstrap.cc:492] T b19203edb44242acb3ad7dfbea6459a2 P 0188f247aeb8461bade4be31a5251288: Bootstrap complete.
I20240503 08:11:54.758292 27361 ts_tablet_manager.cc:1367] T b19203edb44242acb3ad7dfbea6459a2 P 0188f247aeb8461bade4be31a5251288: Time spent bootstrapping tablet: real 0.030s user 0.031s sys 0.000s
I20240503 08:11:54.759008 27361 raft_consensus.cc:348] T b19203edb44242acb3ad7dfbea6459a2 P 0188f247aeb8461bade4be31a5251288 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } }
I20240503 08:11:54.759315 27361 raft_consensus.cc:729] T b19203edb44242acb3ad7dfbea6459a2 P 0188f247aeb8461bade4be31a5251288 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0188f247aeb8461bade4be31a5251288, State: Initialized, Role: FOLLOWER
I20240503 08:11:54.759459 27361 consensus_queue.cc:260] T b19203edb44242acb3ad7dfbea6459a2 P 0188f247aeb8461bade4be31a5251288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "0188f247aeb8461bade4be31a5251288" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 38171 } }
I20240503 08:11:54.760123 27361 ts_tablet_manager.cc:1398] T b19203edb44242acb3ad7dfbea6459a2 P 0188f247aeb8461bade4be31a5251288: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:54.760881 23447 tablet_copy_service.cc:342] P 76e30a81c8b8451898078e9636a1104b: Request end of tablet copy session 0188f247aeb8461bade4be31a5251288-b19203edb44242acb3ad7dfbea6459a2 received from {username='slave'} at 127.20.230.72:49487
I20240503 08:11:54.760993 23447 tablet_copy_service.cc:433] P 76e30a81c8b8451898078e9636a1104b: ending tablet copy session 0188f247aeb8461bade4be31a5251288-b19203edb44242acb3ad7dfbea6459a2 on tablet b19203edb44242acb3ad7dfbea6459a2 with peer 0188f247aeb8461bade4be31a5251288
I20240503 08:11:55.304294 27175 raft_consensus.cc:1203] T b19203edb44242acb3ad7dfbea6459a2 P 0188f247aeb8461bade4be31a5251288 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:11:55.305830 27372 raft_consensus.cc:2932] T b19203edb44242acb3ad7dfbea6459a2 P 76e30a81c8b8451898078e9636a1104b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0188f247aeb8461bade4be31a5251288 (127.20.230.72) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "0188f247aeb8461bade4be31a5251288" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 38171 } } }
I20240503 08:11:55.306468 27175 raft_consensus.cc:2932] T b19203edb44242acb3ad7dfbea6459a2 P 0188f247aeb8461bade4be31a5251288 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0188f247aeb8461bade4be31a5251288 (127.20.230.72) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "0188f247aeb8461bade4be31a5251288" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 38171 } } }
I20240503 08:11:55.308878 27372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:55462 (request call id 0) took 1081 ms. Trace:
I20240503 08:11:55.309099 27372 rpcz_store.cc:276] 0503 08:11:54.227191 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 08:11:54.227265 (+ 74us) service_pool.cc:225] Handling call
0503 08:11:54.228416 (+ 1151us) log.cc:842] Serialized 260 byte log entry
0503 08:11:55.308823 (+1080407us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":66,"raft.queue_time_us":342,"raft.run_cpu_time_us":313,"raft.run_wall_time_us":322,"spinlock_wait_cycles":201600,"thread_start_us":457,"threads_started":2,"wal-append.queue_time_us":298,"wal-append.run_cpu_time_us":544,"wal-append.run_wall_time_us":1000620}
I20240503 08:11:55.311715 23282 catalog_manager.cc:5484] T b19203edb44242acb3ad7dfbea6459a2 P 76e30a81c8b8451898078e9636a1104b reported cstate change: config changed from index -1 to 4, VOTER 0188f247aeb8461bade4be31a5251288 (127.20.230.72) added. New cstate: current_term: 1 leader_uuid: "76e30a81c8b8451898078e9636a1104b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "0188f247aeb8461bade4be31a5251288" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 38171 } } }
I20240503 08:11:55.436902 27155 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:55.453326 24772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:55.460579 24641 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:55.470746 24902 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:55.472563 23537 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:55.476465 23407 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:55.497658 25173 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:55.539129 25033 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:55.578388 23427 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b19203edb44242acb3ad7dfbea6459a2"
dest_uuid: "76e30a81c8b8451898078e9636a1104b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55492
I20240503 08:11:55.578634 23427 raft_consensus.cc:595] T b19203edb44242acb3ad7dfbea6459a2 P 76e30a81c8b8451898078e9636a1104b [term 1 LEADER]: Received request to transfer leadership
I20240503 08:11:55.672613 24902 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:55.676975 23537 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:55.698210 27155 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:55.701401 23407 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:55.702792 25173 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:55.726855 24772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:55.732774 24641 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:55.736827 25033 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:55.849440 27487 raft_consensus.cc:979] T b19203edb44242acb3ad7dfbea6459a2 P 76e30a81c8b8451898078e9636a1104b: : Instructing follower 0188f247aeb8461bade4be31a5251288 to start an election
I20240503 08:11:55.849594 27487 raft_consensus.cc:1067] T b19203edb44242acb3ad7dfbea6459a2 P 76e30a81c8b8451898078e9636a1104b [term 1 LEADER]: Signalling peer 0188f247aeb8461bade4be31a5251288 to start an election
I20240503 08:11:55.850384 27175 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b19203edb44242acb3ad7dfbea6459a2"
dest_uuid: "0188f247aeb8461bade4be31a5251288"
from {username='slave'} at 127.20.230.65:41957
I20240503 08:11:55.850620 27175 raft_consensus.cc:482] T b19203edb44242acb3ad7dfbea6459a2 P 0188f247aeb8461bade4be31a5251288 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:11:55.850801 27175 raft_consensus.cc:3037] T b19203edb44242acb3ad7dfbea6459a2 P 0188f247aeb8461bade4be31a5251288 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:55.853518 27175 raft_consensus.cc:504] T b19203edb44242acb3ad7dfbea6459a2 P 0188f247aeb8461bade4be31a5251288 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "0188f247aeb8461bade4be31a5251288" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 38171 } }
I20240503 08:11:55.854948 27175 leader_election.cc:288] T b19203edb44242acb3ad7dfbea6459a2 P 0188f247aeb8461bade4be31a5251288 [CANDIDATE]: Term 2 election: Requested vote from peers 76e30a81c8b8451898078e9636a1104b (127.20.230.65:37573)
I20240503 08:11:55.855589 23427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19203edb44242acb3ad7dfbea6459a2" candidate_uuid: "0188f247aeb8461bade4be31a5251288" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "76e30a81c8b8451898078e9636a1104b"
I20240503 08:11:55.855789 23427 raft_consensus.cc:3032] T b19203edb44242acb3ad7dfbea6459a2 P 76e30a81c8b8451898078e9636a1104b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:11:55.855866 23427 raft_consensus.cc:729] T b19203edb44242acb3ad7dfbea6459a2 P 76e30a81c8b8451898078e9636a1104b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 76e30a81c8b8451898078e9636a1104b, State: Running, Role: LEADER
I20240503 08:11:55.856132 23427 consensus_queue.cc:260] T b19203edb44242acb3ad7dfbea6459a2 P 76e30a81c8b8451898078e9636a1104b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "0188f247aeb8461bade4be31a5251288" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 38171 } }
I20240503 08:11:55.856422 23427 raft_consensus.cc:3037] T b19203edb44242acb3ad7dfbea6459a2 P 76e30a81c8b8451898078e9636a1104b [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:55.859014 23427 raft_consensus.cc:2454] T b19203edb44242acb3ad7dfbea6459a2 P 76e30a81c8b8451898078e9636a1104b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0188f247aeb8461bade4be31a5251288 in term 2.
I20240503 08:11:55.859750 27108 leader_election.cc:302] T b19203edb44242acb3ad7dfbea6459a2 P 0188f247aeb8461bade4be31a5251288 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0188f247aeb8461bade4be31a5251288, 76e30a81c8b8451898078e9636a1104b; no voters:
I20240503 08:11:55.861020 27488 raft_consensus.cc:2781] T b19203edb44242acb3ad7dfbea6459a2 P 0188f247aeb8461bade4be31a5251288 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:11:55.861470 27488 raft_consensus.cc:686] T b19203edb44242acb3ad7dfbea6459a2 P 0188f247aeb8461bade4be31a5251288 [term 2 LEADER]: Becoming Leader. State: Replica: 0188f247aeb8461bade4be31a5251288, State: Running, Role: LEADER
I20240503 08:11:55.862062 27488 consensus_queue.cc:237] T b19203edb44242acb3ad7dfbea6459a2 P 0188f247aeb8461bade4be31a5251288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "0188f247aeb8461bade4be31a5251288" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 38171 } }
I20240503 08:11:55.865602 23285 catalog_manager.cc:5484] T b19203edb44242acb3ad7dfbea6459a2 P 0188f247aeb8461bade4be31a5251288 reported cstate change: term changed from 1 to 2, leader changed from 76e30a81c8b8451898078e9636a1104b (127.20.230.65) to 0188f247aeb8461bade4be31a5251288 (127.20.230.72). New cstate: current_term: 2 leader_uuid: "0188f247aeb8461bade4be31a5251288" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 } } peers { permanent_uuid: "0188f247aeb8461bade4be31a5251288" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 38171 } } }
I20240503 08:11:55.891613 27155 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:55.894029 23537 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:55.910867 24902 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:55.929666 24641 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:55.949693 25173 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:55.957953 23407 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:55.958721 24772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:55.992000 25033 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:56.040838 23557 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2797c89387af4da5aba9d7d6657eb275"
dest_uuid: "30d88bcd65b347dabcbac54465e52b21"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60308
I20240503 08:11:56.041059 23557 raft_consensus.cc:595] T 2797c89387af4da5aba9d7d6657eb275 P 30d88bcd65b347dabcbac54465e52b21 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:11:56.135838 27155 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:56.144104 23537 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:56.146858 24902 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:56.153182 24772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:56.164476 24641 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:56.180608 23407 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:56.188825 25173 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:56.207540 25033 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:56.286861 27464 raft_consensus.cc:979] T 2797c89387af4da5aba9d7d6657eb275 P 30d88bcd65b347dabcbac54465e52b21: : Instructing follower 0188f247aeb8461bade4be31a5251288 to start an election
I20240503 08:11:56.287030 27464 raft_consensus.cc:1067] T 2797c89387af4da5aba9d7d6657eb275 P 30d88bcd65b347dabcbac54465e52b21 [term 1 LEADER]: Signalling peer 0188f247aeb8461bade4be31a5251288 to start an election
I20240503 08:11:56.287673 27175 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2797c89387af4da5aba9d7d6657eb275"
dest_uuid: "0188f247aeb8461bade4be31a5251288"
from {username='slave'} at 127.20.230.66:47979
I20240503 08:11:56.287889 27175 raft_consensus.cc:482] T 2797c89387af4da5aba9d7d6657eb275 P 0188f247aeb8461bade4be31a5251288 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:11:56.287986 27175 raft_consensus.cc:3037] T 2797c89387af4da5aba9d7d6657eb275 P 0188f247aeb8461bade4be31a5251288 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:56.290719 27175 raft_consensus.cc:504] T 2797c89387af4da5aba9d7d6657eb275 P 0188f247aeb8461bade4be31a5251288 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "0188f247aeb8461bade4be31a5251288" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 38171 } }
I20240503 08:11:56.291332 27175 leader_election.cc:288] T 2797c89387af4da5aba9d7d6657eb275 P 0188f247aeb8461bade4be31a5251288 [CANDIDATE]: Term 2 election: Requested vote from peers 30d88bcd65b347dabcbac54465e52b21 (127.20.230.66:33351)
I20240503 08:11:56.291899 23557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2797c89387af4da5aba9d7d6657eb275" candidate_uuid: "0188f247aeb8461bade4be31a5251288" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "30d88bcd65b347dabcbac54465e52b21"
I20240503 08:11:56.292088 23557 raft_consensus.cc:3032] T 2797c89387af4da5aba9d7d6657eb275 P 30d88bcd65b347dabcbac54465e52b21 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:11:56.292179 23557 raft_consensus.cc:729] T 2797c89387af4da5aba9d7d6657eb275 P 30d88bcd65b347dabcbac54465e52b21 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30d88bcd65b347dabcbac54465e52b21, State: Running, Role: LEADER
I20240503 08:11:56.292428 23557 consensus_queue.cc:260] T 2797c89387af4da5aba9d7d6657eb275 P 30d88bcd65b347dabcbac54465e52b21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "0188f247aeb8461bade4be31a5251288" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 38171 } }
I20240503 08:11:56.292668 23557 raft_consensus.cc:3037] T 2797c89387af4da5aba9d7d6657eb275 P 30d88bcd65b347dabcbac54465e52b21 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:56.294515 23557 raft_consensus.cc:2454] T 2797c89387af4da5aba9d7d6657eb275 P 30d88bcd65b347dabcbac54465e52b21 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0188f247aeb8461bade4be31a5251288 in term 2.
I20240503 08:11:56.294904 27111 leader_election.cc:302] T 2797c89387af4da5aba9d7d6657eb275 P 0188f247aeb8461bade4be31a5251288 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0188f247aeb8461bade4be31a5251288, 30d88bcd65b347dabcbac54465e52b21; no voters:
I20240503 08:11:56.295186 27488 raft_consensus.cc:2781] T 2797c89387af4da5aba9d7d6657eb275 P 0188f247aeb8461bade4be31a5251288 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:11:56.295346 27488 raft_consensus.cc:686] T 2797c89387af4da5aba9d7d6657eb275 P 0188f247aeb8461bade4be31a5251288 [term 2 LEADER]: Becoming Leader. State: Replica: 0188f247aeb8461bade4be31a5251288, State: Running, Role: LEADER
I20240503 08:11:56.295598 27488 consensus_queue.cc:237] T 2797c89387af4da5aba9d7d6657eb275 P 0188f247aeb8461bade4be31a5251288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "0188f247aeb8461bade4be31a5251288" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 38171 } }
I20240503 08:11:56.297950 23286 catalog_manager.cc:5484] T 2797c89387af4da5aba9d7d6657eb275 P 0188f247aeb8461bade4be31a5251288 reported cstate change: term changed from 1 to 2, leader changed from 30d88bcd65b347dabcbac54465e52b21 (127.20.230.66) to 0188f247aeb8461bade4be31a5251288 (127.20.230.72). New cstate: current_term: 2 leader_uuid: "0188f247aeb8461bade4be31a5251288" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 } } peers { permanent_uuid: "0188f247aeb8461bade4be31a5251288" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 38171 } } }
I20240503 08:11:56.315852 23427 raft_consensus.cc:1261] T b19203edb44242acb3ad7dfbea6459a2 P 76e30a81c8b8451898078e9636a1104b [term 2 FOLLOWER]: Refusing update from remote peer 0188f247aeb8461bade4be31a5251288: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 08:11:56.316560 27488 consensus_queue.cc:1035] T b19203edb44242acb3ad7dfbea6459a2 P 0188f247aeb8461bade4be31a5251288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76e30a81c8b8451898078e9636a1104b" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 37573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:11:56.548420 27155 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:56.552254 23537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:56.567709 24902 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:56.569705 24772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:56.598173 23407 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:56.610422 24641 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:56.611737 25173 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:56.615639 25033 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:56.710573 27175 consensus_queue.cc:237] T b19203edb44242acb3ad7dfbea6459a2 P 0188f247aeb8461bade4be31a5251288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0188f247aeb8461bade4be31a5251288" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 38171 } }
I20240503 08:11:56.711525 27490 raft_consensus.cc:2932] T b19203edb44242acb3ad7dfbea6459a2 P 0188f247aeb8461bade4be31a5251288 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 76e30a81c8b8451898078e9636a1104b (127.20.230.65) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0188f247aeb8461bade4be31a5251288" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 38171 } } }
I20240503 08:11:56.716441 23282 catalog_manager.cc:5484] T b19203edb44242acb3ad7dfbea6459a2 P 0188f247aeb8461bade4be31a5251288 reported cstate change: config changed from index 4 to 6, VOTER 76e30a81c8b8451898078e9636a1104b (127.20.230.65) evicted. New cstate: current_term: 2 leader_uuid: "0188f247aeb8461bade4be31a5251288" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0188f247aeb8461bade4be31a5251288" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 38171 } } }
I20240503 08:11:56.720149 23407 tablet_service.cc:1514] Processing DeleteTablet for tablet b19203edb44242acb3ad7dfbea6459a2 with delete_type TABLET_DATA_TOMBSTONED (TS 76e30a81c8b8451898078e9636a1104b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41888
I20240503 08:11:56.721000 27602 tablet_replica.cc:330] T b19203edb44242acb3ad7dfbea6459a2 P 76e30a81c8b8451898078e9636a1104b: stopping tablet replica
I20240503 08:11:56.721195 27602 raft_consensus.cc:2229] T b19203edb44242acb3ad7dfbea6459a2 P 76e30a81c8b8451898078e9636a1104b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:56.721369 27602 raft_consensus.cc:2258] T b19203edb44242acb3ad7dfbea6459a2 P 76e30a81c8b8451898078e9636a1104b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:56.722996 27602 ts_tablet_manager.cc:1875] T b19203edb44242acb3ad7dfbea6459a2 P 76e30a81c8b8451898078e9636a1104b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:11:56.728742 27602 ts_tablet_manager.cc:1888] T b19203edb44242acb3ad7dfbea6459a2 P 76e30a81c8b8451898078e9636a1104b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 08:11:56.728897 27602 log.cc:1196] T b19203edb44242acb3ad7dfbea6459a2 P 76e30a81c8b8451898078e9636a1104b: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/b19203edb44242acb3ad7dfbea6459a2
I20240503 08:11:56.729799 23270 catalog_manager.cc:4830] TS 76e30a81c8b8451898078e9636a1104b (127.20.230.65:37573): tablet b19203edb44242acb3ad7dfbea6459a2 (table rebalance_test_table_2 [id=9e0052fa5a20450fad9ea538bf678fcd]) successfully deleted
I20240503 08:11:56.807021 27155 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:56.814477 24772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:56.820422 23537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:56.829836 25173 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:56.830658 24902 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:56.833415 23407 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:56.843925 24641 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:56.852124 25033 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:56.909026 23557 raft_consensus.cc:1261] T 2797c89387af4da5aba9d7d6657eb275 P 30d88bcd65b347dabcbac54465e52b21 [term 2 FOLLOWER]: Refusing update from remote peer 0188f247aeb8461bade4be31a5251288: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 08:11:56.909899 27490 consensus_queue.cc:1035] T 2797c89387af4da5aba9d7d6657eb275 P 0188f247aeb8461bade4be31a5251288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30d88bcd65b347dabcbac54465e52b21" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 33351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:11:57.224133 27155 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:57.226287 23537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:57.244463 24902 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:57.251448 24772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:57.259379 24641 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:57.279531 23407 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:57.298202 25173 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:57.313724 25033 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:57.402217 27175 consensus_queue.cc:237] T 2797c89387af4da5aba9d7d6657eb275 P 0188f247aeb8461bade4be31a5251288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0188f247aeb8461bade4be31a5251288" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 38171 } }
I20240503 08:11:57.403016 27490 raft_consensus.cc:2932] T 2797c89387af4da5aba9d7d6657eb275 P 0188f247aeb8461bade4be31a5251288 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 30d88bcd65b347dabcbac54465e52b21 (127.20.230.66) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0188f247aeb8461bade4be31a5251288" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 38171 } } }
I20240503 08:11:57.408411 23280 catalog_manager.cc:5484] T 2797c89387af4da5aba9d7d6657eb275 P 0188f247aeb8461bade4be31a5251288 reported cstate change: config changed from index 4 to 6, VOTER 30d88bcd65b347dabcbac54465e52b21 (127.20.230.66) evicted. New cstate: current_term: 2 leader_uuid: "0188f247aeb8461bade4be31a5251288" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0188f247aeb8461bade4be31a5251288" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 38171 } } }
I20240503 08:11:57.413322 23537 tablet_service.cc:1514] Processing DeleteTablet for tablet 2797c89387af4da5aba9d7d6657eb275 with delete_type TABLET_DATA_TOMBSTONED (TS 30d88bcd65b347dabcbac54465e52b21 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35234
I20240503 08:11:57.418143 27704 tablet_replica.cc:330] T 2797c89387af4da5aba9d7d6657eb275 P 30d88bcd65b347dabcbac54465e52b21: stopping tablet replica
I20240503 08:11:57.418334 27704 raft_consensus.cc:2229] T 2797c89387af4da5aba9d7d6657eb275 P 30d88bcd65b347dabcbac54465e52b21 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:57.418589 27704 raft_consensus.cc:2258] T 2797c89387af4da5aba9d7d6657eb275 P 30d88bcd65b347dabcbac54465e52b21 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:57.420253 27704 ts_tablet_manager.cc:1875] T 2797c89387af4da5aba9d7d6657eb275 P 30d88bcd65b347dabcbac54465e52b21: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:11:57.426239 27704 ts_tablet_manager.cc:1888] T 2797c89387af4da5aba9d7d6657eb275 P 30d88bcd65b347dabcbac54465e52b21: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 08:11:57.426342 27704 log.cc:1196] T 2797c89387af4da5aba9d7d6657eb275 P 30d88bcd65b347dabcbac54465e52b21: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/2797c89387af4da5aba9d7d6657eb275
I20240503 08:11:57.427026 23273 catalog_manager.cc:4830] TS 30d88bcd65b347dabcbac54465e52b21 (127.20.230.66:33351): tablet 2797c89387af4da5aba9d7d6657eb275 (table rebalance_test_table_4 [id=a233217db4ea40e88dd7f9aa60cca533]) successfully deleted
I20240503 08:11:57.516054 24902 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:57.525422 24772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:57.528085 27155 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:57.534461 23537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:57.539314 24641 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:57.541327 23407 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:57.557255 25033 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:57.566464 25173 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:57.730792 27155 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:57.746834 24772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:57.762616 24902 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:57.766170 23537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:57.766335 24641 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:57.767644 23407 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:57.795146 25173 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:57.812937 25033 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:58.109119 27155 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:58.114682 24902 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:58.115767 23537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:58.121388 23407 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:58.136785 24641 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:58.138520 25173 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:58.152081 24772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:58.154013 25033 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
0842171f0fd24fa69ee8e4099dc61148 | 127.20.230.126:38671 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.20.230.84:40151 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
0188f247aeb8461bade4be31a5251288 | 127.20.230.72:38171 | HEALTHY | <none> | 2 | 0
30d88bcd65b347dabcbac54465e52b21 | 127.20.230.66:33351 | HEALTHY | <none> | 2 | 0
509c32504f544f8b90ebf66b6dd951f7 | 127.20.230.69:41435 | HEALTHY | <none> | 2 | 0
51d0bc1740db4723a71f1d56b7e6e6b7 | 127.20.230.68:44977 | HEALTHY | <none> | 2 | 0
5ea58b5961b54598878bad3f5660913c | 127.20.230.67:37877 | HEALTHY | <none> | 2 | 0
76e30a81c8b8451898078e9636a1104b | 127.20.230.65:37573 | HEALTHY | <none> | 2 | 0
7823cb7f30b045209bc5ccfb110bf199 | 127.20.230.71:38073 | HEALTHY | <none> | 2 | 0
c82f2e5e9e13405d9328620179077098 | 127.20.230.70:36811 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.20.230.65 | experimental | 127.20.230.65:37573
local_ip_for_outbound_sockets | 127.20.230.66 | experimental | 127.20.230.66:33351
local_ip_for_outbound_sockets | 127.20.230.67 | experimental | 127.20.230.67:37877
local_ip_for_outbound_sockets | 127.20.230.68 | experimental | 127.20.230.68:44977
local_ip_for_outbound_sockets | 127.20.230.69 | experimental | 127.20.230.69:41435
local_ip_for_outbound_sockets | 127.20.230.70 | experimental | 127.20.230.70:36811
local_ip_for_outbound_sockets | 127.20.230.71 | experimental | 127.20.230.71:38073
local_ip_for_outbound_sockets | 127.20.230.72 | experimental | 127.20.230.72:38171
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.20.230.65:37573
server_dump_info_path | /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.20.230.66:33351
server_dump_info_path | /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.20.230.67:37877
server_dump_info_path | /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.20.230.68:44977
server_dump_info_path | /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.20.230.69:41435
server_dump_info_path | /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.20.230.70:36811
server_dump_info_path | /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.20.230.71:38073
server_dump_info_path | /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.20.230.72:38171
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.20.230.84:40151 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 08:11:58.227025 21401 log_verifier.cc:126] Checking tablet 0d69b1adfa064beb82bdcc16d53717d7
I20240503 08:11:58.237008 21401 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 0d69b1adfa064beb82bdcc16d53717d7
I20240503 08:11:58.237097 21401 log_verifier.cc:126] Checking tablet 2449909504504d1e84cf34f1af673af2
I20240503 08:11:58.246332 21401 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 2449909504504d1e84cf34f1af673af2
I20240503 08:11:58.246400 21401 log_verifier.cc:126] Checking tablet 2797c89387af4da5aba9d7d6657eb275
I20240503 08:11:58.257382 21401 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 2797c89387af4da5aba9d7d6657eb275
I20240503 08:11:58.257457 21401 log_verifier.cc:126] Checking tablet 29dea2294fd74738a55ac1bd5b88121b
I20240503 08:11:58.266741 21401 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 29dea2294fd74738a55ac1bd5b88121b
I20240503 08:11:58.266809 21401 log_verifier.cc:126] Checking tablet 43fb21a411474f62afad731b47bad5a9
I20240503 08:11:58.275897 21401 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 43fb21a411474f62afad731b47bad5a9
I20240503 08:11:58.275966 21401 log_verifier.cc:126] Checking tablet 494f9374df1d45f5a0a8f7705e72557a
I20240503 08:11:58.284572 21401 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 494f9374df1d45f5a0a8f7705e72557a
I20240503 08:11:58.284639 21401 log_verifier.cc:126] Checking tablet 6bae8d67e9894646aae3f6230eda07c7
I20240503 08:11:58.293470 21401 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 6bae8d67e9894646aae3f6230eda07c7
I20240503 08:11:58.293532 21401 log_verifier.cc:126] Checking tablet 6bb8e518a6b6496fb52f41f0b6914ff4
I20240503 08:11:58.302094 21401 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 6bb8e518a6b6496fb52f41f0b6914ff4
I20240503 08:11:58.302155 21401 log_verifier.cc:126] Checking tablet 7878470e8ac949bb8971fe8367f7db87
I20240503 08:11:58.310988 21401 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 7878470e8ac949bb8971fe8367f7db87
I20240503 08:11:58.311053 21401 log_verifier.cc:126] Checking tablet 7879251014c84b11a0f49d707d80d85b
I20240503 08:11:58.315714 21401 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 7879251014c84b11a0f49d707d80d85b
I20240503 08:11:58.315775 21401 log_verifier.cc:126] Checking tablet 78b4f0eae3d14b97b8f74cbc24389044
I20240503 08:11:58.328965 21401 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 78b4f0eae3d14b97b8f74cbc24389044
I20240503 08:11:58.329046 21401 log_verifier.cc:126] Checking tablet 85b933bc55b1492fb70055a765a2b09b
I20240503 08:11:58.341379 21401 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 85b933bc55b1492fb70055a765a2b09b
I20240503 08:11:58.341445 21401 log_verifier.cc:126] Checking tablet 8c6217accd494f5182a862c31013b76e
I20240503 08:11:58.350145 21401 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 8c6217accd494f5182a862c31013b76e
I20240503 08:11:58.350215 21401 log_verifier.cc:126] Checking tablet 98dfd1367b224981ab0fd74678bb5ac2
I20240503 08:11:58.359136 21401 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 98dfd1367b224981ab0fd74678bb5ac2
I20240503 08:11:58.359206 21401 log_verifier.cc:126] Checking tablet b19203edb44242acb3ad7dfbea6459a2
I20240503 08:11:58.370379 21401 log_verifier.cc:177] Verified matching terms for 6 ops in tablet b19203edb44242acb3ad7dfbea6459a2
I20240503 08:11:58.370447 21401 log_verifier.cc:126] Checking tablet b2779eae1c08452ca7b5b0874bbac47e
I20240503 08:11:58.379611 21401 log_verifier.cc:177] Verified matching terms for 6 ops in tablet b2779eae1c08452ca7b5b0874bbac47e
I20240503 08:11:58.380448 21401 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8bBFGA/build/asan/bin/kudu pid 23345
I20240503 08:11:58.381618 23467 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:58.576778 21401 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8bBFGA/build/asan/bin/kudu with pid 23345
I20240503 08:11:58.603083 21401 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8bBFGA/build/asan/bin/kudu pid 23475
I20240503 08:11:58.604097 23596 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:58.799337 21401 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8bBFGA/build/asan/bin/kudu with pid 23475
I20240503 08:11:58.824779 21401 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8bBFGA/build/asan/bin/kudu pid 24579
I20240503 08:11:58.830509 24700 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:59.001634 21401 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8bBFGA/build/asan/bin/kudu with pid 24579
W20240503 08:11:59.024503 23271 connection.cc:535] server connection from 127.20.230.67:45661 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:11:59.024703 21401 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8bBFGA/build/asan/bin/kudu pid 24710
I20240503 08:11:59.030083 24832 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:59.207643 21401 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8bBFGA/build/asan/bin/kudu with pid 24710
I20240503 08:11:59.229804 21401 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8bBFGA/build/asan/bin/kudu pid 24840
I20240503 08:11:59.235543 24962 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:59.408448 21401 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8bBFGA/build/asan/bin/kudu with pid 24840
I20240503 08:11:59.432715 21401 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8bBFGA/build/asan/bin/kudu pid 24970
I20240503 08:11:59.438205 25093 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:59.608506 21401 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8bBFGA/build/asan/bin/kudu with pid 24970
I20240503 08:11:59.631124 21401 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8bBFGA/build/asan/bin/kudu pid 25107
I20240503 08:11:59.636941 25232 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:59.806087 21401 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8bBFGA/build/asan/bin/kudu with pid 25107
I20240503 08:11:59.828565 21401 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8bBFGA/build/asan/bin/kudu pid 27093
I20240503 08:11:59.834075 27214 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:59.945669 21401 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8bBFGA/build/asan/bin/kudu with pid 27093
I20240503 08:11:59.962262 21401 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8bBFGA/build/asan/bin/kudu pid 23255
I20240503 08:11:59.963331 23316 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:12:00.186967 21401 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8bBFGA/build/asan/bin/kudu with pid 23255
2024-05-03T08:12:00Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/0 (43645 ms)
[----------] 1 test from RebalanceParamTest (43645 ms total)
[----------] 1 test from DDLDuringRebalancingTest
[ RUN ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/0
I20240503 08:12:00.253382 21401 test_util.cc:268] Using random seed: 1176998317
I20240503 08:12:00.255074 21401 ts_itest-base.cc:112] Starting cluster with:
I20240503 08:12:00.255136 21401 ts_itest-base.cc:113] --------------
I20240503 08:12:00.255193 21401 ts_itest-base.cc:114] 8 tablet servers
I20240503 08:12:00.255240 21401 ts_itest-base.cc:115] 3 replicas per TS
I20240503 08:12:00.255302 21401 ts_itest-base.cc:116] --------------
2024-05-03T08:12:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:12:00Z Disabled control of system clock
I20240503 08:12:00.281076 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.230.126:39173
--webserver_interface=127.20.230.126
--webserver_port=0
--builtin_ntp_servers=127.20.230.84:44873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.230.126:39173
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240503 08:12:00.587966 27831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:00.588251 27831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:00.588337 27831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:00.595392 27831 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:12:00.595496 27831 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:12:00.595546 27831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:00.595600 27831 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:12:00.595643 27831 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:12:00.604075 27831 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:44873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.20.230.126:39173
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.230.126:39173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.230.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:00.605185 27831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:00.606783 27831 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:00.616649 27836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:00.616957 27837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:00.617606 27839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:00.618279 27831 server_base.cc:1034] running on GCE node
I20240503 08:12:00.618937 27831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:00.620153 27831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:00.621353 27831 hybrid_clock.cc:648] HybridClock initialized: now 1714723920621312 us; error 34 us; skew 500 ppm
I20240503 08:12:00.621666 27831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:00.628190 27831 webserver.cc:460] Webserver started at http://127.20.230.126:46553/ using document root <none> and password file <none>
I20240503 08:12:00.628825 27831 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:00.628918 27831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:00.629177 27831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:00.631596 27831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "d2b7d1c3c8a84f53b716ab764eecfa71"
format_stamp: "Formatted at 2024-05-03 08:12:00 on dist-test-slave-6vp8"
I20240503 08:12:00.632328 27831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "d2b7d1c3c8a84f53b716ab764eecfa71"
format_stamp: "Formatted at 2024-05-03 08:12:00 on dist-test-slave-6vp8"
I20240503 08:12:00.637554 27831 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20240503 08:12:00.640550 27845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:00.641732 27831 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:00.641912 27831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d2b7d1c3c8a84f53b716ab764eecfa71"
format_stamp: "Formatted at 2024-05-03 08:12:00 on dist-test-slave-6vp8"
I20240503 08:12:00.642091 27831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:00.674293 27831 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:00.675073 27831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:00.675343 27831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:00.691329 27831 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.126:39173
I20240503 08:12:00.691358 27896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.126:39173 every 8 connection(s)
I20240503 08:12:00.692565 27831 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 08:12:00.695901 27897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:00.698261 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 27831
I20240503 08:12:00.698424 21401 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 08:12:00.706321 27897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Bootstrap starting.
I20240503 08:12:00.709614 27897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:00.710636 27897 log.cc:824] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Log is configured to *not* fsync() on all Append() calls
I20240503 08:12:00.712819 27897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: No bootstrap required, opened a new log
I20240503 08:12:00.718335 27897 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2b7d1c3c8a84f53b716ab764eecfa71" member_type: VOTER last_known_addr { host: "127.20.230.126" port: 39173 } }
I20240503 08:12:00.718631 27897 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:00.718720 27897 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2b7d1c3c8a84f53b716ab764eecfa71, State: Initialized, Role: FOLLOWER
I20240503 08:12:00.719257 27897 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2b7d1c3c8a84f53b716ab764eecfa71" member_type: VOTER last_known_addr { host: "127.20.230.126" port: 39173 } }
I20240503 08:12:00.719455 27897 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:12:00.719574 27897 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:12:00.719748 27897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:00.721623 27897 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2b7d1c3c8a84f53b716ab764eecfa71" member_type: VOTER last_known_addr { host: "127.20.230.126" port: 39173 } }
I20240503 08:12:00.722045 27897 leader_election.cc:302] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2b7d1c3c8a84f53b716ab764eecfa71; no voters:
I20240503 08:12:00.722549 27897 leader_election.cc:288] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:12:00.722658 27902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:00.723171 27902 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71 [term 1 LEADER]: Becoming Leader. State: Replica: d2b7d1c3c8a84f53b716ab764eecfa71, State: Running, Role: LEADER
I20240503 08:12:00.723667 27902 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2b7d1c3c8a84f53b716ab764eecfa71" member_type: VOTER last_known_addr { host: "127.20.230.126" port: 39173 } }
I20240503 08:12:00.724187 27897 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:12:00.726294 27904 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2b7d1c3c8a84f53b716ab764eecfa71. Latest consensus state: current_term: 1 leader_uuid: "d2b7d1c3c8a84f53b716ab764eecfa71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2b7d1c3c8a84f53b716ab764eecfa71" member_type: VOTER last_known_addr { host: "127.20.230.126" port: 39173 } } }
I20240503 08:12:00.726631 27904 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71 [sys.catalog]: This master's current role is: LEADER
I20240503 08:12:00.726526 27903 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2b7d1c3c8a84f53b716ab764eecfa71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2b7d1c3c8a84f53b716ab764eecfa71" member_type: VOTER last_known_addr { host: "127.20.230.126" port: 39173 } } }
I20240503 08:12:00.726778 27903 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71 [sys.catalog]: This master's current role is: LEADER
I20240503 08:12:00.731038 27907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:12:00.736196 27907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:12:00.742980 27907 catalog_manager.cc:1347] Generated new cluster ID: 2134a6ab8758418bab5ba2baae07c1bd
I20240503 08:12:00.743074 27907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:12:00.762248 27907 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:12:00.763284 27907 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:12:00.773655 27907 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Generated new TSK 0
I20240503 08:12:00.774384 27907 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:12:00.789067 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.65:0
--local_ip_for_outbound_sockets=127.20.230.65
--webserver_interface=127.20.230.65
--webserver_port=0
--tserver_master_addrs=127.20.230.126:39173
--builtin_ntp_servers=127.20.230.84:44873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 08:12:01.091043 27921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:01.091297 27921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:01.091390 27921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:01.098453 27921 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:12:01.098557 27921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:01.098670 27921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.65
I20240503 08:12:01.107051 27921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:44873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.230.65
--webserver_port=0
--tserver_master_addrs=127.20.230.126:39173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.65
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:01.108148 27921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:01.109704 27921 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:01.120678 27926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:01.121145 27927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:01.121822 27929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:01.122373 27921 server_base.cc:1034] running on GCE node
I20240503 08:12:01.122875 27921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:01.123667 27921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:01.124840 27921 hybrid_clock.cc:648] HybridClock initialized: now 1714723921124790 us; error 39 us; skew 500 ppm
I20240503 08:12:01.125149 27921 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:01.127125 27921 webserver.cc:460] Webserver started at http://127.20.230.65:42275/ using document root <none> and password file <none>
I20240503 08:12:01.127777 27921 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:01.127883 27921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:01.128157 27921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:01.130507 27921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "833582ff68c94269be364ebd187a75db"
format_stamp: "Formatted at 2024-05-03 08:12:01 on dist-test-slave-6vp8"
I20240503 08:12:01.131222 27921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "833582ff68c94269be364ebd187a75db"
format_stamp: "Formatted at 2024-05-03 08:12:01 on dist-test-slave-6vp8"
I20240503 08:12:01.135718 27921 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 08:12:01.138608 27935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:01.139734 27921 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 08:12:01.139868 27921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "833582ff68c94269be364ebd187a75db"
format_stamp: "Formatted at 2024-05-03 08:12:01 on dist-test-slave-6vp8"
I20240503 08:12:01.140030 27921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:01.160126 27921 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:01.160807 27921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:01.161042 27921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:01.161877 27921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:01.163156 27921 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:01.163251 27921 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:01.163352 27921 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:01.163416 27921 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:01.190984 27921 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.65:39081
I20240503 08:12:01.191031 28047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.65:39081 every 8 connection(s)
I20240503 08:12:01.192166 27921 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 08:12:01.195899 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 27921
I20240503 08:12:01.196103 21401 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 08:12:01.199105 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.66:0
--local_ip_for_outbound_sockets=127.20.230.66
--webserver_interface=127.20.230.66
--webserver_port=0
--tserver_master_addrs=127.20.230.126:39173
--builtin_ntp_servers=127.20.230.84:44873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:12:01.203646 28048 heartbeater.cc:347] Connected to a master server at 127.20.230.126:39173
I20240503 08:12:01.203894 28048 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:01.204533 28048 heartbeater.cc:511] Master 127.20.230.126:39173 requested a full tablet report, sending...
I20240503 08:12:01.206405 27862 ts_manager.cc:194] Registered new tserver with Master: 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:12:01.207809 27862 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.65:38273
W20240503 08:12:01.501715 28051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:01.502075 28051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:01.502213 28051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:01.509683 28051 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:12:01.509877 28051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:01.510039 28051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.66
I20240503 08:12:01.519394 28051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:44873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.230.66
--webserver_port=0
--tserver_master_addrs=127.20.230.126:39173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.66
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:01.520766 28051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:01.522578 28051 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:01.535009 28057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:01.535429 28056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:01.535980 28059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:01.536684 28051 server_base.cc:1034] running on GCE node
I20240503 08:12:01.537317 28051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:01.538362 28051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:01.539567 28051 hybrid_clock.cc:648] HybridClock initialized: now 1714723921539521 us; error 49 us; skew 500 ppm
I20240503 08:12:01.539887 28051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:01.542155 28051 webserver.cc:460] Webserver started at http://127.20.230.66:44789/ using document root <none> and password file <none>
I20240503 08:12:01.542899 28051 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:01.543013 28051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:01.543326 28051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:01.546241 28051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "1b411c6bb25549368041773d7b710d48"
format_stamp: "Formatted at 2024-05-03 08:12:01 on dist-test-slave-6vp8"
I20240503 08:12:01.547062 28051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "1b411c6bb25549368041773d7b710d48"
format_stamp: "Formatted at 2024-05-03 08:12:01 on dist-test-slave-6vp8"
I20240503 08:12:01.552332 28051 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20240503 08:12:01.556010 28065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:01.557523 28051 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:01.557685 28051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "1b411c6bb25549368041773d7b710d48"
format_stamp: "Formatted at 2024-05-03 08:12:01 on dist-test-slave-6vp8"
I20240503 08:12:01.557906 28051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:01.572525 28051 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:01.573393 28051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:01.573671 28051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:01.574697 28051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:01.576275 28051 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:01.576375 28051 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:01.576486 28051 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:01.576553 28051 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:01.605944 28051 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.66:37143
I20240503 08:12:01.606012 28177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.66:37143 every 8 connection(s)
I20240503 08:12:01.607270 28051 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 08:12:01.616602 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 28051
I20240503 08:12:01.616797 21401 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 08:12:01.619551 28178 heartbeater.cc:347] Connected to a master server at 127.20.230.126:39173
I20240503 08:12:01.619848 28178 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:01.620079 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.67:0
--local_ip_for_outbound_sockets=127.20.230.67
--webserver_interface=127.20.230.67
--webserver_port=0
--tserver_master_addrs=127.20.230.126:39173
--builtin_ntp_servers=127.20.230.84:44873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:12:01.620627 28178 heartbeater.cc:511] Master 127.20.230.126:39173 requested a full tablet report, sending...
I20240503 08:12:01.621968 27861 ts_manager.cc:194] Registered new tserver with Master: 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:01.622747 27861 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.66:50517
W20240503 08:12:01.934093 28181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:01.934347 28181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:01.934474 28181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:01.941505 28181 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:12:01.941615 28181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:01.941792 28181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.67
I20240503 08:12:01.950433 28181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:44873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.230.67
--webserver_port=0
--tserver_master_addrs=127.20.230.126:39173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.67
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:01.951602 28181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:01.953244 28181 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:01.964464 28189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:01.965196 28187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:01.965538 28186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:01.966187 28181 server_base.cc:1034] running on GCE node
I20240503 08:12:01.966681 28181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:01.967461 28181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:01.968644 28181 hybrid_clock.cc:648] HybridClock initialized: now 1714723921968606 us; error 30 us; skew 500 ppm
I20240503 08:12:01.968912 28181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:01.970856 28181 webserver.cc:460] Webserver started at http://127.20.230.67:32983/ using document root <none> and password file <none>
I20240503 08:12:01.971566 28181 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:01.971668 28181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:01.971956 28181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:01.974334 28181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "aae860d7ae2b4002bf7794c278f9cc56"
format_stamp: "Formatted at 2024-05-03 08:12:01 on dist-test-slave-6vp8"
I20240503 08:12:01.975026 28181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "aae860d7ae2b4002bf7794c278f9cc56"
format_stamp: "Formatted at 2024-05-03 08:12:01 on dist-test-slave-6vp8"
I20240503 08:12:01.979504 28181 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 08:12:01.982445 28195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:01.983580 28181 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:12:01.983718 28181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "aae860d7ae2b4002bf7794c278f9cc56"
format_stamp: "Formatted at 2024-05-03 08:12:01 on dist-test-slave-6vp8"
I20240503 08:12:01.983886 28181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:01.996284 28181 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:01.997026 28181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:01.997280 28181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:01.998162 28181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:01.999537 28181 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:01.999627 28181 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:01.999728 28181 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:01.999792 28181 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:02.027868 28181 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.67:39919
I20240503 08:12:02.027915 28307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.67:39919 every 8 connection(s)
I20240503 08:12:02.029067 28181 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:12:02.032310 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 28181
I20240503 08:12:02.032526 21401 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 08:12:02.035971 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.68:0
--local_ip_for_outbound_sockets=127.20.230.68
--webserver_interface=127.20.230.68
--webserver_port=0
--tserver_master_addrs=127.20.230.126:39173
--builtin_ntp_servers=127.20.230.84:44873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:12:02.041688 28308 heartbeater.cc:347] Connected to a master server at 127.20.230.126:39173
I20240503 08:12:02.041988 28308 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:02.042677 28308 heartbeater.cc:511] Master 127.20.230.126:39173 requested a full tablet report, sending...
I20240503 08:12:02.043951 27861 ts_manager.cc:194] Registered new tserver with Master: aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:12:02.044760 27861 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.67:56183
I20240503 08:12:02.210675 28048 heartbeater.cc:503] Master 127.20.230.126:39173 was elected leader, sending a full tablet report...
W20240503 08:12:02.342260 28311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:02.342540 28311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:02.342648 28311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:02.349881 28311 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:12:02.350004 28311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:02.350152 28311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.68
I20240503 08:12:02.358768 28311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:44873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.230.68
--webserver_port=0
--tserver_master_addrs=127.20.230.126:39173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.68
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:02.359978 28311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:02.361626 28311 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:02.372563 28316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:02.373066 28319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:02.373651 28317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:02.374420 28311 server_base.cc:1034] running on GCE node
I20240503 08:12:02.374899 28311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:02.375697 28311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:02.376879 28311 hybrid_clock.cc:648] HybridClock initialized: now 1714723922376828 us; error 45 us; skew 500 ppm
I20240503 08:12:02.377135 28311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:02.379084 28311 webserver.cc:460] Webserver started at http://127.20.230.68:40313/ using document root <none> and password file <none>
I20240503 08:12:02.379698 28311 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:02.379784 28311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:02.380029 28311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:02.382382 28311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "d09a11a03989421b9f28019dc1c14cc2"
format_stamp: "Formatted at 2024-05-03 08:12:02 on dist-test-slave-6vp8"
I20240503 08:12:02.383055 28311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "d09a11a03989421b9f28019dc1c14cc2"
format_stamp: "Formatted at 2024-05-03 08:12:02 on dist-test-slave-6vp8"
I20240503 08:12:02.387579 28311 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20240503 08:12:02.390579 28325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:02.391721 28311 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20240503 08:12:02.391858 28311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "d09a11a03989421b9f28019dc1c14cc2"
format_stamp: "Formatted at 2024-05-03 08:12:02 on dist-test-slave-6vp8"
I20240503 08:12:02.392012 28311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:02.430961 28311 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:02.431727 28311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:02.431979 28311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:02.432863 28311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:02.434231 28311 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:02.434317 28311 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:02.434417 28311 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:02.434476 28311 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:02.462419 28311 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.68:46211
I20240503 08:12:02.462519 28437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.68:46211 every 8 connection(s)
I20240503 08:12:02.463634 28311 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:12:02.464964 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 28311
I20240503 08:12:02.465094 21401 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 08:12:02.468061 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.69:0
--local_ip_for_outbound_sockets=127.20.230.69
--webserver_interface=127.20.230.69
--webserver_port=0
--tserver_master_addrs=127.20.230.126:39173
--builtin_ntp_servers=127.20.230.84:44873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:12:02.477414 28438 heartbeater.cc:347] Connected to a master server at 127.20.230.126:39173
I20240503 08:12:02.477656 28438 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:02.478303 28438 heartbeater.cc:511] Master 127.20.230.126:39173 requested a full tablet report, sending...
I20240503 08:12:02.479519 27861 ts_manager.cc:194] Registered new tserver with Master: d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:02.480263 27861 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.68:35061
I20240503 08:12:02.625252 28178 heartbeater.cc:503] Master 127.20.230.126:39173 was elected leader, sending a full tablet report...
W20240503 08:12:02.771430 28441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:02.771685 28441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:02.771781 28441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:02.779026 28441 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:12:02.779125 28441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:02.779253 28441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.69
I20240503 08:12:02.787654 28441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:44873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.230.69
--webserver_port=0
--tserver_master_addrs=127.20.230.126:39173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.69
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:02.788795 28441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:02.790545 28441 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:02.802021 28446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:02.802497 28447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:02.802995 28449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:02.803790 28441 server_base.cc:1034] running on GCE node
I20240503 08:12:02.804322 28441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:02.805140 28441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:02.806370 28441 hybrid_clock.cc:648] HybridClock initialized: now 1714723922806328 us; error 29 us; skew 500 ppm
I20240503 08:12:02.806665 28441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:02.808728 28441 webserver.cc:460] Webserver started at http://127.20.230.69:40539/ using document root <none> and password file <none>
I20240503 08:12:02.809415 28441 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:02.809525 28441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:02.809844 28441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:02.812245 28441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "6d1d7de19459435fbde1999e3dfd226b"
format_stamp: "Formatted at 2024-05-03 08:12:02 on dist-test-slave-6vp8"
I20240503 08:12:02.812986 28441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "6d1d7de19459435fbde1999e3dfd226b"
format_stamp: "Formatted at 2024-05-03 08:12:02 on dist-test-slave-6vp8"
I20240503 08:12:02.817687 28441 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20240503 08:12:02.820647 28455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:02.821847 28441 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:12:02.821982 28441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "6d1d7de19459435fbde1999e3dfd226b"
format_stamp: "Formatted at 2024-05-03 08:12:02 on dist-test-slave-6vp8"
I20240503 08:12:02.822211 28441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:02.855078 28441 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:02.855880 28441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:02.856154 28441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:02.857069 28441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:02.858482 28441 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:02.858577 28441 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:02.858683 28441 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:02.858752 28441 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:02.886230 28441 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.69:34725
I20240503 08:12:02.886266 28567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.69:34725 every 8 connection(s)
I20240503 08:12:02.887482 28441 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:12:02.896667 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 28441
I20240503 08:12:02.896833 21401 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 08:12:02.899178 28568 heartbeater.cc:347] Connected to a master server at 127.20.230.126:39173
I20240503 08:12:02.899441 28568 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:02.899804 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.70:0
--local_ip_for_outbound_sockets=127.20.230.70
--webserver_interface=127.20.230.70
--webserver_port=0
--tserver_master_addrs=127.20.230.126:39173
--builtin_ntp_servers=127.20.230.84:44873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:12:02.900125 28568 heartbeater.cc:511] Master 127.20.230.126:39173 requested a full tablet report, sending...
I20240503 08:12:02.901383 27862 ts_manager.cc:194] Registered new tserver with Master: 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:12:02.902289 27862 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.69:48885
I20240503 08:12:03.047185 28308 heartbeater.cc:503] Master 127.20.230.126:39173 was elected leader, sending a full tablet report...
W20240503 08:12:03.212157 28571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:03.212432 28571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:03.212533 28571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:03.219883 28571 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:12:03.219995 28571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:03.220114 28571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.70
I20240503 08:12:03.228876 28571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:44873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.20.230.70
--webserver_port=0
--tserver_master_addrs=127.20.230.126:39173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.70
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:03.230083 28571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:03.231760 28571 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:03.242779 28577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:03.243358 28576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:03.243938 28579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:03.244743 28571 server_base.cc:1034] running on GCE node
I20240503 08:12:03.245250 28571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:03.246124 28571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:03.247313 28571 hybrid_clock.cc:648] HybridClock initialized: now 1714723923247262 us; error 45 us; skew 500 ppm
I20240503 08:12:03.247591 28571 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:03.249562 28571 webserver.cc:460] Webserver started at http://127.20.230.70:41939/ using document root <none> and password file <none>
I20240503 08:12:03.250265 28571 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:03.250370 28571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:03.250651 28571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:03.253011 28571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "ce1496a47d084365ad786cd09e472a2c"
format_stamp: "Formatted at 2024-05-03 08:12:03 on dist-test-slave-6vp8"
I20240503 08:12:03.253785 28571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "ce1496a47d084365ad786cd09e472a2c"
format_stamp: "Formatted at 2024-05-03 08:12:03 on dist-test-slave-6vp8"
I20240503 08:12:03.258459 28571 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20240503 08:12:03.261435 28585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:03.262672 28571 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 08:12:03.262810 28571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "ce1496a47d084365ad786cd09e472a2c"
format_stamp: "Formatted at 2024-05-03 08:12:03 on dist-test-slave-6vp8"
I20240503 08:12:03.262982 28571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:03.287912 28571 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:03.288674 28571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:03.288951 28571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:03.289816 28571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:03.291175 28571 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:03.291277 28571 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:03.291378 28571 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:03.291445 28571 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:03.318970 28571 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.70:33747
I20240503 08:12:03.319029 28697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.70:33747 every 8 connection(s)
I20240503 08:12:03.320147 28571 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 08:12:03.326782 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 28571
I20240503 08:12:03.326954 21401 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 08:12:03.330199 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.71:0
--local_ip_for_outbound_sockets=127.20.230.71
--webserver_interface=127.20.230.71
--webserver_port=0
--tserver_master_addrs=127.20.230.126:39173
--builtin_ntp_servers=127.20.230.84:44873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:12:03.332167 28698 heartbeater.cc:347] Connected to a master server at 127.20.230.126:39173
I20240503 08:12:03.332461 28698 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:03.333266 28698 heartbeater.cc:511] Master 127.20.230.126:39173 requested a full tablet report, sending...
I20240503 08:12:03.334599 27862 ts_manager.cc:194] Registered new tserver with Master: ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:12:03.335340 27862 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.70:42953
I20240503 08:12:03.482973 28438 heartbeater.cc:503] Master 127.20.230.126:39173 was elected leader, sending a full tablet report...
W20240503 08:12:03.639559 28701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:03.639803 28701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:03.639886 28701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:03.647060 28701 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:12:03.647171 28701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:03.647297 28701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.71
I20240503 08:12:03.655903 28701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:44873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.20.230.71
--webserver_port=0
--tserver_master_addrs=127.20.230.126:39173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.71
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:03.657124 28701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:03.658762 28701 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:03.669857 28706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:03.670408 28709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:03.671051 28707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:03.671808 28701 server_base.cc:1034] running on GCE node
I20240503 08:12:03.672529 28701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:03.673434 28701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:03.674628 28701 hybrid_clock.cc:648] HybridClock initialized: now 1714723923674595 us; error 27 us; skew 500 ppm
I20240503 08:12:03.674928 28701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:03.677068 28701 webserver.cc:460] Webserver started at http://127.20.230.71:35683/ using document root <none> and password file <none>
I20240503 08:12:03.677726 28701 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:03.677851 28701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:03.678131 28701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:03.680480 28701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
format_stamp: "Formatted at 2024-05-03 08:12:03 on dist-test-slave-6vp8"
I20240503 08:12:03.681175 28701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
format_stamp: "Formatted at 2024-05-03 08:12:03 on dist-test-slave-6vp8"
I20240503 08:12:03.685983 28701 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.003s
I20240503 08:12:03.689034 28715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:03.690244 28701 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:12:03.690380 28701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
format_stamp: "Formatted at 2024-05-03 08:12:03 on dist-test-slave-6vp8"
I20240503 08:12:03.690552 28701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:03.706250 28701 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:03.706996 28701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:03.707267 28701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:03.708124 28701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:03.709443 28701 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:03.709535 28701 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:03.709636 28701 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:03.709702 28701 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:03.736991 28701 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.71:41915
I20240503 08:12:03.737035 28827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.71:41915 every 8 connection(s)
I20240503 08:12:03.738260 28701 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 08:12:03.748237 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 28701
I20240503 08:12:03.748430 21401 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 08:12:03.751048 28828 heartbeater.cc:347] Connected to a master server at 127.20.230.126:39173
I20240503 08:12:03.751318 28828 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:03.751681 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.72:0
--local_ip_for_outbound_sockets=127.20.230.72
--webserver_interface=127.20.230.72
--webserver_port=0
--tserver_master_addrs=127.20.230.126:39173
--builtin_ntp_servers=127.20.230.84:44873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:12:03.752054 28828 heartbeater.cc:511] Master 127.20.230.126:39173 requested a full tablet report, sending...
I20240503 08:12:03.753302 27862 ts_manager.cc:194] Registered new tserver with Master: ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:03.754249 27862 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.71:47933
I20240503 08:12:03.904561 28568 heartbeater.cc:503] Master 127.20.230.126:39173 was elected leader, sending a full tablet report...
W20240503 08:12:04.056926 28831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:04.057186 28831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:04.057269 28831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:04.064514 28831 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:12:04.064617 28831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:04.064733 28831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.72
I20240503 08:12:04.073482 28831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:44873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.20.230.72
--webserver_port=0
--tserver_master_addrs=127.20.230.126:39173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.72
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:04.074673 28831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:04.076324 28831 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:04.087503 28836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:04.087904 28837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:04.088651 28839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:04.089251 28831 server_base.cc:1034] running on GCE node
I20240503 08:12:04.089757 28831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:04.090613 28831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:04.091809 28831 hybrid_clock.cc:648] HybridClock initialized: now 1714723924091749 us; error 44 us; skew 500 ppm
I20240503 08:12:04.092103 28831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:04.094085 28831 webserver.cc:460] Webserver started at http://127.20.230.72:44491/ using document root <none> and password file <none>
I20240503 08:12:04.094743 28831 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:04.094833 28831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:04.095132 28831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:04.097481 28831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "b00f244c69df401997ea99d492218a98"
format_stamp: "Formatted at 2024-05-03 08:12:04 on dist-test-slave-6vp8"
I20240503 08:12:04.098227 28831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "b00f244c69df401997ea99d492218a98"
format_stamp: "Formatted at 2024-05-03 08:12:04 on dist-test-slave-6vp8"
I20240503 08:12:04.102669 28831 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240503 08:12:04.105571 28845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:04.106755 28831 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 08:12:04.106885 28831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "b00f244c69df401997ea99d492218a98"
format_stamp: "Formatted at 2024-05-03 08:12:04 on dist-test-slave-6vp8"
I20240503 08:12:04.107050 28831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:04.130313 28831 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:04.131110 28831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:04.131359 28831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:04.132284 28831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:04.133667 28831 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:04.133780 28831 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:04.133880 28831 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:04.133944 28831 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:04.161880 28831 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.72:40893
I20240503 08:12:04.161921 28957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.72:40893 every 8 connection(s)
I20240503 08:12:04.163134 28831 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 08:12:04.169273 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 28831
I20240503 08:12:04.169446 21401 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 08:12:04.175257 28958 heartbeater.cc:347] Connected to a master server at 127.20.230.126:39173
I20240503 08:12:04.175500 28958 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:04.176146 28958 heartbeater.cc:511] Master 127.20.230.126:39173 requested a full tablet report, sending...
I20240503 08:12:04.177284 27862 ts_manager.cc:194] Registered new tserver with Master: b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:04.178020 27862 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.72:33631
I20240503 08:12:04.184893 21401 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 08:12:04.200048 21401 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 08:12:04.200161 21401 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8bBFGA/build/asan/bin/kudu pid 28441
I20240503 08:12:04.207185 28563 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:12:04.285856 21401 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8bBFGA/build/asan/bin/kudu with pid 28441
I20240503 08:12:04.298344 21401 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8bBFGA/build/asan/bin/kudu pid 28571
I20240503 08:12:04.305801 28693 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:12:04.382567 28698 heartbeater.cc:503] Master 127.20.230.126:39173 was elected leader, sending a full tablet report...
I20240503 08:12:04.384435 21401 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8bBFGA/build/asan/bin/kudu with pid 28571
I20240503 08:12:04.396507 21401 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8bBFGA/build/asan/bin/kudu pid 28701
I20240503 08:12:04.403630 28823 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:12:04.485500 21401 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8bBFGA/build/asan/bin/kudu with pid 28701
I20240503 08:12:04.497884 21401 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8bBFGA/build/asan/bin/kudu pid 28831
I20240503 08:12:04.505712 28953 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:12:04.584884 21401 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8bBFGA/build/asan/bin/kudu with pid 28831
I20240503 08:12:08.349274 27862 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:08.388154 28372 tablet_service.cc:1467] Processing CreateTablet for tablet 02da1912b357405fa7e48e8d6ae8329d (DEFAULT_TABLE table=rebalance_test_table_0 [id=06bbbe2220124afd9e58aad99f4c6e3d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:08.387280 27983 tablet_service.cc:1467] Processing CreateTablet for tablet d845da554a404c20be95a0bc11f49826 (DEFAULT_TABLE table=rebalance_test_table_0 [id=06bbbe2220124afd9e58aad99f4c6e3d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:08.389568 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02da1912b357405fa7e48e8d6ae8329d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:08.390043 28373 tablet_service.cc:1467] Processing CreateTablet for tablet d845da554a404c20be95a0bc11f49826 (DEFAULT_TABLE table=rebalance_test_table_0 [id=06bbbe2220124afd9e58aad99f4c6e3d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:08.390497 28373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d845da554a404c20be95a0bc11f49826. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:08.390038 28242 tablet_service.cc:1467] Processing CreateTablet for tablet 02da1912b357405fa7e48e8d6ae8329d (DEFAULT_TABLE table=rebalance_test_table_0 [id=06bbbe2220124afd9e58aad99f4c6e3d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:08.391013 28243 tablet_service.cc:1467] Processing CreateTablet for tablet bd1407f5301a41438e79d872961d74d6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=06bbbe2220124afd9e58aad99f4c6e3d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:08.391294 28242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02da1912b357405fa7e48e8d6ae8329d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:08.391698 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd1407f5301a41438e79d872961d74d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:08.392689 27983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d845da554a404c20be95a0bc11f49826. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:08.387331 27982 tablet_service.cc:1467] Processing CreateTablet for tablet bd1407f5301a41438e79d872961d74d6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=06bbbe2220124afd9e58aad99f4c6e3d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:08.393131 27982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd1407f5301a41438e79d872961d74d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:08.387517 27981 tablet_service.cc:1467] Processing CreateTablet for tablet 02da1912b357405fa7e48e8d6ae8329d (DEFAULT_TABLE table=rebalance_test_table_0 [id=06bbbe2220124afd9e58aad99f4c6e3d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:08.394357 27981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02da1912b357405fa7e48e8d6ae8329d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:08.397394 28113 tablet_service.cc:1467] Processing CreateTablet for tablet d845da554a404c20be95a0bc11f49826 (DEFAULT_TABLE table=rebalance_test_table_0 [id=06bbbe2220124afd9e58aad99f4c6e3d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:08.398654 28113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d845da554a404c20be95a0bc11f49826. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:08.402215 28112 tablet_service.cc:1467] Processing CreateTablet for tablet bd1407f5301a41438e79d872961d74d6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=06bbbe2220124afd9e58aad99f4c6e3d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:08.402622 28112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd1407f5301a41438e79d872961d74d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:08.410276 28987 tablet_bootstrap.cc:492] T 02da1912b357405fa7e48e8d6ae8329d P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:08.413604 28987 tablet_bootstrap.cc:654] T 02da1912b357405fa7e48e8d6ae8329d P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:08.415973 28987 log.cc:824] T 02da1912b357405fa7e48e8d6ae8329d P aae860d7ae2b4002bf7794c278f9cc56: Log is configured to *not* fsync() on all Append() calls
I20240503 08:12:08.426154 28990 tablet_bootstrap.cc:492] T 02da1912b357405fa7e48e8d6ae8329d P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:08.431245 28990 tablet_bootstrap.cc:654] T 02da1912b357405fa7e48e8d6ae8329d P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:08.432549 28991 tablet_bootstrap.cc:492] T bd1407f5301a41438e79d872961d74d6 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:08.432725 28988 tablet_bootstrap.cc:492] T 02da1912b357405fa7e48e8d6ae8329d P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:08.435875 28991 tablet_bootstrap.cc:654] T bd1407f5301a41438e79d872961d74d6 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:08.436149 28988 tablet_bootstrap.cc:654] T 02da1912b357405fa7e48e8d6ae8329d P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:08.436268 28990 log.cc:824] T 02da1912b357405fa7e48e8d6ae8329d P 833582ff68c94269be364ebd187a75db: Log is configured to *not* fsync() on all Append() calls
I20240503 08:12:08.437072 28991 log.cc:824] T bd1407f5301a41438e79d872961d74d6 P 1b411c6bb25549368041773d7b710d48: Log is configured to *not* fsync() on all Append() calls
I20240503 08:12:08.437383 28988 log.cc:824] T 02da1912b357405fa7e48e8d6ae8329d P d09a11a03989421b9f28019dc1c14cc2: Log is configured to *not* fsync() on all Append() calls
I20240503 08:12:08.439412 28990 tablet_bootstrap.cc:492] T 02da1912b357405fa7e48e8d6ae8329d P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:08.439487 28987 tablet_bootstrap.cc:492] T 02da1912b357405fa7e48e8d6ae8329d P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:08.439785 28990 ts_tablet_manager.cc:1367] T 02da1912b357405fa7e48e8d6ae8329d P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.005s
I20240503 08:12:08.439787 28987 ts_tablet_manager.cc:1367] T 02da1912b357405fa7e48e8d6ae8329d P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.030s user 0.013s sys 0.013s
I20240503 08:12:08.445959 28988 tablet_bootstrap.cc:492] T 02da1912b357405fa7e48e8d6ae8329d P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:08.446277 28988 ts_tablet_manager.cc:1367] T 02da1912b357405fa7e48e8d6ae8329d P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20240503 08:12:08.448072 28990 raft_consensus.cc:348] T 02da1912b357405fa7e48e8d6ae8329d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:08.448607 28990 raft_consensus.cc:374] T 02da1912b357405fa7e48e8d6ae8329d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:08.448746 28990 raft_consensus.cc:729] T 02da1912b357405fa7e48e8d6ae8329d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:08.449532 28990 consensus_queue.cc:260] T 02da1912b357405fa7e48e8d6ae8329d P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:08.450537 28987 raft_consensus.cc:348] T 02da1912b357405fa7e48e8d6ae8329d P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:08.450995 28987 raft_consensus.cc:374] T 02da1912b357405fa7e48e8d6ae8329d P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:08.451122 28987 raft_consensus.cc:729] T 02da1912b357405fa7e48e8d6ae8329d P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:08.452730 28987 consensus_queue.cc:260] T 02da1912b357405fa7e48e8d6ae8329d P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:08.454005 28991 tablet_bootstrap.cc:492] T bd1407f5301a41438e79d872961d74d6 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:08.454313 28991 ts_tablet_manager.cc:1367] T bd1407f5301a41438e79d872961d74d6 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.011s
I20240503 08:12:08.456396 28988 raft_consensus.cc:348] T 02da1912b357405fa7e48e8d6ae8329d P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:08.456869 28988 raft_consensus.cc:374] T 02da1912b357405fa7e48e8d6ae8329d P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:08.457017 28988 raft_consensus.cc:729] T 02da1912b357405fa7e48e8d6ae8329d P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:08.457590 28990 ts_tablet_manager.cc:1398] T 02da1912b357405fa7e48e8d6ae8329d P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.018s user 0.009s sys 0.003s
I20240503 08:12:08.457783 28988 consensus_queue.cc:260] T 02da1912b357405fa7e48e8d6ae8329d P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:08.460140 28990 tablet_bootstrap.cc:492] T bd1407f5301a41438e79d872961d74d6 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:08.460660 28987 ts_tablet_manager.cc:1398] T 02da1912b357405fa7e48e8d6ae8329d P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.021s user 0.011s sys 0.008s
I20240503 08:12:08.460836 28995 raft_consensus.cc:482] T 02da1912b357405fa7e48e8d6ae8329d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:08.461117 28987 tablet_bootstrap.cc:492] T bd1407f5301a41438e79d872961d74d6 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:08.461022 28995 raft_consensus.cc:504] T 02da1912b357405fa7e48e8d6ae8329d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:08.462930 28987 tablet_bootstrap.cc:654] T bd1407f5301a41438e79d872961d74d6 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:08.467310 28991 raft_consensus.cc:348] T bd1407f5301a41438e79d872961d74d6 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:08.467788 28991 raft_consensus.cc:374] T bd1407f5301a41438e79d872961d74d6 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:08.467914 28991 raft_consensus.cc:729] T bd1407f5301a41438e79d872961d74d6 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:08.468631 28991 consensus_queue.cc:260] T bd1407f5301a41438e79d872961d74d6 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:08.472801 28995 leader_election.cc:288] T 02da1912b357405fa7e48e8d6ae8329d P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:08.493620 28990 tablet_bootstrap.cc:654] T bd1407f5301a41438e79d872961d74d6 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:08.513219 28990 tablet_bootstrap.cc:492] T bd1407f5301a41438e79d872961d74d6 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:08.513458 28990 ts_tablet_manager.cc:1367] T bd1407f5301a41438e79d872961d74d6 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.053s user 0.000s sys 0.004s
I20240503 08:12:08.514537 28990 raft_consensus.cc:348] T bd1407f5301a41438e79d872961d74d6 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:08.514793 28990 raft_consensus.cc:374] T bd1407f5301a41438e79d872961d74d6 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:08.514909 28990 raft_consensus.cc:729] T bd1407f5301a41438e79d872961d74d6 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:08.515190 28990 consensus_queue.cc:260] T bd1407f5301a41438e79d872961d74d6 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:08.515739 28990 ts_tablet_manager.cc:1398] T bd1407f5301a41438e79d872961d74d6 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:08.516028 28990 tablet_bootstrap.cc:492] T d845da554a404c20be95a0bc11f49826 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:08.517607 28990 tablet_bootstrap.cc:654] T d845da554a404c20be95a0bc11f49826 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:08.518141 28988 ts_tablet_manager.cc:1398] T 02da1912b357405fa7e48e8d6ae8329d P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.072s user 0.039s sys 0.020s
I20240503 08:12:08.519443 28393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02da1912b357405fa7e48e8d6ae8329d" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:08.519973 28393 raft_consensus.cc:2454] T 02da1912b357405fa7e48e8d6ae8329d P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:08.520696 28263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02da1912b357405fa7e48e8d6ae8329d" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:12:08.520913 27939 leader_election.cc:302] T 02da1912b357405fa7e48e8d6ae8329d P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:08.521226 28263 raft_consensus.cc:2454] T 02da1912b357405fa7e48e8d6ae8329d P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:08.521343 28995 raft_consensus.cc:2781] T 02da1912b357405fa7e48e8d6ae8329d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:08.521466 28995 raft_consensus.cc:482] T 02da1912b357405fa7e48e8d6ae8329d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:08.521541 28995 raft_consensus.cc:3037] T 02da1912b357405fa7e48e8d6ae8329d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:08.522130 28988 tablet_bootstrap.cc:492] T d845da554a404c20be95a0bc11f49826 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:08.523295 28988 tablet_bootstrap.cc:654] T d845da554a404c20be95a0bc11f49826 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:08.524909 28995 raft_consensus.cc:504] T 02da1912b357405fa7e48e8d6ae8329d P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:08.525651 28995 leader_election.cc:288] T 02da1912b357405fa7e48e8d6ae8329d P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:08.525960 28263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02da1912b357405fa7e48e8d6ae8329d" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:12:08.526147 28263 raft_consensus.cc:3037] T 02da1912b357405fa7e48e8d6ae8329d P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:08.526407 28393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02da1912b357405fa7e48e8d6ae8329d" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:12:08.526587 28393 raft_consensus.cc:3037] T 02da1912b357405fa7e48e8d6ae8329d P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:08.527361 28991 ts_tablet_manager.cc:1398] T bd1407f5301a41438e79d872961d74d6 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.073s user 0.021s sys 0.035s
I20240503 08:12:08.527496 28987 tablet_bootstrap.cc:492] T bd1407f5301a41438e79d872961d74d6 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:08.527635 28987 ts_tablet_manager.cc:1367] T bd1407f5301a41438e79d872961d74d6 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.067s user 0.025s sys 0.034s
I20240503 08:12:08.527772 28991 tablet_bootstrap.cc:492] T d845da554a404c20be95a0bc11f49826 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:08.528589 28987 raft_consensus.cc:348] T bd1407f5301a41438e79d872961d74d6 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:08.528777 28987 raft_consensus.cc:374] T bd1407f5301a41438e79d872961d74d6 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:08.528857 28987 raft_consensus.cc:729] T bd1407f5301a41438e79d872961d74d6 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:08.529126 28987 consensus_queue.cc:260] T bd1407f5301a41438e79d872961d74d6 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:08.529286 28991 tablet_bootstrap.cc:654] T d845da554a404c20be95a0bc11f49826 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:08.529249 28263 raft_consensus.cc:2454] T 02da1912b357405fa7e48e8d6ae8329d P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:08.529655 28987 ts_tablet_manager.cc:1398] T bd1407f5301a41438e79d872961d74d6 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:08.529819 27937 leader_election.cc:302] T 02da1912b357405fa7e48e8d6ae8329d P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:08.530097 28393 raft_consensus.cc:2454] T 02da1912b357405fa7e48e8d6ae8329d P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:08.530135 28995 raft_consensus.cc:2781] T 02da1912b357405fa7e48e8d6ae8329d P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:08.534296 28995 raft_consensus.cc:686] T 02da1912b357405fa7e48e8d6ae8329d P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Becoming Leader. State: Replica: 833582ff68c94269be364ebd187a75db, State: Running, Role: LEADER
W20240503 08:12:08.534637 28309 tablet.cc:2369] T 02da1912b357405fa7e48e8d6ae8329d P aae860d7ae2b4002bf7794c278f9cc56: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:12:08.534801 28995 consensus_queue.cc:237] T 02da1912b357405fa7e48e8d6ae8329d P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:08.536989 28990 tablet_bootstrap.cc:492] T d845da554a404c20be95a0bc11f49826 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:08.537086 28991 tablet_bootstrap.cc:492] T d845da554a404c20be95a0bc11f49826 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:08.537221 28991 ts_tablet_manager.cc:1367] T d845da554a404c20be95a0bc11f49826 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.002s
I20240503 08:12:08.537256 28990 ts_tablet_manager.cc:1367] T d845da554a404c20be95a0bc11f49826 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.007s
I20240503 08:12:08.537402 28988 tablet_bootstrap.cc:492] T d845da554a404c20be95a0bc11f49826 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:08.537643 28988 ts_tablet_manager.cc:1367] T d845da554a404c20be95a0bc11f49826 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.005s
I20240503 08:12:08.538007 28990 raft_consensus.cc:348] T d845da554a404c20be95a0bc11f49826 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:08.538167 28991 raft_consensus.cc:348] T d845da554a404c20be95a0bc11f49826 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:08.538283 28990 raft_consensus.cc:374] T d845da554a404c20be95a0bc11f49826 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:08.538357 28990 raft_consensus.cc:729] T d845da554a404c20be95a0bc11f49826 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:08.538394 28991 raft_consensus.cc:374] T d845da554a404c20be95a0bc11f49826 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:08.538518 28991 raft_consensus.cc:729] T d845da554a404c20be95a0bc11f49826 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:08.538633 28990 consensus_queue.cc:260] T d845da554a404c20be95a0bc11f49826 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:08.538753 28988 raft_consensus.cc:348] T d845da554a404c20be95a0bc11f49826 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:08.538822 28991 consensus_queue.cc:260] T d845da554a404c20be95a0bc11f49826 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:08.539161 28988 raft_consensus.cc:374] T d845da554a404c20be95a0bc11f49826 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:08.539228 28990 ts_tablet_manager.cc:1398] T d845da554a404c20be95a0bc11f49826 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:08.539266 28991 ts_tablet_manager.cc:1398] T d845da554a404c20be95a0bc11f49826 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:08.539285 28988 raft_consensus.cc:729] T d845da554a404c20be95a0bc11f49826 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:08.539614 28988 consensus_queue.cc:260] T d845da554a404c20be95a0bc11f49826 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:08.540323 28988 ts_tablet_manager.cc:1398] T d845da554a404c20be95a0bc11f49826 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:08.542598 27862 catalog_manager.cc:5484] T 02da1912b357405fa7e48e8d6ae8329d P 833582ff68c94269be364ebd187a75db reported cstate change: term changed from 0 to 1, leader changed from <none> to 833582ff68c94269be364ebd187a75db (127.20.230.65). New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
W20240503 08:12:08.614241 28179 tablet.cc:2369] T bd1407f5301a41438e79d872961d74d6 P 1b411c6bb25549368041773d7b710d48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:12:08.625757 28996 raft_consensus.cc:482] T bd1407f5301a41438e79d872961d74d6 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:08.625986 28996 raft_consensus.cc:504] T bd1407f5301a41438e79d872961d74d6 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:08.627645 28996 leader_election.cc:288] T bd1407f5301a41438e79d872961d74d6 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:08.634542 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1407f5301a41438e79d872961d74d6" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:12:08.634765 28133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1407f5301a41438e79d872961d74d6" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:12:08.635025 28003 raft_consensus.cc:2454] T bd1407f5301a41438e79d872961d74d6 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:12:08.635278 28133 raft_consensus.cc:2454] T bd1407f5301a41438e79d872961d74d6 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:12:08.635695 28199 leader_election.cc:302] T bd1407f5301a41438e79d872961d74d6 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:08.636080 28996 raft_consensus.cc:2781] T bd1407f5301a41438e79d872961d74d6 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:08.636245 28996 raft_consensus.cc:482] T bd1407f5301a41438e79d872961d74d6 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:08.636340 28996 raft_consensus.cc:3037] T bd1407f5301a41438e79d872961d74d6 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:08.638527 28996 raft_consensus.cc:504] T bd1407f5301a41438e79d872961d74d6 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:08.639168 28996 leader_election.cc:288] T bd1407f5301a41438e79d872961d74d6 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Requested vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:08.639684 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1407f5301a41438e79d872961d74d6" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:12:08.639739 28133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1407f5301a41438e79d872961d74d6" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:12:08.639910 28003 raft_consensus.cc:3037] T bd1407f5301a41438e79d872961d74d6 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:08.639915 28133 raft_consensus.cc:3037] T bd1407f5301a41438e79d872961d74d6 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:08.642902 28133 raft_consensus.cc:2454] T bd1407f5301a41438e79d872961d74d6 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:08.642902 28003 raft_consensus.cc:2454] T bd1407f5301a41438e79d872961d74d6 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:08.643327 28198 leader_election.cc:302] T bd1407f5301a41438e79d872961d74d6 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:08.643646 28996 raft_consensus.cc:2781] T bd1407f5301a41438e79d872961d74d6 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:08.644058 28996 raft_consensus.cc:686] T bd1407f5301a41438e79d872961d74d6 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Becoming Leader. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Running, Role: LEADER
I20240503 08:12:08.644640 28996 consensus_queue.cc:237] T bd1407f5301a41438e79d872961d74d6 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:08.648887 27862 catalog_manager.cc:5484] T bd1407f5301a41438e79d872961d74d6 P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: term changed from 0 to 1, leader changed from <none> to aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67). New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
W20240503 08:12:08.698305 28049 tablet.cc:2369] T 02da1912b357405fa7e48e8d6ae8329d P 833582ff68c94269be364ebd187a75db: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:12:08.718777 28439 tablet.cc:2369] T 02da1912b357405fa7e48e8d6ae8329d P d09a11a03989421b9f28019dc1c14cc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:12:08.954962 28995 consensus_queue.cc:1035] T 02da1912b357405fa7e48e8d6ae8329d P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:08.960894 28995 consensus_queue.cc:1035] T 02da1912b357405fa7e48e8d6ae8329d P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:08.990044 28999 raft_consensus.cc:482] T d845da554a404c20be95a0bc11f49826 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:08.990262 28999 raft_consensus.cc:504] T d845da554a404c20be95a0bc11f49826 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:08.991777 28999 leader_election.cc:288] T d845da554a404c20be95a0bc11f49826 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:08.997942 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d845da554a404c20be95a0bc11f49826" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:12:08.998180 28003 raft_consensus.cc:2454] T d845da554a404c20be95a0bc11f49826 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:08.998620 28069 leader_election.cc:302] T d845da554a404c20be95a0bc11f49826 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 833582ff68c94269be364ebd187a75db; no voters:
I20240503 08:12:08.998958 28999 raft_consensus.cc:2781] T d845da554a404c20be95a0bc11f49826 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:08.999058 28999 raft_consensus.cc:482] T d845da554a404c20be95a0bc11f49826 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:08.999003 28393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d845da554a404c20be95a0bc11f49826" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:08.999140 28999 raft_consensus.cc:3037] T d845da554a404c20be95a0bc11f49826 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:08.999342 28393 raft_consensus.cc:2454] T d845da554a404c20be95a0bc11f49826 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:09.001202 28999 raft_consensus.cc:504] T d845da554a404c20be95a0bc11f49826 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:09.001750 28999 leader_election.cc:288] T d845da554a404c20be95a0bc11f49826 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Requested vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:09.002127 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d845da554a404c20be95a0bc11f49826" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:12:09.002203 28393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d845da554a404c20be95a0bc11f49826" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:12:09.002400 28003 raft_consensus.cc:3037] T d845da554a404c20be95a0bc11f49826 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:09.002411 28393 raft_consensus.cc:3037] T d845da554a404c20be95a0bc11f49826 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:09.005048 28003 raft_consensus.cc:2454] T d845da554a404c20be95a0bc11f49826 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:12:09.005048 28393 raft_consensus.cc:2454] T d845da554a404c20be95a0bc11f49826 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:12:09.005409 28069 leader_election.cc:302] T d845da554a404c20be95a0bc11f49826 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:09.005637 28999 raft_consensus.cc:2781] T d845da554a404c20be95a0bc11f49826 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:09.006033 28999 raft_consensus.cc:686] T d845da554a404c20be95a0bc11f49826 P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Becoming Leader. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Running, Role: LEADER
I20240503 08:12:09.006470 28999 consensus_queue.cc:237] T d845da554a404c20be95a0bc11f49826 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:09.010345 27862 catalog_manager.cc:5484] T d845da554a404c20be95a0bc11f49826 P 1b411c6bb25549368041773d7b710d48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b411c6bb25549368041773d7b710d48 (127.20.230.66). New cstate: current_term: 1 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:12:09.224400 28996 consensus_queue.cc:1035] T bd1407f5301a41438e79d872961d74d6 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:09.230101 29023 consensus_queue.cc:1035] T bd1407f5301a41438e79d872961d74d6 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:09.418061 29021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:09.418354 29021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:09.425428 29021 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 08:12:09.537029 28999 consensus_queue.cc:1035] T d845da554a404c20be95a0bc11f49826 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:09.585376 29060 consensus_queue.cc:1035] T d845da554a404c20be95a0bc11f49826 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 233.103 ms
time per row: 0.116552 ms
I20240503 08:12:09.801676 27860 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:09.809985 28243 tablet_service.cc:1467] Processing CreateTablet for tablet d94f5824a71f454f8a1bfa13dd032684 (DEFAULT_TABLE table=rebalance_test_table_1 [id=05d29869a30847b6a0c137521965898a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:09.810509 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d94f5824a71f454f8a1bfa13dd032684. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:09.810772 28373 tablet_service.cc:1467] Processing CreateTablet for tablet 775b8190036b4b828431fca19ce23ea9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=05d29869a30847b6a0c137521965898a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:09.811214 28373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 775b8190036b4b828431fca19ce23ea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:09.811179 28372 tablet_service.cc:1467] Processing CreateTablet for tablet d94f5824a71f454f8a1bfa13dd032684 (DEFAULT_TABLE table=rebalance_test_table_1 [id=05d29869a30847b6a0c137521965898a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:09.811427 27981 tablet_service.cc:1467] Processing CreateTablet for tablet d9098f3d36b24d578a3f8d7d3ccad052 (DEFAULT_TABLE table=rebalance_test_table_1 [id=05d29869a30847b6a0c137521965898a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:09.811525 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d94f5824a71f454f8a1bfa13dd032684. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:09.811806 27981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9098f3d36b24d578a3f8d7d3ccad052. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:09.818631 28112 tablet_service.cc:1467] Processing CreateTablet for tablet 775b8190036b4b828431fca19ce23ea9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=05d29869a30847b6a0c137521965898a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:09.818729 28111 tablet_service.cc:1467] Processing CreateTablet for tablet d9098f3d36b24d578a3f8d7d3ccad052 (DEFAULT_TABLE table=rebalance_test_table_1 [id=05d29869a30847b6a0c137521965898a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:09.819114 28112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 775b8190036b4b828431fca19ce23ea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:09.819392 28111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9098f3d36b24d578a3f8d7d3ccad052. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:09.819892 28113 tablet_service.cc:1467] Processing CreateTablet for tablet d94f5824a71f454f8a1bfa13dd032684 (DEFAULT_TABLE table=rebalance_test_table_1 [id=05d29869a30847b6a0c137521965898a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:09.820240 28113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d94f5824a71f454f8a1bfa13dd032684. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:09.820263 28241 tablet_service.cc:1467] Processing CreateTablet for tablet d9098f3d36b24d578a3f8d7d3ccad052 (DEFAULT_TABLE table=rebalance_test_table_1 [id=05d29869a30847b6a0c137521965898a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:09.820657 28241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9098f3d36b24d578a3f8d7d3ccad052. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:09.822201 28242 tablet_service.cc:1467] Processing CreateTablet for tablet 775b8190036b4b828431fca19ce23ea9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=05d29869a30847b6a0c137521965898a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:09.822573 28242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 775b8190036b4b828431fca19ce23ea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:09.832816 29091 tablet_bootstrap.cc:492] T 775b8190036b4b828431fca19ce23ea9 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:09.832834 29088 tablet_bootstrap.cc:492] T d94f5824a71f454f8a1bfa13dd032684 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:09.834556 29091 tablet_bootstrap.cc:654] T 775b8190036b4b828431fca19ce23ea9 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:09.834630 29088 tablet_bootstrap.cc:654] T d94f5824a71f454f8a1bfa13dd032684 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:09.835997 29089 tablet_bootstrap.cc:492] T d94f5824a71f454f8a1bfa13dd032684 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:09.838161 29089 tablet_bootstrap.cc:654] T d94f5824a71f454f8a1bfa13dd032684 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:09.840147 29088 tablet_bootstrap.cc:492] T d94f5824a71f454f8a1bfa13dd032684 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:09.840289 29088 ts_tablet_manager.cc:1367] T d94f5824a71f454f8a1bfa13dd032684 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20240503 08:12:09.841051 29089 tablet_bootstrap.cc:492] T d94f5824a71f454f8a1bfa13dd032684 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:09.841064 29088 raft_consensus.cc:348] T d94f5824a71f454f8a1bfa13dd032684 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:09.841255 29089 ts_tablet_manager.cc:1367] T d94f5824a71f454f8a1bfa13dd032684 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 08:12:09.841321 29088 raft_consensus.cc:374] T d94f5824a71f454f8a1bfa13dd032684 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:09.841452 29088 raft_consensus.cc:729] T d94f5824a71f454f8a1bfa13dd032684 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:09.841748 29088 consensus_queue.cc:260] T d94f5824a71f454f8a1bfa13dd032684 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:09.842072 29089 raft_consensus.cc:348] T d94f5824a71f454f8a1bfa13dd032684 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:09.842387 29089 raft_consensus.cc:374] T d94f5824a71f454f8a1bfa13dd032684 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:09.842485 29088 ts_tablet_manager.cc:1398] T d94f5824a71f454f8a1bfa13dd032684 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:12:09.842530 29089 raft_consensus.cc:729] T d94f5824a71f454f8a1bfa13dd032684 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:09.842777 29088 tablet_bootstrap.cc:492] T d9098f3d36b24d578a3f8d7d3ccad052 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:09.842787 29089 consensus_queue.cc:260] T d94f5824a71f454f8a1bfa13dd032684 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:09.843343 29089 ts_tablet_manager.cc:1398] T d94f5824a71f454f8a1bfa13dd032684 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:09.843660 29089 tablet_bootstrap.cc:492] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:09.844273 29088 tablet_bootstrap.cc:654] T d9098f3d36b24d578a3f8d7d3ccad052 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:09.845132 29089 tablet_bootstrap.cc:654] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:09.850703 29090 tablet_bootstrap.cc:492] T d9098f3d36b24d578a3f8d7d3ccad052 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:09.850764 29091 tablet_bootstrap.cc:492] T 775b8190036b4b828431fca19ce23ea9 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:09.851316 29091 ts_tablet_manager.cc:1367] T 775b8190036b4b828431fca19ce23ea9 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.005s
I20240503 08:12:09.851925 29089 tablet_bootstrap.cc:492] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:09.852077 29089 ts_tablet_manager.cc:1367] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240503 08:12:09.852262 29091 raft_consensus.cc:348] T 775b8190036b4b828431fca19ce23ea9 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:09.852504 29088 tablet_bootstrap.cc:492] T d9098f3d36b24d578a3f8d7d3ccad052 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:09.852530 29091 raft_consensus.cc:374] T 775b8190036b4b828431fca19ce23ea9 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:09.852641 29091 raft_consensus.cc:729] T 775b8190036b4b828431fca19ce23ea9 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:09.852770 29088 ts_tablet_manager.cc:1367] T d9098f3d36b24d578a3f8d7d3ccad052 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20240503 08:12:09.852771 29089 raft_consensus.cc:348] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:09.853035 29089 raft_consensus.cc:374] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:09.852937 29091 consensus_queue.cc:260] T 775b8190036b4b828431fca19ce23ea9 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:09.853133 29089 raft_consensus.cc:729] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:09.853736 29091 ts_tablet_manager.cc:1398] T 775b8190036b4b828431fca19ce23ea9 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:09.854074 29091 tablet_bootstrap.cc:492] T d94f5824a71f454f8a1bfa13dd032684 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:09.853998 29088 raft_consensus.cc:348] T d9098f3d36b24d578a3f8d7d3ccad052 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:09.854338 29088 raft_consensus.cc:374] T d9098f3d36b24d578a3f8d7d3ccad052 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:09.854497 29088 raft_consensus.cc:729] T d9098f3d36b24d578a3f8d7d3ccad052 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:09.854902 29088 consensus_queue.cc:260] T d9098f3d36b24d578a3f8d7d3ccad052 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:09.855626 29088 ts_tablet_manager.cc:1398] T d9098f3d36b24d578a3f8d7d3ccad052 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:09.855803 29091 tablet_bootstrap.cc:654] T d94f5824a71f454f8a1bfa13dd032684 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:09.856148 29088 tablet_bootstrap.cc:492] T 775b8190036b4b828431fca19ce23ea9 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:09.858258 29088 tablet_bootstrap.cc:654] T 775b8190036b4b828431fca19ce23ea9 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:09.859975 29091 tablet_bootstrap.cc:492] T d94f5824a71f454f8a1bfa13dd032684 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:09.860123 29091 ts_tablet_manager.cc:1367] T d94f5824a71f454f8a1bfa13dd032684 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20240503 08:12:09.860826 29091 raft_consensus.cc:348] T d94f5824a71f454f8a1bfa13dd032684 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:09.861020 29091 raft_consensus.cc:374] T d94f5824a71f454f8a1bfa13dd032684 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:09.861097 29091 raft_consensus.cc:729] T d94f5824a71f454f8a1bfa13dd032684 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:09.861194 29089 consensus_queue.cc:260] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:09.861322 29091 consensus_queue.cc:260] T d94f5824a71f454f8a1bfa13dd032684 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:09.861802 29091 ts_tablet_manager.cc:1398] T d94f5824a71f454f8a1bfa13dd032684 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:12:09.861845 29089 ts_tablet_manager.cc:1398] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 08:12:09.862097 29091 tablet_bootstrap.cc:492] T d9098f3d36b24d578a3f8d7d3ccad052 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:09.863480 29091 tablet_bootstrap.cc:654] T d9098f3d36b24d578a3f8d7d3ccad052 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:09.863832 29090 tablet_bootstrap.cc:654] T d9098f3d36b24d578a3f8d7d3ccad052 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:09.865303 29088 tablet_bootstrap.cc:492] T 775b8190036b4b828431fca19ce23ea9 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:09.865464 29088 ts_tablet_manager.cc:1367] T 775b8190036b4b828431fca19ce23ea9 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240503 08:12:09.866230 29088 raft_consensus.cc:348] T 775b8190036b4b828431fca19ce23ea9 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:09.866412 29088 raft_consensus.cc:374] T 775b8190036b4b828431fca19ce23ea9 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:09.866489 29088 raft_consensus.cc:729] T 775b8190036b4b828431fca19ce23ea9 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:09.866703 29088 consensus_queue.cc:260] T 775b8190036b4b828431fca19ce23ea9 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:09.867749 29088 ts_tablet_manager.cc:1398] T 775b8190036b4b828431fca19ce23ea9 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:09.869076 29090 tablet_bootstrap.cc:492] T d9098f3d36b24d578a3f8d7d3ccad052 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:09.869171 29091 tablet_bootstrap.cc:492] T d9098f3d36b24d578a3f8d7d3ccad052 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:09.869227 29090 ts_tablet_manager.cc:1367] T d9098f3d36b24d578a3f8d7d3ccad052 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.004s
I20240503 08:12:09.869289 29091 ts_tablet_manager.cc:1367] T d9098f3d36b24d578a3f8d7d3ccad052 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240503 08:12:09.869973 29091 raft_consensus.cc:348] T d9098f3d36b24d578a3f8d7d3ccad052 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:09.870059 29090 raft_consensus.cc:348] T d9098f3d36b24d578a3f8d7d3ccad052 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:09.870249 29090 raft_consensus.cc:374] T d9098f3d36b24d578a3f8d7d3ccad052 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:09.870332 29090 raft_consensus.cc:729] T d9098f3d36b24d578a3f8d7d3ccad052 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:09.870199 29091 raft_consensus.cc:374] T d9098f3d36b24d578a3f8d7d3ccad052 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:09.870579 29090 consensus_queue.cc:260] T d9098f3d36b24d578a3f8d7d3ccad052 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:09.870519 29091 raft_consensus.cc:729] T d9098f3d36b24d578a3f8d7d3ccad052 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:09.870944 29091 consensus_queue.cc:260] T d9098f3d36b24d578a3f8d7d3ccad052 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:09.871145 29090 ts_tablet_manager.cc:1398] T d9098f3d36b24d578a3f8d7d3ccad052 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:09.871459 29091 ts_tablet_manager.cc:1398] T d9098f3d36b24d578a3f8d7d3ccad052 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:09.892697 29066 raft_consensus.cc:482] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:09.892885 29066 raft_consensus.cc:504] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:09.893195 28996 raft_consensus.cc:482] T d9098f3d36b24d578a3f8d7d3ccad052 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:09.893337 28996 raft_consensus.cc:504] T d9098f3d36b24d578a3f8d7d3ccad052 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:09.893970 28996 leader_election.cc:288] T d9098f3d36b24d578a3f8d7d3ccad052 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:12:09.894467 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9098f3d36b24d578a3f8d7d3ccad052" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:12:09.894551 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9098f3d36b24d578a3f8d7d3ccad052" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:12:09.894675 28132 raft_consensus.cc:2454] T d9098f3d36b24d578a3f8d7d3ccad052 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:12:09.894770 28002 raft_consensus.cc:2454] T d9098f3d36b24d578a3f8d7d3ccad052 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:12:09.894758 29066 leader_election.cc:288] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:09.895231 28198 leader_election.cc:302] T d9098f3d36b24d578a3f8d7d3ccad052 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:09.895493 28996 raft_consensus.cc:2781] T d9098f3d36b24d578a3f8d7d3ccad052 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:09.895609 28996 raft_consensus.cc:482] T d9098f3d36b24d578a3f8d7d3ccad052 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:09.895686 28996 raft_consensus.cc:3037] T d9098f3d36b24d578a3f8d7d3ccad052 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:09.898337 28996 raft_consensus.cc:504] T d9098f3d36b24d578a3f8d7d3ccad052 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:09.899039 28996 leader_election.cc:288] T d9098f3d36b24d578a3f8d7d3ccad052 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Requested vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:12:09.899497 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9098f3d36b24d578a3f8d7d3ccad052" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:12:09.899471 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9098f3d36b24d578a3f8d7d3ccad052" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:12:09.899681 28002 raft_consensus.cc:3037] T d9098f3d36b24d578a3f8d7d3ccad052 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:09.899698 28132 raft_consensus.cc:3037] T d9098f3d36b24d578a3f8d7d3ccad052 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:09.902426 28002 raft_consensus.cc:2454] T d9098f3d36b24d578a3f8d7d3ccad052 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:09.902539 28132 raft_consensus.cc:2454] T d9098f3d36b24d578a3f8d7d3ccad052 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:09.902796 28199 leader_election.cc:302] T d9098f3d36b24d578a3f8d7d3ccad052 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:09.903155 28996 raft_consensus.cc:2781] T d9098f3d36b24d578a3f8d7d3ccad052 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:09.903295 28996 raft_consensus.cc:686] T d9098f3d36b24d578a3f8d7d3ccad052 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Becoming Leader. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Running, Role: LEADER
I20240503 08:12:09.903581 28996 consensus_queue.cc:237] T d9098f3d36b24d578a3f8d7d3ccad052 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:09.904170 28263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "775b8190036b4b828431fca19ce23ea9" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:12:09.904381 28263 raft_consensus.cc:2454] T 775b8190036b4b828431fca19ce23ea9 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:12:09.904892 28327 leader_election.cc:302] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:09.905254 29066 raft_consensus.cc:2781] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:09.905212 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "775b8190036b4b828431fca19ce23ea9" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:12:09.905360 29066 raft_consensus.cc:482] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:09.905439 28132 raft_consensus.cc:2454] T 775b8190036b4b828431fca19ce23ea9 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:12:09.905453 29066 raft_consensus.cc:3037] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:09.907459 29066 raft_consensus.cc:504] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:09.907997 29066 leader_election.cc:288] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:09.908428 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "775b8190036b4b828431fca19ce23ea9" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:12:09.908454 28263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "775b8190036b4b828431fca19ce23ea9" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:12:09.908618 28132 raft_consensus.cc:3037] T 775b8190036b4b828431fca19ce23ea9 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:09.908812 28263 raft_consensus.cc:3037] T 775b8190036b4b828431fca19ce23ea9 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:09.910351 28999 raft_consensus.cc:482] T d94f5824a71f454f8a1bfa13dd032684 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:09.910475 28999 raft_consensus.cc:504] T d94f5824a71f454f8a1bfa13dd032684 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:09.910285 27859 catalog_manager.cc:5484] T d9098f3d36b24d578a3f8d7d3ccad052 P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: term changed from 0 to 1, leader changed from <none> to aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67). New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:12:09.911129 28999 leader_election.cc:288] T d94f5824a71f454f8a1bfa13dd032684 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:09.911720 28392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d94f5824a71f454f8a1bfa13dd032684" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:09.911937 28392 raft_consensus.cc:2454] T d94f5824a71f454f8a1bfa13dd032684 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:09.912189 28263 raft_consensus.cc:2454] T 775b8190036b4b828431fca19ce23ea9 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:12:09.912380 28069 leader_election.cc:302] T d94f5824a71f454f8a1bfa13dd032684 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:09.912624 28999 raft_consensus.cc:2781] T d94f5824a71f454f8a1bfa13dd032684 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:09.912739 28999 raft_consensus.cc:482] T d94f5824a71f454f8a1bfa13dd032684 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:09.912828 28999 raft_consensus.cc:3037] T d94f5824a71f454f8a1bfa13dd032684 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:09.912871 28327 leader_election.cc:302] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:09.913132 29066 raft_consensus.cc:2781] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:09.915678 28999 raft_consensus.cc:504] T d94f5824a71f454f8a1bfa13dd032684 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:09.916380 28132 raft_consensus.cc:2454] T 775b8190036b4b828431fca19ce23ea9 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:12:09.916730 28999 leader_election.cc:288] T d94f5824a71f454f8a1bfa13dd032684 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:09.917297 28392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d94f5824a71f454f8a1bfa13dd032684" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:12:09.917495 28392 raft_consensus.cc:3037] T d94f5824a71f454f8a1bfa13dd032684 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:09.917939 29066 raft_consensus.cc:686] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Becoming Leader. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Running, Role: LEADER
I20240503 08:12:09.918316 28263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d94f5824a71f454f8a1bfa13dd032684" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:12:09.918288 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d94f5824a71f454f8a1bfa13dd032684" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:12:09.918540 28263 raft_consensus.cc:2454] T d94f5824a71f454f8a1bfa13dd032684 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:09.918498 29066 consensus_queue.cc:237] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:09.919976 28392 raft_consensus.cc:2454] T d94f5824a71f454f8a1bfa13dd032684 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:12:09.920310 28069 leader_election.cc:302] T d94f5824a71f454f8a1bfa13dd032684 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, d09a11a03989421b9f28019dc1c14cc2; no voters: aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:12:09.920749 28999 raft_consensus.cc:2781] T d94f5824a71f454f8a1bfa13dd032684 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:09.920873 28999 raft_consensus.cc:686] T d94f5824a71f454f8a1bfa13dd032684 P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Becoming Leader. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Running, Role: LEADER
I20240503 08:12:09.921123 28999 consensus_queue.cc:237] T d94f5824a71f454f8a1bfa13dd032684 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:09.921749 27859 catalog_manager.cc:5484] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68). New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:12:09.923341 27860 catalog_manager.cc:5484] T d94f5824a71f454f8a1bfa13dd032684 P 1b411c6bb25549368041773d7b710d48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b411c6bb25549368041773d7b710d48 (127.20.230.66). New cstate: current_term: 1 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
W20240503 08:12:10.258103 29108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:10.258436 29108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:10.265954 29108 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 08:12:10.351083 28262 raft_consensus.cc:3037] T d94f5824a71f454f8a1bfa13dd032684 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:10.353724 28999 consensus_queue.cc:1035] T d94f5824a71f454f8a1bfa13dd032684 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:10.363951 29060 consensus_queue.cc:1035] T d94f5824a71f454f8a1bfa13dd032684 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:10.370205 28996 consensus_queue.cc:1035] T d9098f3d36b24d578a3f8d7d3ccad052 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:10.372822 28996 consensus_queue.cc:1035] T d9098f3d36b24d578a3f8d7d3ccad052 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:10.433966 29066 consensus_queue.cc:1035] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:10.446733 29066 consensus_queue.cc:1035] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:10.532817 28327 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.67:39919, user_credentials={real_user=slave}} blocked reactor thread for 62642.5us
INSERT report
rows total: 2000
time total: 230.075 ms
time per row: 0.115037 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 08:12:11.539151 27861 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:11.547670 27981 tablet_service.cc:1467] Processing CreateTablet for tablet c16e3017f7ca40e5b65b0f8e92f2eaa9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=bc6e5facee9743cbb5a079a9f15c3c2f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:11.548076 27981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c16e3017f7ca40e5b65b0f8e92f2eaa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:11.549391 28111 tablet_service.cc:1467] Processing CreateTablet for tablet c16e3017f7ca40e5b65b0f8e92f2eaa9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=bc6e5facee9743cbb5a079a9f15c3c2f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:11.549474 28112 tablet_service.cc:1467] Processing CreateTablet for tablet 8d70b2d43b2945938aba969c7b6d931d (DEFAULT_TABLE table=rebalance_test_table_2 [id=bc6e5facee9743cbb5a079a9f15c3c2f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:11.550029 27982 tablet_service.cc:1467] Processing CreateTablet for tablet ebb368e3f37945b0aa7526873bf0e5c3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=bc6e5facee9743cbb5a079a9f15c3c2f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:11.550109 27983 tablet_service.cc:1467] Processing CreateTablet for tablet 8d70b2d43b2945938aba969c7b6d931d (DEFAULT_TABLE table=rebalance_test_table_2 [id=bc6e5facee9743cbb5a079a9f15c3c2f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:11.550432 27982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebb368e3f37945b0aa7526873bf0e5c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:11.550482 28111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c16e3017f7ca40e5b65b0f8e92f2eaa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:11.553917 28112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d70b2d43b2945938aba969c7b6d931d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:11.554152 28242 tablet_service.cc:1467] Processing CreateTablet for tablet ebb368e3f37945b0aa7526873bf0e5c3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=bc6e5facee9743cbb5a079a9f15c3c2f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:11.554570 28242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebb368e3f37945b0aa7526873bf0e5c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:11.556560 27983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d70b2d43b2945938aba969c7b6d931d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:11.556885 29170 tablet_bootstrap.cc:492] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:11.558147 28373 tablet_service.cc:1467] Processing CreateTablet for tablet c16e3017f7ca40e5b65b0f8e92f2eaa9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=bc6e5facee9743cbb5a079a9f15c3c2f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:11.558530 28373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c16e3017f7ca40e5b65b0f8e92f2eaa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:11.558533 29170 tablet_bootstrap.cc:654] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:11.562088 28372 tablet_service.cc:1467] Processing CreateTablet for tablet ebb368e3f37945b0aa7526873bf0e5c3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=bc6e5facee9743cbb5a079a9f15c3c2f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:11.562438 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebb368e3f37945b0aa7526873bf0e5c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:11.566049 28371 tablet_service.cc:1467] Processing CreateTablet for tablet 8d70b2d43b2945938aba969c7b6d931d (DEFAULT_TABLE table=rebalance_test_table_2 [id=bc6e5facee9743cbb5a079a9f15c3c2f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:11.566304 28371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d70b2d43b2945938aba969c7b6d931d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:11.568548 29172 tablet_bootstrap.cc:492] T ebb368e3f37945b0aa7526873bf0e5c3 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:11.570185 29172 tablet_bootstrap.cc:654] T ebb368e3f37945b0aa7526873bf0e5c3 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:11.576319 29175 tablet_bootstrap.cc:492] T ebb368e3f37945b0aa7526873bf0e5c3 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:11.577579 29170 tablet_bootstrap.cc:492] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:11.577731 29170 ts_tablet_manager.cc:1367] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.010s
I20240503 08:12:11.577719 29173 tablet_bootstrap.cc:492] T 8d70b2d43b2945938aba969c7b6d931d P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:11.578346 29175 tablet_bootstrap.cc:654] T ebb368e3f37945b0aa7526873bf0e5c3 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:11.578675 29170 raft_consensus.cc:348] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:11.578860 29172 tablet_bootstrap.cc:492] T ebb368e3f37945b0aa7526873bf0e5c3 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:11.578989 29170 raft_consensus.cc:374] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:11.579025 29172 ts_tablet_manager.cc:1367] T ebb368e3f37945b0aa7526873bf0e5c3 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.001s
I20240503 08:12:11.579254 29173 tablet_bootstrap.cc:654] T 8d70b2d43b2945938aba969c7b6d931d P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:11.579901 29172 raft_consensus.cc:348] T ebb368e3f37945b0aa7526873bf0e5c3 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:11.580127 29172 raft_consensus.cc:374] T ebb368e3f37945b0aa7526873bf0e5c3 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:11.580238 29172 raft_consensus.cc:729] T ebb368e3f37945b0aa7526873bf0e5c3 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:11.580534 29172 consensus_queue.cc:260] T ebb368e3f37945b0aa7526873bf0e5c3 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:11.581817 29172 ts_tablet_manager.cc:1398] T ebb368e3f37945b0aa7526873bf0e5c3 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:12:11.582134 29172 tablet_bootstrap.cc:492] T 8d70b2d43b2945938aba969c7b6d931d P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:11.579154 29170 raft_consensus.cc:729] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:11.583804 29172 tablet_bootstrap.cc:654] T 8d70b2d43b2945938aba969c7b6d931d P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:11.584836 29170 consensus_queue.cc:260] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:11.587520 29170 ts_tablet_manager.cc:1398] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 08:12:11.588831 29170 tablet_bootstrap.cc:492] T 8d70b2d43b2945938aba969c7b6d931d P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:11.590345 29175 tablet_bootstrap.cc:492] T ebb368e3f37945b0aa7526873bf0e5c3 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:11.590564 29175 ts_tablet_manager.cc:1367] T ebb368e3f37945b0aa7526873bf0e5c3 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.006s
I20240503 08:12:11.591353 29175 raft_consensus.cc:348] T ebb368e3f37945b0aa7526873bf0e5c3 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:11.591619 29175 raft_consensus.cc:374] T ebb368e3f37945b0aa7526873bf0e5c3 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:11.591750 29175 raft_consensus.cc:729] T ebb368e3f37945b0aa7526873bf0e5c3 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:11.591928 29170 tablet_bootstrap.cc:654] T 8d70b2d43b2945938aba969c7b6d931d P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:11.592129 29175 consensus_queue.cc:260] T ebb368e3f37945b0aa7526873bf0e5c3 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:11.592849 29175 ts_tablet_manager.cc:1398] T ebb368e3f37945b0aa7526873bf0e5c3 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:11.594372 29170 tablet_bootstrap.cc:492] T 8d70b2d43b2945938aba969c7b6d931d P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:11.594527 29170 ts_tablet_manager.cc:1367] T 8d70b2d43b2945938aba969c7b6d931d P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20240503 08:12:11.594841 29172 tablet_bootstrap.cc:492] T 8d70b2d43b2945938aba969c7b6d931d P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:11.594990 29172 ts_tablet_manager.cc:1367] T 8d70b2d43b2945938aba969c7b6d931d P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.010s
I20240503 08:12:11.595050 29173 tablet_bootstrap.cc:492] T 8d70b2d43b2945938aba969c7b6d931d P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:11.595221 29173 ts_tablet_manager.cc:1367] T 8d70b2d43b2945938aba969c7b6d931d P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.008s
I20240503 08:12:11.595199 29170 raft_consensus.cc:348] T 8d70b2d43b2945938aba969c7b6d931d P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:11.595410 29170 raft_consensus.cc:374] T 8d70b2d43b2945938aba969c7b6d931d P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:11.595485 29170 raft_consensus.cc:729] T 8d70b2d43b2945938aba969c7b6d931d P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:11.595717 29170 consensus_queue.cc:260] T 8d70b2d43b2945938aba969c7b6d931d P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:11.595969 29173 raft_consensus.cc:348] T 8d70b2d43b2945938aba969c7b6d931d P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:11.596210 29170 ts_tablet_manager.cc:1398] T 8d70b2d43b2945938aba969c7b6d931d P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:11.596211 29173 raft_consensus.cc:374] T 8d70b2d43b2945938aba969c7b6d931d P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:11.596298 29173 raft_consensus.cc:729] T 8d70b2d43b2945938aba969c7b6d931d P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:11.596554 29173 consensus_queue.cc:260] T 8d70b2d43b2945938aba969c7b6d931d P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:11.597249 29173 ts_tablet_manager.cc:1398] T 8d70b2d43b2945938aba969c7b6d931d P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:11.597527 29173 tablet_bootstrap.cc:492] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:11.597506 29172 raft_consensus.cc:348] T 8d70b2d43b2945938aba969c7b6d931d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:11.597751 29172 raft_consensus.cc:374] T 8d70b2d43b2945938aba969c7b6d931d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:11.597855 29172 raft_consensus.cc:729] T 8d70b2d43b2945938aba969c7b6d931d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:11.598119 29172 consensus_queue.cc:260] T 8d70b2d43b2945938aba969c7b6d931d P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:11.598647 29172 ts_tablet_manager.cc:1398] T 8d70b2d43b2945938aba969c7b6d931d P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 08:12:11.598966 29172 tablet_bootstrap.cc:492] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:11.599261 29173 tablet_bootstrap.cc:654] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:11.601032 29172 tablet_bootstrap.cc:654] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:11.603092 29173 tablet_bootstrap.cc:492] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:11.603092 29172 tablet_bootstrap.cc:492] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:11.603228 29172 ts_tablet_manager.cc:1367] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 08:12:11.603250 29173 ts_tablet_manager.cc:1367] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240503 08:12:11.603881 29173 raft_consensus.cc:348] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:11.603907 29172 raft_consensus.cc:348] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:11.604079 29172 raft_consensus.cc:374] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:11.604095 29173 raft_consensus.cc:374] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:11.604157 29173 raft_consensus.cc:729] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:11.604173 29172 raft_consensus.cc:729] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:11.604377 29172 consensus_queue.cc:260] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:11.604403 29173 consensus_queue.cc:260] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:11.604897 29172 ts_tablet_manager.cc:1398] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:11.605294 29173 ts_tablet_manager.cc:1398] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:11.605530 29173 tablet_bootstrap.cc:492] T ebb368e3f37945b0aa7526873bf0e5c3 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:11.607255 29173 tablet_bootstrap.cc:654] T ebb368e3f37945b0aa7526873bf0e5c3 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:11.609090 29173 tablet_bootstrap.cc:492] T ebb368e3f37945b0aa7526873bf0e5c3 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:11.609225 29173 ts_tablet_manager.cc:1367] T ebb368e3f37945b0aa7526873bf0e5c3 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:11.609855 29173 raft_consensus.cc:348] T ebb368e3f37945b0aa7526873bf0e5c3 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:11.610033 29173 raft_consensus.cc:374] T ebb368e3f37945b0aa7526873bf0e5c3 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:11.610119 29173 raft_consensus.cc:729] T ebb368e3f37945b0aa7526873bf0e5c3 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:11.610317 29173 consensus_queue.cc:260] T ebb368e3f37945b0aa7526873bf0e5c3 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:11.610782 29173 ts_tablet_manager.cc:1398] T ebb368e3f37945b0aa7526873bf0e5c3 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:12:11.663933 29072 raft_consensus.cc:482] T ebb368e3f37945b0aa7526873bf0e5c3 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:11.664101 29072 raft_consensus.cc:504] T ebb368e3f37945b0aa7526873bf0e5c3 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:11.664732 29072 leader_election.cc:288] T ebb368e3f37945b0aa7526873bf0e5c3 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:11.665191 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebb368e3f37945b0aa7526873bf0e5c3" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:12:11.665242 28392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebb368e3f37945b0aa7526873bf0e5c3" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:11.665392 28262 raft_consensus.cc:2454] T ebb368e3f37945b0aa7526873bf0e5c3 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:11.665477 28392 raft_consensus.cc:2454] T ebb368e3f37945b0aa7526873bf0e5c3 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:11.665825 27937 leader_election.cc:302] T ebb368e3f37945b0aa7526873bf0e5c3 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:11.666070 29072 raft_consensus.cc:2781] T ebb368e3f37945b0aa7526873bf0e5c3 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:11.666184 29072 raft_consensus.cc:482] T ebb368e3f37945b0aa7526873bf0e5c3 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:11.666271 29072 raft_consensus.cc:3037] T ebb368e3f37945b0aa7526873bf0e5c3 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:11.668076 29072 raft_consensus.cc:504] T ebb368e3f37945b0aa7526873bf0e5c3 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:11.668625 29072 leader_election.cc:288] T ebb368e3f37945b0aa7526873bf0e5c3 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:11.668987 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebb368e3f37945b0aa7526873bf0e5c3" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:12:11.669044 28392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebb368e3f37945b0aa7526873bf0e5c3" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:12:11.669142 28262 raft_consensus.cc:3037] T ebb368e3f37945b0aa7526873bf0e5c3 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:11.669221 28392 raft_consensus.cc:3037] T ebb368e3f37945b0aa7526873bf0e5c3 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:11.670953 28262 raft_consensus.cc:2454] T ebb368e3f37945b0aa7526873bf0e5c3 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:11.670972 28392 raft_consensus.cc:2454] T ebb368e3f37945b0aa7526873bf0e5c3 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:11.671324 27937 leader_election.cc:302] T ebb368e3f37945b0aa7526873bf0e5c3 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:11.671617 29072 raft_consensus.cc:2781] T ebb368e3f37945b0aa7526873bf0e5c3 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:11.671960 29072 raft_consensus.cc:686] T ebb368e3f37945b0aa7526873bf0e5c3 P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Becoming Leader. State: Replica: 833582ff68c94269be364ebd187a75db, State: Running, Role: LEADER
I20240503 08:12:11.672252 29072 consensus_queue.cc:237] T ebb368e3f37945b0aa7526873bf0e5c3 P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:11.674702 27861 catalog_manager.cc:5484] T ebb368e3f37945b0aa7526873bf0e5c3 P 833582ff68c94269be364ebd187a75db reported cstate change: term changed from 0 to 1, leader changed from <none> to 833582ff68c94269be364ebd187a75db (127.20.230.65). New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:12:11.681449 29072 raft_consensus.cc:482] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:11.681591 29072 raft_consensus.cc:504] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:11.682183 29072 leader_election.cc:288] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:11.682574 28392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c16e3017f7ca40e5b65b0f8e92f2eaa9" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:11.682838 28392 raft_consensus.cc:2454] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:11.683255 27939 leader_election.cc:302] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:11.683506 29072 raft_consensus.cc:2781] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:11.683593 29072 raft_consensus.cc:482] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:11.683699 29072 raft_consensus.cc:3037] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:11.685463 29072 raft_consensus.cc:504] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:11.686079 29072 leader_election.cc:288] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Requested vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:11.686319 28392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c16e3017f7ca40e5b65b0f8e92f2eaa9" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:12:11.686511 28392 raft_consensus.cc:3037] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:11.688373 28133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c16e3017f7ca40e5b65b0f8e92f2eaa9" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:12:11.688455 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c16e3017f7ca40e5b65b0f8e92f2eaa9" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:12:11.688602 28392 raft_consensus.cc:2454] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:11.688625 28133 raft_consensus.cc:2454] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:11.688975 27939 leader_election.cc:302] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:11.689200 29072 raft_consensus.cc:2781] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:11.689352 29072 raft_consensus.cc:686] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Becoming Leader. State: Replica: 833582ff68c94269be364ebd187a75db, State: Running, Role: LEADER
I20240503 08:12:11.689554 29072 consensus_queue.cc:237] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:11.691931 27861 catalog_manager.cc:5484] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 833582ff68c94269be364ebd187a75db reported cstate change: term changed from 0 to 1, leader changed from <none> to 833582ff68c94269be364ebd187a75db (127.20.230.65). New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:12:11.702549 29072 raft_consensus.cc:482] T 8d70b2d43b2945938aba969c7b6d931d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:11.702699 29072 raft_consensus.cc:504] T 8d70b2d43b2945938aba969c7b6d931d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:11.703263 29072 leader_election.cc:288] T 8d70b2d43b2945938aba969c7b6d931d P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:11.703598 28392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d70b2d43b2945938aba969c7b6d931d" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:11.703795 28392 raft_consensus.cc:2454] T 8d70b2d43b2945938aba969c7b6d931d P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:11.703835 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d70b2d43b2945938aba969c7b6d931d" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:12:11.704057 28132 raft_consensus.cc:2454] T 8d70b2d43b2945938aba969c7b6d931d P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:11.704130 27939 leader_election.cc:302] T 8d70b2d43b2945938aba969c7b6d931d P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:11.704434 29072 raft_consensus.cc:2781] T 8d70b2d43b2945938aba969c7b6d931d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:11.704541 29072 raft_consensus.cc:482] T 8d70b2d43b2945938aba969c7b6d931d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:11.704622 29072 raft_consensus.cc:3037] T 8d70b2d43b2945938aba969c7b6d931d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:11.706377 29072 raft_consensus.cc:504] T 8d70b2d43b2945938aba969c7b6d931d P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:11.706840 29072 leader_election.cc:288] T 8d70b2d43b2945938aba969c7b6d931d P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Requested vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:11.707158 28392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d70b2d43b2945938aba969c7b6d931d" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:12:11.707226 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d70b2d43b2945938aba969c7b6d931d" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:12:11.707340 28392 raft_consensus.cc:3037] T 8d70b2d43b2945938aba969c7b6d931d P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:11.707392 28132 raft_consensus.cc:3037] T 8d70b2d43b2945938aba969c7b6d931d P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:11.709059 28392 raft_consensus.cc:2454] T 8d70b2d43b2945938aba969c7b6d931d P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:11.709107 28132 raft_consensus.cc:2454] T 8d70b2d43b2945938aba969c7b6d931d P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:11.709424 27939 leader_election.cc:302] T 8d70b2d43b2945938aba969c7b6d931d P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:11.709676 29072 raft_consensus.cc:2781] T 8d70b2d43b2945938aba969c7b6d931d P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:11.709846 29072 raft_consensus.cc:686] T 8d70b2d43b2945938aba969c7b6d931d P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Becoming Leader. State: Replica: 833582ff68c94269be364ebd187a75db, State: Running, Role: LEADER
I20240503 08:12:11.710090 29072 consensus_queue.cc:237] T 8d70b2d43b2945938aba969c7b6d931d P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:11.712304 27861 catalog_manager.cc:5484] T 8d70b2d43b2945938aba969c7b6d931d P 833582ff68c94269be364ebd187a75db reported cstate change: term changed from 0 to 1, leader changed from <none> to 833582ff68c94269be364ebd187a75db (127.20.230.65). New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
W20240503 08:12:12.050987 29189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:12.051306 29189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:12.058925 29189 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:12:12.059882 29072 consensus_queue.cc:1035] T ebb368e3f37945b0aa7526873bf0e5c3 P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:12.066176 29183 consensus_queue.cc:1035] T ebb368e3f37945b0aa7526873bf0e5c3 P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240503 08:12:12.186503 28132 raft_consensus.cc:1261] T 8d70b2d43b2945938aba969c7b6d931d P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:12.187069 28392 raft_consensus.cc:1261] T 8d70b2d43b2945938aba969c7b6d931d P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:12.187861 29196 consensus_queue.cc:1035] T 8d70b2d43b2945938aba969c7b6d931d P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:12.188158 29183 consensus_queue.cc:1035] T 8d70b2d43b2945938aba969c7b6d931d P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:12.190713 28393 raft_consensus.cc:1261] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:12.190982 28133 raft_consensus.cc:3037] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:12.191519 29072 consensus_queue.cc:1035] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:12.203394 28133 raft_consensus.cc:1261] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:12.204039 29196 consensus_queue.cc:1035] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:12.252282 29194 mvcc.cc:205] Tried to move back new op lower bound from 7023509226245144576 to 7023509224202805248. Current Snapshot: MvccSnapshot[applied={T|T < 7023509226245144576}]
I20240503 08:12:12.267627 29198 mvcc.cc:205] Tried to move back new op lower bound from 7023509226245144576 to 7023509224202805248. Current Snapshot: MvccSnapshot[applied={T|T < 7023509226245144576}]
I20240503 08:12:12.289664 29224 mvcc.cc:205] Tried to move back new op lower bound from 7023509226230767616 to 7023509224286494720. Current Snapshot: MvccSnapshot[applied={T|T < 7023509226230767616}]
INSERT report
rows total: 2000
time total: 230.267 ms
time per row: 0.115134 ms
I20240503 08:12:12.425581 27856 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:12.434535 28111 tablet_service.cc:1467] Processing CreateTablet for tablet 8450e62352a74d2ea1471166d4cb1132 (DEFAULT_TABLE table=rebalance_test_table_3 [id=8e9aac1c8668451485cb809d15f7c27d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:12.435021 28111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8450e62352a74d2ea1471166d4cb1132. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:12.434945 27981 tablet_service.cc:1467] Processing CreateTablet for tablet 8450e62352a74d2ea1471166d4cb1132 (DEFAULT_TABLE table=rebalance_test_table_3 [id=8e9aac1c8668451485cb809d15f7c27d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:12.435047 27982 tablet_service.cc:1467] Processing CreateTablet for tablet d3da6397e0c34c8f96933963dd4fa65e (DEFAULT_TABLE table=rebalance_test_table_3 [id=8e9aac1c8668451485cb809d15f7c27d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:12.435428 27981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8450e62352a74d2ea1471166d4cb1132. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:12.435633 27982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3da6397e0c34c8f96933963dd4fa65e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:12.435647 28242 tablet_service.cc:1467] Processing CreateTablet for tablet 8450e62352a74d2ea1471166d4cb1132 (DEFAULT_TABLE table=rebalance_test_table_3 [id=8e9aac1c8668451485cb809d15f7c27d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:12.436026 28242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8450e62352a74d2ea1471166d4cb1132. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:12.438678 28243 tablet_service.cc:1467] Processing CreateTablet for tablet d3da6397e0c34c8f96933963dd4fa65e (DEFAULT_TABLE table=rebalance_test_table_3 [id=8e9aac1c8668451485cb809d15f7c27d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:12.439039 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3da6397e0c34c8f96933963dd4fa65e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:12.442082 28241 tablet_service.cc:1467] Processing CreateTablet for tablet 8bcbc022caa94bb08a2db4de9b060557 (DEFAULT_TABLE table=rebalance_test_table_3 [id=8e9aac1c8668451485cb809d15f7c27d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:12.442412 28241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bcbc022caa94bb08a2db4de9b060557. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:12.443003 28372 tablet_service.cc:1467] Processing CreateTablet for tablet 8bcbc022caa94bb08a2db4de9b060557 (DEFAULT_TABLE table=rebalance_test_table_3 [id=8e9aac1c8668451485cb809d15f7c27d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:12.443454 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bcbc022caa94bb08a2db4de9b060557. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:12.445957 28371 tablet_service.cc:1467] Processing CreateTablet for tablet d3da6397e0c34c8f96933963dd4fa65e (DEFAULT_TABLE table=rebalance_test_table_3 [id=8e9aac1c8668451485cb809d15f7c27d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:12.446345 28371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3da6397e0c34c8f96933963dd4fa65e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:12.446674 28112 tablet_service.cc:1467] Processing CreateTablet for tablet 8bcbc022caa94bb08a2db4de9b060557 (DEFAULT_TABLE table=rebalance_test_table_3 [id=8e9aac1c8668451485cb809d15f7c27d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:12.447037 28112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bcbc022caa94bb08a2db4de9b060557. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:12.458197 29256 tablet_bootstrap.cc:492] T 8bcbc022caa94bb08a2db4de9b060557 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:12.459837 29256 tablet_bootstrap.cc:654] T 8bcbc022caa94bb08a2db4de9b060557 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:12.461995 29254 tablet_bootstrap.cc:492] T 8450e62352a74d2ea1471166d4cb1132 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:12.462055 29257 tablet_bootstrap.cc:492] T 8450e62352a74d2ea1471166d4cb1132 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:12.461570 29255 tablet_bootstrap.cc:492] T 8450e62352a74d2ea1471166d4cb1132 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:12.464244 29254 tablet_bootstrap.cc:654] T 8450e62352a74d2ea1471166d4cb1132 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:12.464283 29257 tablet_bootstrap.cc:654] T 8450e62352a74d2ea1471166d4cb1132 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:12.466653 29255 tablet_bootstrap.cc:654] T 8450e62352a74d2ea1471166d4cb1132 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:12.478502 29257 tablet_bootstrap.cc:492] T 8450e62352a74d2ea1471166d4cb1132 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:12.478653 29257 ts_tablet_manager.cc:1367] T 8450e62352a74d2ea1471166d4cb1132 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.007s
I20240503 08:12:12.479238 29257 raft_consensus.cc:348] T 8450e62352a74d2ea1471166d4cb1132 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:12.479396 29257 raft_consensus.cc:374] T 8450e62352a74d2ea1471166d4cb1132 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:12.479485 29257 raft_consensus.cc:729] T 8450e62352a74d2ea1471166d4cb1132 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:12.479692 29257 consensus_queue.cc:260] T 8450e62352a74d2ea1471166d4cb1132 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:12.480659 29255 tablet_bootstrap.cc:492] T 8450e62352a74d2ea1471166d4cb1132 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:12.480855 29255 ts_tablet_manager.cc:1367] T 8450e62352a74d2ea1471166d4cb1132 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.004s
I20240503 08:12:12.482270 29255 raft_consensus.cc:348] T 8450e62352a74d2ea1471166d4cb1132 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:12.482537 29255 raft_consensus.cc:374] T 8450e62352a74d2ea1471166d4cb1132 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:12.482668 29255 raft_consensus.cc:729] T 8450e62352a74d2ea1471166d4cb1132 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:12.483021 29257 ts_tablet_manager.cc:1398] T 8450e62352a74d2ea1471166d4cb1132 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 08:12:12.482936 29255 consensus_queue.cc:260] T 8450e62352a74d2ea1471166d4cb1132 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:12.484143 29255 ts_tablet_manager.cc:1398] T 8450e62352a74d2ea1471166d4cb1132 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:12.484464 29255 tablet_bootstrap.cc:492] T 8bcbc022caa94bb08a2db4de9b060557 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:12.486135 29257 tablet_bootstrap.cc:492] T d3da6397e0c34c8f96933963dd4fa65e P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:12.486725 29255 tablet_bootstrap.cc:654] T 8bcbc022caa94bb08a2db4de9b060557 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:12.488546 29257 tablet_bootstrap.cc:654] T d3da6397e0c34c8f96933963dd4fa65e P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:12.491046 29254 tablet_bootstrap.cc:492] T 8450e62352a74d2ea1471166d4cb1132 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:12.491217 29254 ts_tablet_manager.cc:1367] T 8450e62352a74d2ea1471166d4cb1132 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.011s
I20240503 08:12:12.492038 29254 raft_consensus.cc:348] T 8450e62352a74d2ea1471166d4cb1132 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:12.492238 29254 raft_consensus.cc:374] T 8450e62352a74d2ea1471166d4cb1132 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:12.492326 29254 raft_consensus.cc:729] T 8450e62352a74d2ea1471166d4cb1132 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:12.492587 29254 consensus_queue.cc:260] T 8450e62352a74d2ea1471166d4cb1132 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:12.493618 29256 tablet_bootstrap.cc:492] T 8bcbc022caa94bb08a2db4de9b060557 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:12.493796 29256 ts_tablet_manager.cc:1367] T 8bcbc022caa94bb08a2db4de9b060557 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.036s user 0.000s sys 0.016s
I20240503 08:12:12.494585 29256 raft_consensus.cc:348] T 8bcbc022caa94bb08a2db4de9b060557 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:12.494791 29256 raft_consensus.cc:374] T 8bcbc022caa94bb08a2db4de9b060557 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:12.494891 29256 raft_consensus.cc:729] T 8bcbc022caa94bb08a2db4de9b060557 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:12.495139 29256 consensus_queue.cc:260] T 8bcbc022caa94bb08a2db4de9b060557 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:12.498775 29256 ts_tablet_manager.cc:1398] T 8bcbc022caa94bb08a2db4de9b060557 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 08:12:12.499063 29256 tablet_bootstrap.cc:492] T d3da6397e0c34c8f96933963dd4fa65e P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:12.501137 29256 tablet_bootstrap.cc:654] T d3da6397e0c34c8f96933963dd4fa65e P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:12.502278 29254 ts_tablet_manager.cc:1398] T 8450e62352a74d2ea1471166d4cb1132 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.011s user 0.001s sys 0.002s
I20240503 08:12:12.502607 29254 tablet_bootstrap.cc:492] T d3da6397e0c34c8f96933963dd4fa65e P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:12.504097 29254 tablet_bootstrap.cc:654] T d3da6397e0c34c8f96933963dd4fa65e P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:12.508209 29254 tablet_bootstrap.cc:492] T d3da6397e0c34c8f96933963dd4fa65e P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:12.508360 29254 ts_tablet_manager.cc:1367] T d3da6397e0c34c8f96933963dd4fa65e P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20240503 08:12:12.509032 29254 raft_consensus.cc:348] T d3da6397e0c34c8f96933963dd4fa65e P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:12.509243 29254 raft_consensus.cc:374] T d3da6397e0c34c8f96933963dd4fa65e P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:12.509308 29254 raft_consensus.cc:729] T d3da6397e0c34c8f96933963dd4fa65e P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:12.511066 29256 tablet_bootstrap.cc:492] T d3da6397e0c34c8f96933963dd4fa65e P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:12.511220 29256 ts_tablet_manager.cc:1367] T d3da6397e0c34c8f96933963dd4fa65e P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.002s
I20240503 08:12:12.511952 29256 raft_consensus.cc:348] T d3da6397e0c34c8f96933963dd4fa65e P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:12.512174 29256 raft_consensus.cc:374] T d3da6397e0c34c8f96933963dd4fa65e P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:12.512256 29256 raft_consensus.cc:729] T d3da6397e0c34c8f96933963dd4fa65e P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:12.512537 29256 consensus_queue.cc:260] T d3da6397e0c34c8f96933963dd4fa65e P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:12.513635 29256 ts_tablet_manager.cc:1398] T d3da6397e0c34c8f96933963dd4fa65e P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:12.513986 29254 consensus_queue.cc:260] T d3da6397e0c34c8f96933963dd4fa65e P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:12.514639 29254 ts_tablet_manager.cc:1398] T d3da6397e0c34c8f96933963dd4fa65e P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240503 08:12:12.516397 29257 tablet_bootstrap.cc:492] T d3da6397e0c34c8f96933963dd4fa65e P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:12.516582 29257 ts_tablet_manager.cc:1367] T d3da6397e0c34c8f96933963dd4fa65e P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.025s
I20240503 08:12:12.517207 29255 tablet_bootstrap.cc:492] T 8bcbc022caa94bb08a2db4de9b060557 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:12.517421 29255 ts_tablet_manager.cc:1367] T 8bcbc022caa94bb08a2db4de9b060557 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.033s user 0.020s sys 0.007s
I20240503 08:12:12.517319 29257 raft_consensus.cc:348] T d3da6397e0c34c8f96933963dd4fa65e P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:12.517566 29257 raft_consensus.cc:374] T d3da6397e0c34c8f96933963dd4fa65e P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:12.517648 29257 raft_consensus.cc:729] T d3da6397e0c34c8f96933963dd4fa65e P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:12.517910 29257 consensus_queue.cc:260] T d3da6397e0c34c8f96933963dd4fa65e P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:12.518460 29257 ts_tablet_manager.cc:1398] T d3da6397e0c34c8f96933963dd4fa65e P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:12.518776 29257 tablet_bootstrap.cc:492] T 8bcbc022caa94bb08a2db4de9b060557 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:12.520156 29255 raft_consensus.cc:348] T 8bcbc022caa94bb08a2db4de9b060557 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:12.520460 29257 tablet_bootstrap.cc:654] T 8bcbc022caa94bb08a2db4de9b060557 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:12.520586 29255 raft_consensus.cc:374] T 8bcbc022caa94bb08a2db4de9b060557 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:12.520681 29255 raft_consensus.cc:729] T 8bcbc022caa94bb08a2db4de9b060557 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:12.521064 29255 consensus_queue.cc:260] T 8bcbc022caa94bb08a2db4de9b060557 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:12.522150 29255 ts_tablet_manager.cc:1398] T 8bcbc022caa94bb08a2db4de9b060557 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:12:12.524307 29257 tablet_bootstrap.cc:492] T 8bcbc022caa94bb08a2db4de9b060557 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:12.525285 29257 ts_tablet_manager.cc:1367] T 8bcbc022caa94bb08a2db4de9b060557 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20240503 08:12:12.526001 29257 raft_consensus.cc:348] T 8bcbc022caa94bb08a2db4de9b060557 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:12.526187 29257 raft_consensus.cc:374] T 8bcbc022caa94bb08a2db4de9b060557 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:12.526264 29257 raft_consensus.cc:729] T 8bcbc022caa94bb08a2db4de9b060557 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:12.526448 29257 consensus_queue.cc:260] T 8bcbc022caa94bb08a2db4de9b060557 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:12.526926 29257 ts_tablet_manager.cc:1398] T 8bcbc022caa94bb08a2db4de9b060557 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:12:12.700071 29239 raft_consensus.cc:482] T d3da6397e0c34c8f96933963dd4fa65e P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:12.700229 29239 raft_consensus.cc:504] T d3da6397e0c34c8f96933963dd4fa65e P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:12.700803 29239 leader_election.cc:288] T d3da6397e0c34c8f96933963dd4fa65e P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:12:12.701237 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3da6397e0c34c8f96933963dd4fa65e" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:12.701311 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3da6397e0c34c8f96933963dd4fa65e" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:12:12.701454 28391 raft_consensus.cc:2454] T d3da6397e0c34c8f96933963dd4fa65e P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:12.701565 28262 raft_consensus.cc:2454] T d3da6397e0c34c8f96933963dd4fa65e P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:12.701870 27939 leader_election.cc:302] T d3da6397e0c34c8f96933963dd4fa65e P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:12.702124 29239 raft_consensus.cc:2781] T d3da6397e0c34c8f96933963dd4fa65e P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:12.702263 29239 raft_consensus.cc:482] T d3da6397e0c34c8f96933963dd4fa65e P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:12.702349 29239 raft_consensus.cc:3037] T d3da6397e0c34c8f96933963dd4fa65e P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:12.704106 29239 raft_consensus.cc:504] T d3da6397e0c34c8f96933963dd4fa65e P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:12.704632 29239 leader_election.cc:288] T d3da6397e0c34c8f96933963dd4fa65e P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Requested vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:12:12.705026 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3da6397e0c34c8f96933963dd4fa65e" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:12:12.705093 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3da6397e0c34c8f96933963dd4fa65e" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:12:12.705235 28391 raft_consensus.cc:3037] T d3da6397e0c34c8f96933963dd4fa65e P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:12.705288 28262 raft_consensus.cc:3037] T d3da6397e0c34c8f96933963dd4fa65e P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:12.707981 28262 raft_consensus.cc:2454] T d3da6397e0c34c8f96933963dd4fa65e P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:12.707991 28391 raft_consensus.cc:2454] T d3da6397e0c34c8f96933963dd4fa65e P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:12.708410 27937 leader_election.cc:302] T d3da6397e0c34c8f96933963dd4fa65e P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:12.708675 29239 raft_consensus.cc:2781] T d3da6397e0c34c8f96933963dd4fa65e P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:12.708858 29239 raft_consensus.cc:686] T d3da6397e0c34c8f96933963dd4fa65e P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Becoming Leader. State: Replica: 833582ff68c94269be364ebd187a75db, State: Running, Role: LEADER
I20240503 08:12:12.709131 29239 consensus_queue.cc:237] T d3da6397e0c34c8f96933963dd4fa65e P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:12.711474 27859 catalog_manager.cc:5484] T d3da6397e0c34c8f96933963dd4fa65e P 833582ff68c94269be364ebd187a75db reported cstate change: term changed from 0 to 1, leader changed from <none> to 833582ff68c94269be364ebd187a75db (127.20.230.65). New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:12:12.742138 29023 raft_consensus.cc:482] T 8450e62352a74d2ea1471166d4cb1132 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:12.742343 29023 raft_consensus.cc:504] T 8450e62352a74d2ea1471166d4cb1132 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:12.742910 29023 leader_election.cc:288] T 8450e62352a74d2ea1471166d4cb1132 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:12:12.743345 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8450e62352a74d2ea1471166d4cb1132" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:12:12.743356 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8450e62352a74d2ea1471166d4cb1132" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:12:12.743554 28132 raft_consensus.cc:2454] T 8450e62352a74d2ea1471166d4cb1132 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:12:12.743604 28003 raft_consensus.cc:2454] T 8450e62352a74d2ea1471166d4cb1132 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:12:12.743992 28198 leader_election.cc:302] T 8450e62352a74d2ea1471166d4cb1132 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:12.744275 29023 raft_consensus.cc:2781] T 8450e62352a74d2ea1471166d4cb1132 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:12.744366 29023 raft_consensus.cc:482] T 8450e62352a74d2ea1471166d4cb1132 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:12.744462 29023 raft_consensus.cc:3037] T 8450e62352a74d2ea1471166d4cb1132 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:12.746352 29023 raft_consensus.cc:504] T 8450e62352a74d2ea1471166d4cb1132 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:12.746876 29023 leader_election.cc:288] T 8450e62352a74d2ea1471166d4cb1132 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Requested vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:12:12.747241 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8450e62352a74d2ea1471166d4cb1132" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:12:12.747295 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8450e62352a74d2ea1471166d4cb1132" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:12:12.747455 28003 raft_consensus.cc:3037] T 8450e62352a74d2ea1471166d4cb1132 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:12.747442 28132 raft_consensus.cc:3037] T 8450e62352a74d2ea1471166d4cb1132 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:12.750118 28132 raft_consensus.cc:2454] T 8450e62352a74d2ea1471166d4cb1132 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:12.750124 28003 raft_consensus.cc:2454] T 8450e62352a74d2ea1471166d4cb1132 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:12.750505 28198 leader_election.cc:302] T 8450e62352a74d2ea1471166d4cb1132 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 833582ff68c94269be364ebd187a75db, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:12.750771 29023 raft_consensus.cc:2781] T 8450e62352a74d2ea1471166d4cb1132 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:12.750928 29023 raft_consensus.cc:686] T 8450e62352a74d2ea1471166d4cb1132 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Becoming Leader. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Running, Role: LEADER
I20240503 08:12:12.751188 29023 consensus_queue.cc:237] T 8450e62352a74d2ea1471166d4cb1132 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:12.753685 27859 catalog_manager.cc:5484] T 8450e62352a74d2ea1471166d4cb1132 P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: term changed from 0 to 1, leader changed from <none> to aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67). New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:12:12.774736 29066 raft_consensus.cc:482] T 8bcbc022caa94bb08a2db4de9b060557 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:12.774967 29066 raft_consensus.cc:504] T 8bcbc022caa94bb08a2db4de9b060557 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:12.775620 29066 leader_election.cc:288] T 8bcbc022caa94bb08a2db4de9b060557 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:12.776083 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bcbc022caa94bb08a2db4de9b060557" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:12:12.776129 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bcbc022caa94bb08a2db4de9b060557" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:12:12.776306 28262 raft_consensus.cc:2454] T 8bcbc022caa94bb08a2db4de9b060557 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:12:12.776376 28132 raft_consensus.cc:2454] T 8bcbc022caa94bb08a2db4de9b060557 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:12:12.776713 28327 leader_election.cc:302] T 8bcbc022caa94bb08a2db4de9b060557 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:12.777006 29066 raft_consensus.cc:2781] T 8bcbc022caa94bb08a2db4de9b060557 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:12.777107 29066 raft_consensus.cc:482] T 8bcbc022caa94bb08a2db4de9b060557 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:12.777220 29066 raft_consensus.cc:3037] T 8bcbc022caa94bb08a2db4de9b060557 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:12.779141 29066 raft_consensus.cc:504] T 8bcbc022caa94bb08a2db4de9b060557 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:12.779695 29066 leader_election.cc:288] T 8bcbc022caa94bb08a2db4de9b060557 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:12.780068 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bcbc022caa94bb08a2db4de9b060557" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:12:12.780123 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bcbc022caa94bb08a2db4de9b060557" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:12:12.780243 28262 raft_consensus.cc:3037] T 8bcbc022caa94bb08a2db4de9b060557 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:12.780268 28132 raft_consensus.cc:3037] T 8bcbc022caa94bb08a2db4de9b060557 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:12.782071 28132 raft_consensus.cc:2454] T 8bcbc022caa94bb08a2db4de9b060557 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:12:12.782272 28262 raft_consensus.cc:2454] T 8bcbc022caa94bb08a2db4de9b060557 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:12:12.782434 28328 leader_election.cc:302] T 8bcbc022caa94bb08a2db4de9b060557 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:12.782667 29066 raft_consensus.cc:2781] T 8bcbc022caa94bb08a2db4de9b060557 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:12.783021 29066 raft_consensus.cc:686] T 8bcbc022caa94bb08a2db4de9b060557 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Becoming Leader. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Running, Role: LEADER
I20240503 08:12:12.783282 29066 consensus_queue.cc:237] T 8bcbc022caa94bb08a2db4de9b060557 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:12.785655 27859 catalog_manager.cc:5484] T 8bcbc022caa94bb08a2db4de9b060557 P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68). New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:12:13.116392 29239 consensus_queue.cc:1035] T d3da6397e0c34c8f96933963dd4fa65e P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:13.121266 29239 consensus_queue.cc:1035] T d3da6397e0c34c8f96933963dd4fa65e P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:13.146792 29243 consensus_queue.cc:1035] T 8450e62352a74d2ea1471166d4cb1132 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:13.151090 29023 consensus_queue.cc:1035] T 8450e62352a74d2ea1471166d4cb1132 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:13.161151 29271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:13.161479 29271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:13.168797 29271 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 08:12:13.245533 29066 consensus_queue.cc:1035] T 8bcbc022caa94bb08a2db4de9b060557 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:13.252280 29066 consensus_queue.cc:1035] T 8bcbc022caa94bb08a2db4de9b060557 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 189.379 ms
time per row: 0.0946896 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 08:12:14.371315 27859 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:14.380237 28242 tablet_service.cc:1467] Processing CreateTablet for tablet c6498d83236c48f88bd21c303c9c9ab3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=acb2583756a845cab3b0cb05c246fe65]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:14.380573 27981 tablet_service.cc:1467] Processing CreateTablet for tablet c6498d83236c48f88bd21c303c9c9ab3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=acb2583756a845cab3b0cb05c246fe65]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:14.380724 28242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6498d83236c48f88bd21c303c9c9ab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:14.380962 27981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6498d83236c48f88bd21c303c9c9ab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:14.381655 27982 tablet_service.cc:1467] Processing CreateTablet for tablet 6bc2a8940e5b42eb8152be26a9952662 (DEFAULT_TABLE table=rebalance_test_table_4 [id=acb2583756a845cab3b0cb05c246fe65]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:14.381958 28111 tablet_service.cc:1467] Processing CreateTablet for tablet c6498d83236c48f88bd21c303c9c9ab3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=acb2583756a845cab3b0cb05c246fe65]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:14.382092 27982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bc2a8940e5b42eb8152be26a9952662. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:14.382369 28111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6498d83236c48f88bd21c303c9c9ab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:14.385892 28112 tablet_service.cc:1467] Processing CreateTablet for tablet 6bc2a8940e5b42eb8152be26a9952662 (DEFAULT_TABLE table=rebalance_test_table_4 [id=acb2583756a845cab3b0cb05c246fe65]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:14.386245 28112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bc2a8940e5b42eb8152be26a9952662. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:14.386484 28372 tablet_service.cc:1467] Processing CreateTablet for tablet 6bc2a8940e5b42eb8152be26a9952662 (DEFAULT_TABLE table=rebalance_test_table_4 [id=acb2583756a845cab3b0cb05c246fe65]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:14.386852 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bc2a8940e5b42eb8152be26a9952662. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:14.386765 28243 tablet_service.cc:1467] Processing CreateTablet for tablet 48746a3d20534795b50aae78bfd660fe (DEFAULT_TABLE table=rebalance_test_table_4 [id=acb2583756a845cab3b0cb05c246fe65]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:14.387166 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48746a3d20534795b50aae78bfd660fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:14.390117 28371 tablet_service.cc:1467] Processing CreateTablet for tablet 48746a3d20534795b50aae78bfd660fe (DEFAULT_TABLE table=rebalance_test_table_4 [id=acb2583756a845cab3b0cb05c246fe65]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:14.390470 28371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48746a3d20534795b50aae78bfd660fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:14.394054 27983 tablet_service.cc:1467] Processing CreateTablet for tablet 48746a3d20534795b50aae78bfd660fe (DEFAULT_TABLE table=rebalance_test_table_4 [id=acb2583756a845cab3b0cb05c246fe65]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:14.394418 27983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48746a3d20534795b50aae78bfd660fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:14.412899 29338 tablet_bootstrap.cc:492] T c6498d83236c48f88bd21c303c9c9ab3 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:14.414364 29339 tablet_bootstrap.cc:492] T 48746a3d20534795b50aae78bfd660fe P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:14.415319 29338 tablet_bootstrap.cc:654] T c6498d83236c48f88bd21c303c9c9ab3 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:14.416497 29339 tablet_bootstrap.cc:654] T 48746a3d20534795b50aae78bfd660fe P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:14.417737 29337 tablet_bootstrap.cc:492] T c6498d83236c48f88bd21c303c9c9ab3 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:14.419111 29337 tablet_bootstrap.cc:654] T c6498d83236c48f88bd21c303c9c9ab3 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:14.420682 29339 tablet_bootstrap.cc:492] T 48746a3d20534795b50aae78bfd660fe P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:14.420810 29339 ts_tablet_manager.cc:1367] T 48746a3d20534795b50aae78bfd660fe P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 08:12:14.420979 29336 tablet_bootstrap.cc:492] T c6498d83236c48f88bd21c303c9c9ab3 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:14.421308 29338 tablet_bootstrap.cc:492] T c6498d83236c48f88bd21c303c9c9ab3 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:14.421525 29338 ts_tablet_manager.cc:1367] T c6498d83236c48f88bd21c303c9c9ab3 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20240503 08:12:14.421550 29337 tablet_bootstrap.cc:492] T c6498d83236c48f88bd21c303c9c9ab3 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:14.421598 29339 raft_consensus.cc:348] T 48746a3d20534795b50aae78bfd660fe P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:14.421730 29337 ts_tablet_manager.cc:1367] T c6498d83236c48f88bd21c303c9c9ab3 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 08:12:14.421917 29339 raft_consensus.cc:374] T 48746a3d20534795b50aae78bfd660fe P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:14.422051 29339 raft_consensus.cc:729] T 48746a3d20534795b50aae78bfd660fe P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:14.422327 29338 raft_consensus.cc:348] T c6498d83236c48f88bd21c303c9c9ab3 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:14.422324 29339 consensus_queue.cc:260] T 48746a3d20534795b50aae78bfd660fe P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:14.422513 29336 tablet_bootstrap.cc:654] T c6498d83236c48f88bd21c303c9c9ab3 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:14.422574 29338 raft_consensus.cc:374] T c6498d83236c48f88bd21c303c9c9ab3 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:14.422492 29337 raft_consensus.cc:348] T c6498d83236c48f88bd21c303c9c9ab3 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:14.422657 29337 raft_consensus.cc:374] T c6498d83236c48f88bd21c303c9c9ab3 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:14.422758 29338 raft_consensus.cc:729] T c6498d83236c48f88bd21c303c9c9ab3 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:14.422782 29337 raft_consensus.cc:729] T c6498d83236c48f88bd21c303c9c9ab3 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:14.422978 29337 consensus_queue.cc:260] T c6498d83236c48f88bd21c303c9c9ab3 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:14.423506 29337 ts_tablet_manager.cc:1398] T c6498d83236c48f88bd21c303c9c9ab3 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:14.423818 29337 tablet_bootstrap.cc:492] T 6bc2a8940e5b42eb8152be26a9952662 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:14.424355 29339 ts_tablet_manager.cc:1398] T 48746a3d20534795b50aae78bfd660fe P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:12:14.424641 29339 tablet_bootstrap.cc:492] T 6bc2a8940e5b42eb8152be26a9952662 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:14.425318 29337 tablet_bootstrap.cc:654] T 6bc2a8940e5b42eb8152be26a9952662 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:14.426072 29339 tablet_bootstrap.cc:654] T 6bc2a8940e5b42eb8152be26a9952662 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:14.423058 29338 consensus_queue.cc:260] T c6498d83236c48f88bd21c303c9c9ab3 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:14.430390 29336 tablet_bootstrap.cc:492] T c6498d83236c48f88bd21c303c9c9ab3 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:14.430936 29336 ts_tablet_manager.cc:1367] T c6498d83236c48f88bd21c303c9c9ab3 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20240503 08:12:14.432152 29338 ts_tablet_manager.cc:1398] T c6498d83236c48f88bd21c303c9c9ab3 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240503 08:12:14.432904 29338 tablet_bootstrap.cc:492] T 48746a3d20534795b50aae78bfd660fe P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:14.432847 29336 raft_consensus.cc:348] T c6498d83236c48f88bd21c303c9c9ab3 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:14.433050 29336 raft_consensus.cc:374] T c6498d83236c48f88bd21c303c9c9ab3 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:14.433138 29336 raft_consensus.cc:729] T c6498d83236c48f88bd21c303c9c9ab3 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:14.433359 29336 consensus_queue.cc:260] T c6498d83236c48f88bd21c303c9c9ab3 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:14.433871 29336 ts_tablet_manager.cc:1398] T c6498d83236c48f88bd21c303c9c9ab3 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240503 08:12:14.434134 29336 tablet_bootstrap.cc:492] T 48746a3d20534795b50aae78bfd660fe P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:14.435590 29336 tablet_bootstrap.cc:654] T 48746a3d20534795b50aae78bfd660fe P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:14.438630 29338 tablet_bootstrap.cc:654] T 48746a3d20534795b50aae78bfd660fe P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:14.442162 29339 tablet_bootstrap.cc:492] T 6bc2a8940e5b42eb8152be26a9952662 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:14.442312 29339 ts_tablet_manager.cc:1367] T 6bc2a8940e5b42eb8152be26a9952662 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.004s
I20240503 08:12:14.442430 29336 tablet_bootstrap.cc:492] T 48746a3d20534795b50aae78bfd660fe P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:14.442592 29336 ts_tablet_manager.cc:1367] T 48746a3d20534795b50aae78bfd660fe P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20240503 08:12:14.442782 29337 tablet_bootstrap.cc:492] T 6bc2a8940e5b42eb8152be26a9952662 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:14.442938 29338 tablet_bootstrap.cc:492] T 48746a3d20534795b50aae78bfd660fe P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:14.442970 29337 ts_tablet_manager.cc:1367] T 6bc2a8940e5b42eb8152be26a9952662 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.010s
I20240503 08:12:14.443137 29338 ts_tablet_manager.cc:1367] T 48746a3d20534795b50aae78bfd660fe P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.005s
I20240503 08:12:14.443310 29336 raft_consensus.cc:348] T 48746a3d20534795b50aae78bfd660fe P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:14.443276 29339 raft_consensus.cc:348] T 6bc2a8940e5b42eb8152be26a9952662 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:14.443539 29336 raft_consensus.cc:374] T 48746a3d20534795b50aae78bfd660fe P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:14.443552 29339 raft_consensus.cc:374] T 6bc2a8940e5b42eb8152be26a9952662 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:14.443612 29336 raft_consensus.cc:729] T 48746a3d20534795b50aae78bfd660fe P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:14.443621 29339 raft_consensus.cc:729] T 6bc2a8940e5b42eb8152be26a9952662 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:14.443681 29337 raft_consensus.cc:348] T 6bc2a8940e5b42eb8152be26a9952662 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:14.443899 29337 raft_consensus.cc:374] T 6bc2a8940e5b42eb8152be26a9952662 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:14.443828 29336 consensus_queue.cc:260] T 48746a3d20534795b50aae78bfd660fe P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:14.443854 29338 raft_consensus.cc:348] T 48746a3d20534795b50aae78bfd660fe P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:14.443970 29337 raft_consensus.cc:729] T 6bc2a8940e5b42eb8152be26a9952662 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:14.444031 29338 raft_consensus.cc:374] T 48746a3d20534795b50aae78bfd660fe P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:14.444095 29338 raft_consensus.cc:729] T 48746a3d20534795b50aae78bfd660fe P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:14.444303 29336 ts_tablet_manager.cc:1398] T 48746a3d20534795b50aae78bfd660fe P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:14.444204 29337 consensus_queue.cc:260] T 6bc2a8940e5b42eb8152be26a9952662 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:14.444310 29338 consensus_queue.cc:260] T 48746a3d20534795b50aae78bfd660fe P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:14.444705 29337 ts_tablet_manager.cc:1398] T 6bc2a8940e5b42eb8152be26a9952662 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:12:14.444756 29338 ts_tablet_manager.cc:1398] T 48746a3d20534795b50aae78bfd660fe P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:12:14.445142 29336 tablet_bootstrap.cc:492] T 6bc2a8940e5b42eb8152be26a9952662 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:14.446385 29336 tablet_bootstrap.cc:654] T 6bc2a8940e5b42eb8152be26a9952662 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:14.443924 29339 consensus_queue.cc:260] T 6bc2a8940e5b42eb8152be26a9952662 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:14.447990 29336 tablet_bootstrap.cc:492] T 6bc2a8940e5b42eb8152be26a9952662 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:14.448037 29339 ts_tablet_manager.cc:1398] T 6bc2a8940e5b42eb8152be26a9952662 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 08:12:14.448112 29336 ts_tablet_manager.cc:1367] T 6bc2a8940e5b42eb8152be26a9952662 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:12:14.448781 29336 raft_consensus.cc:348] T 6bc2a8940e5b42eb8152be26a9952662 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:14.448988 29336 raft_consensus.cc:374] T 6bc2a8940e5b42eb8152be26a9952662 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:14.449102 29336 raft_consensus.cc:729] T 6bc2a8940e5b42eb8152be26a9952662 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:14.449347 29336 consensus_queue.cc:260] T 6bc2a8940e5b42eb8152be26a9952662 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:14.449903 29336 ts_tablet_manager.cc:1398] T 6bc2a8940e5b42eb8152be26a9952662 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:14.576836 29331 raft_consensus.cc:482] T c6498d83236c48f88bd21c303c9c9ab3 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:14.577034 29331 raft_consensus.cc:504] T c6498d83236c48f88bd21c303c9c9ab3 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:14.577684 29331 leader_election.cc:288] T c6498d83236c48f88bd21c303c9c9ab3 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:14.578055 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6498d83236c48f88bd21c303c9c9ab3" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:12:14.578171 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6498d83236c48f88bd21c303c9c9ab3" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:12:14.578264 28003 raft_consensus.cc:2454] T c6498d83236c48f88bd21c303c9c9ab3 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:12:14.578438 28132 raft_consensus.cc:2454] T c6498d83236c48f88bd21c303c9c9ab3 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:12:14.578651 28199 leader_election.cc:302] T c6498d83236c48f88bd21c303c9c9ab3 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:14.578894 29331 raft_consensus.cc:2781] T c6498d83236c48f88bd21c303c9c9ab3 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:14.578972 29331 raft_consensus.cc:482] T c6498d83236c48f88bd21c303c9c9ab3 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:14.579031 29331 raft_consensus.cc:3037] T c6498d83236c48f88bd21c303c9c9ab3 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:14.580832 29331 raft_consensus.cc:504] T c6498d83236c48f88bd21c303c9c9ab3 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:14.581326 29331 leader_election.cc:288] T c6498d83236c48f88bd21c303c9c9ab3 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Requested vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:14.581708 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6498d83236c48f88bd21c303c9c9ab3" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:12:14.581866 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6498d83236c48f88bd21c303c9c9ab3" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:12:14.581924 28003 raft_consensus.cc:3037] T c6498d83236c48f88bd21c303c9c9ab3 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:14.582051 28132 raft_consensus.cc:3037] T c6498d83236c48f88bd21c303c9c9ab3 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:14.583693 28003 raft_consensus.cc:2454] T c6498d83236c48f88bd21c303c9c9ab3 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:14.583822 28132 raft_consensus.cc:2454] T c6498d83236c48f88bd21c303c9c9ab3 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:14.584033 28199 leader_election.cc:302] T c6498d83236c48f88bd21c303c9c9ab3 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:14.584297 29331 raft_consensus.cc:2781] T c6498d83236c48f88bd21c303c9c9ab3 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:14.584635 29331 raft_consensus.cc:686] T c6498d83236c48f88bd21c303c9c9ab3 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Becoming Leader. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Running, Role: LEADER
I20240503 08:12:14.584867 29331 consensus_queue.cc:237] T c6498d83236c48f88bd21c303c9c9ab3 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:14.587432 27858 catalog_manager.cc:5484] T c6498d83236c48f88bd21c303c9c9ab3 P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: term changed from 0 to 1, leader changed from <none> to aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67). New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:12:14.629685 29072 raft_consensus.cc:482] T 48746a3d20534795b50aae78bfd660fe P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:14.629907 29072 raft_consensus.cc:504] T 48746a3d20534795b50aae78bfd660fe P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:14.630510 29072 leader_election.cc:288] T 48746a3d20534795b50aae78bfd660fe P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:14.630947 28263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48746a3d20534795b50aae78bfd660fe" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:12:14.630950 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48746a3d20534795b50aae78bfd660fe" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:14.631124 28263 raft_consensus.cc:2454] T 48746a3d20534795b50aae78bfd660fe P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:14.631181 28391 raft_consensus.cc:2454] T 48746a3d20534795b50aae78bfd660fe P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:14.631655 27937 leader_election.cc:302] T 48746a3d20534795b50aae78bfd660fe P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, aae860d7ae2b4002bf7794c278f9cc56, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:14.631924 29072 raft_consensus.cc:2781] T 48746a3d20534795b50aae78bfd660fe P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:14.632005 29072 raft_consensus.cc:482] T 48746a3d20534795b50aae78bfd660fe P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:14.632097 29072 raft_consensus.cc:3037] T 48746a3d20534795b50aae78bfd660fe P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:14.634104 29072 raft_consensus.cc:504] T 48746a3d20534795b50aae78bfd660fe P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:14.634630 29072 leader_election.cc:288] T 48746a3d20534795b50aae78bfd660fe P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:14.634967 28263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48746a3d20534795b50aae78bfd660fe" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:12:14.635049 29060 raft_consensus.cc:482] T 6bc2a8940e5b42eb8152be26a9952662 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:14.635042 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48746a3d20534795b50aae78bfd660fe" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:12:14.635125 28263 raft_consensus.cc:3037] T 48746a3d20534795b50aae78bfd660fe P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:14.635232 28391 raft_consensus.cc:3037] T 48746a3d20534795b50aae78bfd660fe P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:14.635181 29060 raft_consensus.cc:504] T 6bc2a8940e5b42eb8152be26a9952662 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:14.635818 29060 leader_election.cc:288] T 6bc2a8940e5b42eb8152be26a9952662 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:14.636288 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bc2a8940e5b42eb8152be26a9952662" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:12:14.636490 28003 raft_consensus.cc:2454] T 6bc2a8940e5b42eb8152be26a9952662 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:14.636582 28393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bc2a8940e5b42eb8152be26a9952662" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:14.636797 28393 raft_consensus.cc:2454] T 6bc2a8940e5b42eb8152be26a9952662 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:14.636857 28069 leader_election.cc:302] T 6bc2a8940e5b42eb8152be26a9952662 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 833582ff68c94269be364ebd187a75db; no voters:
I20240503 08:12:14.637151 29060 raft_consensus.cc:2781] T 6bc2a8940e5b42eb8152be26a9952662 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:14.637251 29060 raft_consensus.cc:482] T 6bc2a8940e5b42eb8152be26a9952662 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:14.637332 29060 raft_consensus.cc:3037] T 6bc2a8940e5b42eb8152be26a9952662 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:14.637332 28263 raft_consensus.cc:2454] T 48746a3d20534795b50aae78bfd660fe P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:14.637650 27937 leader_election.cc:302] T 48746a3d20534795b50aae78bfd660fe P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:14.637863 29072 raft_consensus.cc:2781] T 48746a3d20534795b50aae78bfd660fe P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:14.638208 29072 raft_consensus.cc:686] T 48746a3d20534795b50aae78bfd660fe P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Becoming Leader. State: Replica: 833582ff68c94269be364ebd187a75db, State: Running, Role: LEADER
I20240503 08:12:14.638360 28391 raft_consensus.cc:2454] T 48746a3d20534795b50aae78bfd660fe P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:14.638446 29072 consensus_queue.cc:237] T 48746a3d20534795b50aae78bfd660fe P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:14.639647 29060 raft_consensus.cc:504] T 6bc2a8940e5b42eb8152be26a9952662 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:14.640154 29060 leader_election.cc:288] T 6bc2a8940e5b42eb8152be26a9952662 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Requested vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:14.640616 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bc2a8940e5b42eb8152be26a9952662" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:12:14.640611 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bc2a8940e5b42eb8152be26a9952662" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:12:14.640748 28003 raft_consensus.cc:3037] T 6bc2a8940e5b42eb8152be26a9952662 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:14.640756 28391 raft_consensus.cc:3037] T 6bc2a8940e5b42eb8152be26a9952662 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:14.642190 27858 catalog_manager.cc:5484] T 48746a3d20534795b50aae78bfd660fe P 833582ff68c94269be364ebd187a75db reported cstate change: term changed from 0 to 1, leader changed from <none> to 833582ff68c94269be364ebd187a75db (127.20.230.65). New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:12:14.642612 28003 raft_consensus.cc:2454] T 6bc2a8940e5b42eb8152be26a9952662 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:12:14.642940 28069 leader_election.cc:302] T 6bc2a8940e5b42eb8152be26a9952662 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 833582ff68c94269be364ebd187a75db; no voters:
I20240503 08:12:14.642990 28391 raft_consensus.cc:2454] T 6bc2a8940e5b42eb8152be26a9952662 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:12:14.643221 29060 raft_consensus.cc:2781] T 6bc2a8940e5b42eb8152be26a9952662 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:14.643574 29060 raft_consensus.cc:686] T 6bc2a8940e5b42eb8152be26a9952662 P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Becoming Leader. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Running, Role: LEADER
I20240503 08:12:14.643857 29060 consensus_queue.cc:237] T 6bc2a8940e5b42eb8152be26a9952662 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:14.647053 27858 catalog_manager.cc:5484] T 6bc2a8940e5b42eb8152be26a9952662 P 1b411c6bb25549368041773d7b710d48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b411c6bb25549368041773d7b710d48 (127.20.230.66). New cstate: current_term: 1 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
W20240503 08:12:15.002538 29355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:15.002854 29355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:15.010504 29355 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 08:12:15.044453 29060 consensus_queue.cc:1035] T 6bc2a8940e5b42eb8152be26a9952662 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:15.055500 29060 consensus_queue.cc:1035] T 6bc2a8940e5b42eb8152be26a9952662 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:15.098079 29331 consensus_queue.cc:1035] T c6498d83236c48f88bd21c303c9c9ab3 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:15.113948 29384 consensus_queue.cc:1035] T c6498d83236c48f88bd21c303c9c9ab3 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:15.163762 28263 raft_consensus.cc:1261] T 48746a3d20534795b50aae78bfd660fe P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:15.164451 29072 consensus_queue.cc:1035] T 48746a3d20534795b50aae78bfd660fe P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:15.172011 28391 raft_consensus.cc:1261] T 48746a3d20534795b50aae78bfd660fe P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:15.182704 29072 consensus_queue.cc:1035] T 48746a3d20534795b50aae78bfd660fe P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:15.243191 29404 mvcc.cc:205] Tried to move back new op lower bound from 7023509238423396352 to 7023509236281065472. Current Snapshot: MvccSnapshot[applied={T|T < 7023509238423396352}]
INSERT report
rows total: 2000
time total: 217.258 ms
time per row: 0.108629 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 08:12:16.236083 27856 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:16.244822 28242 tablet_service.cc:1467] Processing CreateTablet for tablet bab5c25c31a64b578489b1006f631cc0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=490e37ef740f4e96b915339369c454ff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:16.244971 28111 tablet_service.cc:1467] Processing CreateTablet for tablet bab5c25c31a64b578489b1006f631cc0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=490e37ef740f4e96b915339369c454ff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:16.245296 28242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab5c25c31a64b578489b1006f631cc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:16.245329 28111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab5c25c31a64b578489b1006f631cc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:16.245513 28243 tablet_service.cc:1467] Processing CreateTablet for tablet ba1fc93781714efb8eb7ec67c2fceb8a (DEFAULT_TABLE table=rebalance_test_table_5 [id=490e37ef740f4e96b915339369c454ff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:16.246227 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba1fc93781714efb8eb7ec67c2fceb8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:16.247094 28371 tablet_service.cc:1467] Processing CreateTablet for tablet bab5c25c31a64b578489b1006f631cc0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=490e37ef740f4e96b915339369c454ff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:16.247517 28371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab5c25c31a64b578489b1006f631cc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:16.253500 28372 tablet_service.cc:1467] Processing CreateTablet for tablet ba1fc93781714efb8eb7ec67c2fceb8a (DEFAULT_TABLE table=rebalance_test_table_5 [id=490e37ef740f4e96b915339369c454ff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:16.253809 29429 tablet_bootstrap.cc:492] T bab5c25c31a64b578489b1006f631cc0 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:16.253892 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba1fc93781714efb8eb7ec67c2fceb8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:16.253891 29430 tablet_bootstrap.cc:492] T bab5c25c31a64b578489b1006f631cc0 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:16.255230 29430 tablet_bootstrap.cc:654] T bab5c25c31a64b578489b1006f631cc0 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:16.255319 29429 tablet_bootstrap.cc:654] T bab5c25c31a64b578489b1006f631cc0 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:16.257105 29429 tablet_bootstrap.cc:492] T bab5c25c31a64b578489b1006f631cc0 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:16.257252 29429 ts_tablet_manager.cc:1367] T bab5c25c31a64b578489b1006f631cc0 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 08:12:16.257423 29430 tablet_bootstrap.cc:492] T bab5c25c31a64b578489b1006f631cc0 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:16.257553 29430 ts_tablet_manager.cc:1367] T bab5c25c31a64b578489b1006f631cc0 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:12:16.257982 29429 raft_consensus.cc:348] T bab5c25c31a64b578489b1006f631cc0 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:16.258607 29430 raft_consensus.cc:348] T bab5c25c31a64b578489b1006f631cc0 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:16.258625 27981 tablet_service.cc:1467] Processing CreateTablet for tablet 099e37089b2e42b4839e681f8f37365d (DEFAULT_TABLE table=rebalance_test_table_5 [id=490e37ef740f4e96b915339369c454ff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:16.258865 29430 raft_consensus.cc:374] T bab5c25c31a64b578489b1006f631cc0 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:16.259018 29430 raft_consensus.cc:729] T bab5c25c31a64b578489b1006f631cc0 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:16.259038 27981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 099e37089b2e42b4839e681f8f37365d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:16.259261 28111 tablet_service.cc:1467] Processing CreateTablet for tablet ba1fc93781714efb8eb7ec67c2fceb8a (DEFAULT_TABLE table=rebalance_test_table_5 [id=490e37ef740f4e96b915339369c454ff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:16.259351 29430 consensus_queue.cc:260] T bab5c25c31a64b578489b1006f631cc0 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:16.259629 28111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba1fc93781714efb8eb7ec67c2fceb8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:16.259893 29430 ts_tablet_manager.cc:1398] T bab5c25c31a64b578489b1006f631cc0 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:16.262089 28112 tablet_service.cc:1467] Processing CreateTablet for tablet 099e37089b2e42b4839e681f8f37365d (DEFAULT_TABLE table=rebalance_test_table_5 [id=490e37ef740f4e96b915339369c454ff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:16.262449 28112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 099e37089b2e42b4839e681f8f37365d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:16.264550 29434 tablet_bootstrap.cc:492] T 099e37089b2e42b4839e681f8f37365d P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:16.262645 28242 tablet_service.cc:1467] Processing CreateTablet for tablet 099e37089b2e42b4839e681f8f37365d (DEFAULT_TABLE table=rebalance_test_table_5 [id=490e37ef740f4e96b915339369c454ff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:16.264902 28242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 099e37089b2e42b4839e681f8f37365d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:16.265568 29431 tablet_bootstrap.cc:492] T bab5c25c31a64b578489b1006f631cc0 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:16.266283 29434 tablet_bootstrap.cc:654] T 099e37089b2e42b4839e681f8f37365d P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:16.267709 29431 tablet_bootstrap.cc:654] T bab5c25c31a64b578489b1006f631cc0 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:16.258266 29429 raft_consensus.cc:374] T bab5c25c31a64b578489b1006f631cc0 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:16.270895 29429 raft_consensus.cc:729] T bab5c25c31a64b578489b1006f631cc0 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:16.271183 29429 consensus_queue.cc:260] T bab5c25c31a64b578489b1006f631cc0 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:16.272527 29429 ts_tablet_manager.cc:1398] T bab5c25c31a64b578489b1006f631cc0 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240503 08:12:16.272850 29429 tablet_bootstrap.cc:492] T ba1fc93781714efb8eb7ec67c2fceb8a P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:16.275221 29429 tablet_bootstrap.cc:654] T ba1fc93781714efb8eb7ec67c2fceb8a P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:16.275610 29431 tablet_bootstrap.cc:492] T bab5c25c31a64b578489b1006f631cc0 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:16.275805 29431 ts_tablet_manager.cc:1367] T bab5c25c31a64b578489b1006f631cc0 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20240503 08:12:16.276551 29431 raft_consensus.cc:348] T bab5c25c31a64b578489b1006f631cc0 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:16.276798 29431 raft_consensus.cc:374] T bab5c25c31a64b578489b1006f631cc0 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:16.276912 29431 raft_consensus.cc:729] T bab5c25c31a64b578489b1006f631cc0 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:16.277451 29430 tablet_bootstrap.cc:492] T 099e37089b2e42b4839e681f8f37365d P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:16.278136 29434 tablet_bootstrap.cc:492] T 099e37089b2e42b4839e681f8f37365d P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:16.278280 29434 ts_tablet_manager.cc:1367] T 099e37089b2e42b4839e681f8f37365d P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.000s
I20240503 08:12:16.278838 29430 tablet_bootstrap.cc:654] T 099e37089b2e42b4839e681f8f37365d P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:16.279011 29434 raft_consensus.cc:348] T 099e37089b2e42b4839e681f8f37365d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:16.279227 29434 raft_consensus.cc:374] T 099e37089b2e42b4839e681f8f37365d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:16.279318 29434 raft_consensus.cc:729] T 099e37089b2e42b4839e681f8f37365d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:16.279548 29434 consensus_queue.cc:260] T 099e37089b2e42b4839e681f8f37365d P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:16.280316 29434 ts_tablet_manager.cc:1398] T 099e37089b2e42b4839e681f8f37365d P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:16.277185 29431 consensus_queue.cc:260] T bab5c25c31a64b578489b1006f631cc0 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:16.281384 29431 ts_tablet_manager.cc:1398] T bab5c25c31a64b578489b1006f631cc0 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240503 08:12:16.281656 29431 tablet_bootstrap.cc:492] T ba1fc93781714efb8eb7ec67c2fceb8a P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:16.281888 29430 tablet_bootstrap.cc:492] T 099e37089b2e42b4839e681f8f37365d P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:16.282027 29430 ts_tablet_manager.cc:1367] T 099e37089b2e42b4839e681f8f37365d P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 08:12:16.282675 29430 raft_consensus.cc:348] T 099e37089b2e42b4839e681f8f37365d P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:16.282918 29430 raft_consensus.cc:374] T 099e37089b2e42b4839e681f8f37365d P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:16.283010 29430 raft_consensus.cc:729] T 099e37089b2e42b4839e681f8f37365d P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:16.283135 29431 tablet_bootstrap.cc:654] T ba1fc93781714efb8eb7ec67c2fceb8a P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:16.283320 29430 consensus_queue.cc:260] T 099e37089b2e42b4839e681f8f37365d P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:16.284598 29430 ts_tablet_manager.cc:1398] T 099e37089b2e42b4839e681f8f37365d P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:16.284888 29430 tablet_bootstrap.cc:492] T ba1fc93781714efb8eb7ec67c2fceb8a P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:16.286317 29429 tablet_bootstrap.cc:492] T ba1fc93781714efb8eb7ec67c2fceb8a P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:16.286556 29429 ts_tablet_manager.cc:1367] T ba1fc93781714efb8eb7ec67c2fceb8a P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.002s
I20240503 08:12:16.286656 29430 tablet_bootstrap.cc:654] T ba1fc93781714efb8eb7ec67c2fceb8a P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:16.287398 29429 raft_consensus.cc:348] T ba1fc93781714efb8eb7ec67c2fceb8a P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:16.287730 29429 raft_consensus.cc:374] T ba1fc93781714efb8eb7ec67c2fceb8a P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:16.287849 29429 raft_consensus.cc:729] T ba1fc93781714efb8eb7ec67c2fceb8a P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:16.288178 29429 consensus_queue.cc:260] T ba1fc93781714efb8eb7ec67c2fceb8a P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:16.290818 29429 ts_tablet_manager.cc:1398] T ba1fc93781714efb8eb7ec67c2fceb8a P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 08:12:16.291133 29429 tablet_bootstrap.cc:492] T 099e37089b2e42b4839e681f8f37365d P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:16.291751 29431 tablet_bootstrap.cc:492] T ba1fc93781714efb8eb7ec67c2fceb8a P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:16.291950 29431 ts_tablet_manager.cc:1367] T ba1fc93781714efb8eb7ec67c2fceb8a P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20240503 08:12:16.292143 29430 tablet_bootstrap.cc:492] T ba1fc93781714efb8eb7ec67c2fceb8a P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:16.292306 29430 ts_tablet_manager.cc:1367] T ba1fc93781714efb8eb7ec67c2fceb8a P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240503 08:12:16.292479 29429 tablet_bootstrap.cc:654] T 099e37089b2e42b4839e681f8f37365d P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:16.292723 29431 raft_consensus.cc:348] T ba1fc93781714efb8eb7ec67c2fceb8a P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:16.292973 29431 raft_consensus.cc:374] T ba1fc93781714efb8eb7ec67c2fceb8a P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:16.293084 29431 raft_consensus.cc:729] T ba1fc93781714efb8eb7ec67c2fceb8a P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:16.293026 29430 raft_consensus.cc:348] T ba1fc93781714efb8eb7ec67c2fceb8a P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:16.293222 29430 raft_consensus.cc:374] T ba1fc93781714efb8eb7ec67c2fceb8a P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:16.293341 29430 raft_consensus.cc:729] T ba1fc93781714efb8eb7ec67c2fceb8a P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:16.293349 29431 consensus_queue.cc:260] T ba1fc93781714efb8eb7ec67c2fceb8a P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:16.293573 29430 consensus_queue.cc:260] T ba1fc93781714efb8eb7ec67c2fceb8a P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:16.293958 29431 ts_tablet_manager.cc:1398] T ba1fc93781714efb8eb7ec67c2fceb8a P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:16.295157 29430 ts_tablet_manager.cc:1398] T ba1fc93781714efb8eb7ec67c2fceb8a P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:12:16.299552 29429 tablet_bootstrap.cc:492] T 099e37089b2e42b4839e681f8f37365d P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:16.299671 29429 ts_tablet_manager.cc:1367] T 099e37089b2e42b4839e681f8f37365d P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20240503 08:12:16.300225 29429 raft_consensus.cc:348] T 099e37089b2e42b4839e681f8f37365d P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:16.300365 29429 raft_consensus.cc:374] T 099e37089b2e42b4839e681f8f37365d P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:16.300447 29429 raft_consensus.cc:729] T 099e37089b2e42b4839e681f8f37365d P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:16.300631 29429 consensus_queue.cc:260] T 099e37089b2e42b4839e681f8f37365d P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:16.301057 29429 ts_tablet_manager.cc:1398] T 099e37089b2e42b4839e681f8f37365d P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:12:16.330086 29400 raft_consensus.cc:482] T bab5c25c31a64b578489b1006f631cc0 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:16.330296 29400 raft_consensus.cc:504] T bab5c25c31a64b578489b1006f631cc0 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:16.330994 29400 leader_election.cc:288] T bab5c25c31a64b578489b1006f631cc0 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:16.331439 28263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab5c25c31a64b578489b1006f631cc0" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:12:16.331519 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab5c25c31a64b578489b1006f631cc0" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:16.331658 28263 raft_consensus.cc:2454] T bab5c25c31a64b578489b1006f631cc0 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:16.331746 28391 raft_consensus.cc:2454] T bab5c25c31a64b578489b1006f631cc0 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:16.332068 28067 leader_election.cc:302] T bab5c25c31a64b578489b1006f631cc0 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:16.332368 29400 raft_consensus.cc:2781] T bab5c25c31a64b578489b1006f631cc0 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:16.332479 29400 raft_consensus.cc:482] T bab5c25c31a64b578489b1006f631cc0 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:16.332562 29400 raft_consensus.cc:3037] T bab5c25c31a64b578489b1006f631cc0 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:16.334548 29400 raft_consensus.cc:504] T bab5c25c31a64b578489b1006f631cc0 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:16.335000 29400 leader_election.cc:288] T bab5c25c31a64b578489b1006f631cc0 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:16.335330 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab5c25c31a64b578489b1006f631cc0" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:12:16.335510 28262 raft_consensus.cc:3037] T bab5c25c31a64b578489b1006f631cc0 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:16.335485 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab5c25c31a64b578489b1006f631cc0" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:12:16.335638 28391 raft_consensus.cc:3037] T bab5c25c31a64b578489b1006f631cc0 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:16.337281 28262 raft_consensus.cc:2454] T bab5c25c31a64b578489b1006f631cc0 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:12:16.337416 28391 raft_consensus.cc:2454] T bab5c25c31a64b578489b1006f631cc0 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:12:16.337610 28067 leader_election.cc:302] T bab5c25c31a64b578489b1006f631cc0 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:16.337929 29400 raft_consensus.cc:2781] T bab5c25c31a64b578489b1006f631cc0 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:16.338245 29400 raft_consensus.cc:686] T bab5c25c31a64b578489b1006f631cc0 P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Becoming Leader. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Running, Role: LEADER
I20240503 08:12:16.338483 29400 consensus_queue.cc:237] T bab5c25c31a64b578489b1006f631cc0 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:16.340906 27861 catalog_manager.cc:5484] T bab5c25c31a64b578489b1006f631cc0 P 1b411c6bb25549368041773d7b710d48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b411c6bb25549368041773d7b710d48 (127.20.230.66). New cstate: current_term: 1 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:12:16.364085 29269 raft_consensus.cc:482] T ba1fc93781714efb8eb7ec67c2fceb8a P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:16.364246 29269 raft_consensus.cc:504] T ba1fc93781714efb8eb7ec67c2fceb8a P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:16.364892 29269 leader_election.cc:288] T ba1fc93781714efb8eb7ec67c2fceb8a P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:16.365199 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1fc93781714efb8eb7ec67c2fceb8a" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:12:16.365391 28262 raft_consensus.cc:2454] T ba1fc93781714efb8eb7ec67c2fceb8a P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:12:16.365376 28131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1fc93781714efb8eb7ec67c2fceb8a" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:12:16.365584 28131 raft_consensus.cc:2454] T ba1fc93781714efb8eb7ec67c2fceb8a P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:12:16.365782 28327 leader_election.cc:302] T ba1fc93781714efb8eb7ec67c2fceb8a P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:16.366047 29269 raft_consensus.cc:2781] T ba1fc93781714efb8eb7ec67c2fceb8a P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:16.366135 29269 raft_consensus.cc:482] T ba1fc93781714efb8eb7ec67c2fceb8a P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:16.366195 29269 raft_consensus.cc:3037] T ba1fc93781714efb8eb7ec67c2fceb8a P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:16.367921 29269 raft_consensus.cc:504] T ba1fc93781714efb8eb7ec67c2fceb8a P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:16.368425 29269 leader_election.cc:288] T ba1fc93781714efb8eb7ec67c2fceb8a P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:16.368783 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1fc93781714efb8eb7ec67c2fceb8a" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:12:16.368945 28262 raft_consensus.cc:3037] T ba1fc93781714efb8eb7ec67c2fceb8a P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:16.368908 28131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1fc93781714efb8eb7ec67c2fceb8a" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:12:16.369069 28131 raft_consensus.cc:3037] T ba1fc93781714efb8eb7ec67c2fceb8a P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:16.370687 28262 raft_consensus.cc:2454] T ba1fc93781714efb8eb7ec67c2fceb8a P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:12:16.371022 28327 leader_election.cc:302] T ba1fc93781714efb8eb7ec67c2fceb8a P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:16.371304 29269 raft_consensus.cc:2781] T ba1fc93781714efb8eb7ec67c2fceb8a P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:16.371608 29269 raft_consensus.cc:686] T ba1fc93781714efb8eb7ec67c2fceb8a P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Becoming Leader. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Running, Role: LEADER
I20240503 08:12:16.371850 29269 consensus_queue.cc:237] T ba1fc93781714efb8eb7ec67c2fceb8a P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:16.374156 27861 catalog_manager.cc:5484] T ba1fc93781714efb8eb7ec67c2fceb8a P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68). New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:12:16.379462 29400 raft_consensus.cc:482] T 099e37089b2e42b4839e681f8f37365d P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:16.379554 28131 raft_consensus.cc:2454] T ba1fc93781714efb8eb7ec67c2fceb8a P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:12:16.379612 29400 raft_consensus.cc:504] T 099e37089b2e42b4839e681f8f37365d P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:16.380102 29400 leader_election.cc:288] T 099e37089b2e42b4839e681f8f37365d P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:12:16.380496 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099e37089b2e42b4839e681f8f37365d" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:12:16.380506 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099e37089b2e42b4839e681f8f37365d" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:12:16.380699 28003 raft_consensus.cc:2454] T 099e37089b2e42b4839e681f8f37365d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:16.381076 28069 leader_election.cc:302] T 099e37089b2e42b4839e681f8f37365d P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 833582ff68c94269be364ebd187a75db; no voters:
I20240503 08:12:16.381312 29400 raft_consensus.cc:2781] T 099e37089b2e42b4839e681f8f37365d P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:16.381409 29400 raft_consensus.cc:482] T 099e37089b2e42b4839e681f8f37365d P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:16.381431 28262 raft_consensus.cc:2454] T 099e37089b2e42b4839e681f8f37365d P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:16.381518 29400 raft_consensus.cc:3037] T 099e37089b2e42b4839e681f8f37365d P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:16.383332 29400 raft_consensus.cc:504] T 099e37089b2e42b4839e681f8f37365d P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:16.383809 29400 leader_election.cc:288] T 099e37089b2e42b4839e681f8f37365d P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Requested vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:12:16.384177 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099e37089b2e42b4839e681f8f37365d" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:12:16.384208 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099e37089b2e42b4839e681f8f37365d" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:12:16.384346 28262 raft_consensus.cc:3037] T 099e37089b2e42b4839e681f8f37365d P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:16.384393 28003 raft_consensus.cc:3037] T 099e37089b2e42b4839e681f8f37365d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:16.386088 28262 raft_consensus.cc:2454] T 099e37089b2e42b4839e681f8f37365d P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:12:16.386240 28003 raft_consensus.cc:2454] T 099e37089b2e42b4839e681f8f37365d P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:12:16.386423 28067 leader_election.cc:302] T 099e37089b2e42b4839e681f8f37365d P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:16.386673 29400 raft_consensus.cc:2781] T 099e37089b2e42b4839e681f8f37365d P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:16.386821 29400 raft_consensus.cc:686] T 099e37089b2e42b4839e681f8f37365d P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Becoming Leader. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Running, Role: LEADER
I20240503 08:12:16.387050 29400 consensus_queue.cc:237] T 099e37089b2e42b4839e681f8f37365d P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:16.389600 27861 catalog_manager.cc:5484] T 099e37089b2e42b4839e681f8f37365d P 1b411c6bb25549368041773d7b710d48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b411c6bb25549368041773d7b710d48 (127.20.230.66). New cstate: current_term: 1 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
W20240503 08:12:16.731933 29448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:16.732225 29448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:16.739357 29448 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240503 08:12:16.775983 29400 consensus_queue.cc:1035] T 099e37089b2e42b4839e681f8f37365d P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:16.789001 29400 consensus_queue.cc:1035] T 099e37089b2e42b4839e681f8f37365d P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:16.859934 29443 consensus_queue.cc:1035] T bab5c25c31a64b578489b1006f631cc0 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:16.870807 28391 raft_consensus.cc:1261] T bab5c25c31a64b578489b1006f631cc0 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:16.872377 29443 consensus_queue.cc:1035] T bab5c25c31a64b578489b1006f631cc0 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:16.874048 28263 raft_consensus.cc:1261] T ba1fc93781714efb8eb7ec67c2fceb8a P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:16.874480 28131 raft_consensus.cc:1261] T ba1fc93781714efb8eb7ec67c2fceb8a P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:16.874941 29269 consensus_queue.cc:1035] T ba1fc93781714efb8eb7ec67c2fceb8a P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:16.875476 29269 consensus_queue.cc:1035] T ba1fc93781714efb8eb7ec67c2fceb8a P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 301.166 ms
time per row: 0.150583 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 08:12:18.092317 27862 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:18.101119 28111 tablet_service.cc:1467] Processing CreateTablet for tablet 6f19592b330546b2b4935646ff7b0304 (DEFAULT_TABLE table=rebalance_test_table_6 [id=5cde77bbf6c84845aaa35fc722360975]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:18.101285 28371 tablet_service.cc:1467] Processing CreateTablet for tablet 6f19592b330546b2b4935646ff7b0304 (DEFAULT_TABLE table=rebalance_test_table_6 [id=5cde77bbf6c84845aaa35fc722360975]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:18.101598 28111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f19592b330546b2b4935646ff7b0304. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:18.101642 28371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f19592b330546b2b4935646ff7b0304. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:18.102270 28242 tablet_service.cc:1467] Processing CreateTablet for tablet c19f22d2bc0d47e3a8217b2c97760348 (DEFAULT_TABLE table=rebalance_test_table_6 [id=5cde77bbf6c84845aaa35fc722360975]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:18.102679 28242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c19f22d2bc0d47e3a8217b2c97760348. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:18.106544 28112 tablet_service.cc:1467] Processing CreateTablet for tablet 1856d5641d3f48698f5401a1d157369d (DEFAULT_TABLE table=rebalance_test_table_6 [id=5cde77bbf6c84845aaa35fc722360975]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:18.106655 27981 tablet_service.cc:1467] Processing CreateTablet for tablet 6f19592b330546b2b4935646ff7b0304 (DEFAULT_TABLE table=rebalance_test_table_6 [id=5cde77bbf6c84845aaa35fc722360975]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:18.106907 28112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1856d5641d3f48698f5401a1d157369d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:18.106940 27983 tablet_service.cc:1467] Processing CreateTablet for tablet 1856d5641d3f48698f5401a1d157369d (DEFAULT_TABLE table=rebalance_test_table_6 [id=5cde77bbf6c84845aaa35fc722360975]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:18.106670 27982 tablet_service.cc:1467] Processing CreateTablet for tablet c19f22d2bc0d47e3a8217b2c97760348 (DEFAULT_TABLE table=rebalance_test_table_6 [id=5cde77bbf6c84845aaa35fc722360975]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:18.107149 27981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f19592b330546b2b4935646ff7b0304. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:18.107797 27982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c19f22d2bc0d47e3a8217b2c97760348. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:18.107770 28372 tablet_service.cc:1467] Processing CreateTablet for tablet c19f22d2bc0d47e3a8217b2c97760348 (DEFAULT_TABLE table=rebalance_test_table_6 [id=5cde77bbf6c84845aaa35fc722360975]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:18.108152 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c19f22d2bc0d47e3a8217b2c97760348. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:18.108376 27983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1856d5641d3f48698f5401a1d157369d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:18.112929 28373 tablet_service.cc:1467] Processing CreateTablet for tablet 1856d5641d3f48698f5401a1d157369d (DEFAULT_TABLE table=rebalance_test_table_6 [id=5cde77bbf6c84845aaa35fc722360975]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:18.113315 28373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1856d5641d3f48698f5401a1d157369d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:18.120584 29514 tablet_bootstrap.cc:492] T c19f22d2bc0d47e3a8217b2c97760348 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:18.122195 29513 tablet_bootstrap.cc:492] T 6f19592b330546b2b4935646ff7b0304 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:18.122722 29514 tablet_bootstrap.cc:654] T c19f22d2bc0d47e3a8217b2c97760348 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:18.124464 29513 tablet_bootstrap.cc:654] T 6f19592b330546b2b4935646ff7b0304 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:18.129403 29516 tablet_bootstrap.cc:492] T c19f22d2bc0d47e3a8217b2c97760348 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:18.132979 29516 tablet_bootstrap.cc:654] T c19f22d2bc0d47e3a8217b2c97760348 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:18.134790 29512 tablet_bootstrap.cc:492] T 6f19592b330546b2b4935646ff7b0304 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:18.136799 29512 tablet_bootstrap.cc:654] T 6f19592b330546b2b4935646ff7b0304 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:18.137277 29514 tablet_bootstrap.cc:492] T c19f22d2bc0d47e3a8217b2c97760348 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:18.137439 29514 ts_tablet_manager.cc:1367] T c19f22d2bc0d47e3a8217b2c97760348 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.003s
I20240503 08:12:18.137482 29513 tablet_bootstrap.cc:492] T 6f19592b330546b2b4935646ff7b0304 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:18.137665 29513 ts_tablet_manager.cc:1367] T 6f19592b330546b2b4935646ff7b0304 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20240503 08:12:18.138110 29514 raft_consensus.cc:348] T c19f22d2bc0d47e3a8217b2c97760348 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:18.138281 29514 raft_consensus.cc:374] T c19f22d2bc0d47e3a8217b2c97760348 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:18.138366 29514 raft_consensus.cc:729] T c19f22d2bc0d47e3a8217b2c97760348 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:18.138465 29513 raft_consensus.cc:348] T 6f19592b330546b2b4935646ff7b0304 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:18.138585 29514 consensus_queue.cc:260] T c19f22d2bc0d47e3a8217b2c97760348 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:18.138689 29513 raft_consensus.cc:374] T 6f19592b330546b2b4935646ff7b0304 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:18.138770 29513 raft_consensus.cc:729] T 6f19592b330546b2b4935646ff7b0304 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:18.139101 29514 ts_tablet_manager.cc:1398] T c19f22d2bc0d47e3a8217b2c97760348 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:18.139039 29513 consensus_queue.cc:260] T 6f19592b330546b2b4935646ff7b0304 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:18.139609 29513 ts_tablet_manager.cc:1398] T 6f19592b330546b2b4935646ff7b0304 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:18.139890 29513 tablet_bootstrap.cc:492] T c19f22d2bc0d47e3a8217b2c97760348 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:18.139993 29516 tablet_bootstrap.cc:492] T c19f22d2bc0d47e3a8217b2c97760348 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:18.140136 29516 ts_tablet_manager.cc:1367] T c19f22d2bc0d47e3a8217b2c97760348 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240503 08:12:18.140908 29516 raft_consensus.cc:348] T c19f22d2bc0d47e3a8217b2c97760348 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:18.141113 29516 raft_consensus.cc:374] T c19f22d2bc0d47e3a8217b2c97760348 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:18.141220 29516 raft_consensus.cc:729] T c19f22d2bc0d47e3a8217b2c97760348 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:18.141350 29513 tablet_bootstrap.cc:654] T c19f22d2bc0d47e3a8217b2c97760348 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:18.141492 29516 consensus_queue.cc:260] T c19f22d2bc0d47e3a8217b2c97760348 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:18.142076 29516 ts_tablet_manager.cc:1398] T c19f22d2bc0d47e3a8217b2c97760348 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:18.145685 29514 tablet_bootstrap.cc:492] T 6f19592b330546b2b4935646ff7b0304 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:18.148427 29514 tablet_bootstrap.cc:654] T 6f19592b330546b2b4935646ff7b0304 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:18.149616 29512 tablet_bootstrap.cc:492] T 6f19592b330546b2b4935646ff7b0304 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:18.149856 29512 ts_tablet_manager.cc:1367] T 6f19592b330546b2b4935646ff7b0304 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.007s
I20240503 08:12:18.150679 29514 tablet_bootstrap.cc:492] T 6f19592b330546b2b4935646ff7b0304 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:18.150833 29514 ts_tablet_manager.cc:1367] T 6f19592b330546b2b4935646ff7b0304 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 08:12:18.150991 29512 raft_consensus.cc:348] T 6f19592b330546b2b4935646ff7b0304 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:18.151198 29512 raft_consensus.cc:374] T 6f19592b330546b2b4935646ff7b0304 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:18.151295 29512 raft_consensus.cc:729] T 6f19592b330546b2b4935646ff7b0304 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:18.151504 29512 consensus_queue.cc:260] T 6f19592b330546b2b4935646ff7b0304 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:18.151535 29514 raft_consensus.cc:348] T 6f19592b330546b2b4935646ff7b0304 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:18.151731 29514 raft_consensus.cc:374] T 6f19592b330546b2b4935646ff7b0304 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:18.151806 29514 raft_consensus.cc:729] T 6f19592b330546b2b4935646ff7b0304 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:18.152112 29512 ts_tablet_manager.cc:1398] T 6f19592b330546b2b4935646ff7b0304 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:18.152112 29514 consensus_queue.cc:260] T 6f19592b330546b2b4935646ff7b0304 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:18.152494 29512 tablet_bootstrap.cc:492] T 1856d5641d3f48698f5401a1d157369d P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:18.152671 29514 ts_tablet_manager.cc:1398] T 6f19592b330546b2b4935646ff7b0304 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:18.152927 29514 tablet_bootstrap.cc:492] T 1856d5641d3f48698f5401a1d157369d P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:18.154341 29514 tablet_bootstrap.cc:654] T 1856d5641d3f48698f5401a1d157369d P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:18.154668 29513 tablet_bootstrap.cc:492] T c19f22d2bc0d47e3a8217b2c97760348 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:18.154855 29513 ts_tablet_manager.cc:1367] T c19f22d2bc0d47e3a8217b2c97760348 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.004s
I20240503 08:12:18.155619 29513 raft_consensus.cc:348] T c19f22d2bc0d47e3a8217b2c97760348 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:18.155864 29513 raft_consensus.cc:374] T c19f22d2bc0d47e3a8217b2c97760348 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:18.155938 29513 raft_consensus.cc:729] T c19f22d2bc0d47e3a8217b2c97760348 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:18.156210 29513 consensus_queue.cc:260] T c19f22d2bc0d47e3a8217b2c97760348 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:18.156819 29513 ts_tablet_manager.cc:1398] T c19f22d2bc0d47e3a8217b2c97760348 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:18.157415 29512 tablet_bootstrap.cc:654] T 1856d5641d3f48698f5401a1d157369d P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:18.158469 29513 tablet_bootstrap.cc:492] T 1856d5641d3f48698f5401a1d157369d P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:18.158996 29514 tablet_bootstrap.cc:492] T 1856d5641d3f48698f5401a1d157369d P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:18.159135 29514 ts_tablet_manager.cc:1367] T 1856d5641d3f48698f5401a1d157369d P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20240503 08:12:18.159807 29514 raft_consensus.cc:348] T 1856d5641d3f48698f5401a1d157369d P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:18.159960 29513 tablet_bootstrap.cc:654] T 1856d5641d3f48698f5401a1d157369d P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:18.160073 29514 raft_consensus.cc:374] T 1856d5641d3f48698f5401a1d157369d P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:18.160207 29514 raft_consensus.cc:729] T 1856d5641d3f48698f5401a1d157369d P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:18.160476 29514 consensus_queue.cc:260] T 1856d5641d3f48698f5401a1d157369d P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:18.161164 29514 ts_tablet_manager.cc:1398] T 1856d5641d3f48698f5401a1d157369d P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:18.162200 29512 tablet_bootstrap.cc:492] T 1856d5641d3f48698f5401a1d157369d P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:18.162487 29512 ts_tablet_manager.cc:1367] T 1856d5641d3f48698f5401a1d157369d P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240503 08:12:18.163497 29512 raft_consensus.cc:348] T 1856d5641d3f48698f5401a1d157369d P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:18.163700 29512 raft_consensus.cc:374] T 1856d5641d3f48698f5401a1d157369d P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:18.163777 29512 raft_consensus.cc:729] T 1856d5641d3f48698f5401a1d157369d P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:18.164016 29513 tablet_bootstrap.cc:492] T 1856d5641d3f48698f5401a1d157369d P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:18.164176 29513 ts_tablet_manager.cc:1367] T 1856d5641d3f48698f5401a1d157369d P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20240503 08:12:18.164024 29512 consensus_queue.cc:260] T 1856d5641d3f48698f5401a1d157369d P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:18.164922 29513 raft_consensus.cc:348] T 1856d5641d3f48698f5401a1d157369d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:18.165129 29513 raft_consensus.cc:374] T 1856d5641d3f48698f5401a1d157369d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:18.165246 29513 raft_consensus.cc:729] T 1856d5641d3f48698f5401a1d157369d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:18.165475 29513 consensus_queue.cc:260] T 1856d5641d3f48698f5401a1d157369d P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:18.165683 29512 ts_tablet_manager.cc:1398] T 1856d5641d3f48698f5401a1d157369d P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:12:18.165975 29513 ts_tablet_manager.cc:1398] T 1856d5641d3f48698f5401a1d157369d P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:12:18.177708 29403 raft_consensus.cc:482] T 1856d5641d3f48698f5401a1d157369d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:18.177892 29403 raft_consensus.cc:504] T 1856d5641d3f48698f5401a1d157369d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:18.178512 29403 leader_election.cc:288] T 1856d5641d3f48698f5401a1d157369d P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:18.178891 28131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1856d5641d3f48698f5401a1d157369d" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:12:18.178941 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1856d5641d3f48698f5401a1d157369d" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:18.179078 28131 raft_consensus.cc:2454] T 1856d5641d3f48698f5401a1d157369d P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:18.179112 28391 raft_consensus.cc:2454] T 1856d5641d3f48698f5401a1d157369d P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:18.179495 27938 leader_election.cc:302] T 1856d5641d3f48698f5401a1d157369d P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 833582ff68c94269be364ebd187a75db; no voters:
I20240503 08:12:18.179729 29403 raft_consensus.cc:2781] T 1856d5641d3f48698f5401a1d157369d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:18.179838 29403 raft_consensus.cc:482] T 1856d5641d3f48698f5401a1d157369d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:18.179922 29403 raft_consensus.cc:3037] T 1856d5641d3f48698f5401a1d157369d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:18.181807 29403 raft_consensus.cc:504] T 1856d5641d3f48698f5401a1d157369d P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:18.182370 29403 leader_election.cc:288] T 1856d5641d3f48698f5401a1d157369d P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Requested vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:18.182713 28131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1856d5641d3f48698f5401a1d157369d" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:12:18.182790 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1856d5641d3f48698f5401a1d157369d" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:12:18.182868 28131 raft_consensus.cc:3037] T 1856d5641d3f48698f5401a1d157369d P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:18.182963 28391 raft_consensus.cc:3037] T 1856d5641d3f48698f5401a1d157369d P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:18.184958 28391 raft_consensus.cc:2454] T 1856d5641d3f48698f5401a1d157369d P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:18.184971 28131 raft_consensus.cc:2454] T 1856d5641d3f48698f5401a1d157369d P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:18.185303 27938 leader_election.cc:302] T 1856d5641d3f48698f5401a1d157369d P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 833582ff68c94269be364ebd187a75db, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:18.185587 29403 raft_consensus.cc:2781] T 1856d5641d3f48698f5401a1d157369d P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:18.185957 29403 raft_consensus.cc:686] T 1856d5641d3f48698f5401a1d157369d P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Becoming Leader. State: Replica: 833582ff68c94269be364ebd187a75db, State: Running, Role: LEADER
I20240503 08:12:18.186192 29403 consensus_queue.cc:237] T 1856d5641d3f48698f5401a1d157369d P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:18.188655 27859 catalog_manager.cc:5484] T 1856d5641d3f48698f5401a1d157369d P 833582ff68c94269be364ebd187a75db reported cstate change: term changed from 0 to 1, leader changed from <none> to 833582ff68c94269be364ebd187a75db (127.20.230.65). New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:12:18.292340 29403 raft_consensus.cc:482] T c19f22d2bc0d47e3a8217b2c97760348 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:18.292548 29403 raft_consensus.cc:504] T c19f22d2bc0d47e3a8217b2c97760348 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:18.293165 29403 leader_election.cc:288] T c19f22d2bc0d47e3a8217b2c97760348 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:12:18.293792 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c19f22d2bc0d47e3a8217b2c97760348" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:18.293818 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c19f22d2bc0d47e3a8217b2c97760348" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:12:18.294023 28262 raft_consensus.cc:2454] T c19f22d2bc0d47e3a8217b2c97760348 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:18.294041 28391 raft_consensus.cc:2454] T c19f22d2bc0d47e3a8217b2c97760348 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:18.294488 27937 leader_election.cc:302] T c19f22d2bc0d47e3a8217b2c97760348 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:18.294808 29403 raft_consensus.cc:2781] T c19f22d2bc0d47e3a8217b2c97760348 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:18.294922 29403 raft_consensus.cc:482] T c19f22d2bc0d47e3a8217b2c97760348 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:18.295022 29403 raft_consensus.cc:3037] T c19f22d2bc0d47e3a8217b2c97760348 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:18.296973 29403 raft_consensus.cc:504] T c19f22d2bc0d47e3a8217b2c97760348 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:18.297591 29403 leader_election.cc:288] T c19f22d2bc0d47e3a8217b2c97760348 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Requested vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:12:18.298039 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c19f22d2bc0d47e3a8217b2c97760348" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:12:18.298133 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c19f22d2bc0d47e3a8217b2c97760348" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:12:18.298231 28391 raft_consensus.cc:3037] T c19f22d2bc0d47e3a8217b2c97760348 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:18.298359 28262 raft_consensus.cc:3037] T c19f22d2bc0d47e3a8217b2c97760348 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:18.300166 28391 raft_consensus.cc:2454] T c19f22d2bc0d47e3a8217b2c97760348 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:18.300205 28262 raft_consensus.cc:2454] T c19f22d2bc0d47e3a8217b2c97760348 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:18.300537 27939 leader_election.cc:302] T c19f22d2bc0d47e3a8217b2c97760348 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:18.300810 29403 raft_consensus.cc:2781] T c19f22d2bc0d47e3a8217b2c97760348 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:18.300972 29403 raft_consensus.cc:686] T c19f22d2bc0d47e3a8217b2c97760348 P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Becoming Leader. State: Replica: 833582ff68c94269be364ebd187a75db, State: Running, Role: LEADER
I20240503 08:12:18.301234 29403 consensus_queue.cc:237] T c19f22d2bc0d47e3a8217b2c97760348 P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:18.304009 27859 catalog_manager.cc:5484] T c19f22d2bc0d47e3a8217b2c97760348 P 833582ff68c94269be364ebd187a75db reported cstate change: term changed from 0 to 1, leader changed from <none> to 833582ff68c94269be364ebd187a75db (127.20.230.65). New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:12:18.326887 29403 raft_consensus.cc:482] T 6f19592b330546b2b4935646ff7b0304 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:18.327062 29403 raft_consensus.cc:504] T 6f19592b330546b2b4935646ff7b0304 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:18.327706 29403 leader_election.cc:288] T 6f19592b330546b2b4935646ff7b0304 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:18.328038 28131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f19592b330546b2b4935646ff7b0304" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:12:18.328161 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f19592b330546b2b4935646ff7b0304" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:18.328262 28131 raft_consensus.cc:2454] T 6f19592b330546b2b4935646ff7b0304 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:18.328387 28391 raft_consensus.cc:2454] T 6f19592b330546b2b4935646ff7b0304 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:18.328620 27938 leader_election.cc:302] T 6f19592b330546b2b4935646ff7b0304 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 833582ff68c94269be364ebd187a75db; no voters:
I20240503 08:12:18.328860 29403 raft_consensus.cc:2781] T 6f19592b330546b2b4935646ff7b0304 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:18.328974 29403 raft_consensus.cc:482] T 6f19592b330546b2b4935646ff7b0304 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:18.329056 29403 raft_consensus.cc:3037] T 6f19592b330546b2b4935646ff7b0304 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:18.330938 29403 raft_consensus.cc:504] T 6f19592b330546b2b4935646ff7b0304 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:18.331482 29403 leader_election.cc:288] T 6f19592b330546b2b4935646ff7b0304 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Requested vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:18.331823 28131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f19592b330546b2b4935646ff7b0304" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:12:18.332079 28131 raft_consensus.cc:3037] T 6f19592b330546b2b4935646ff7b0304 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:18.332068 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f19592b330546b2b4935646ff7b0304" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:12:18.332276 28391 raft_consensus.cc:3037] T 6f19592b330546b2b4935646ff7b0304 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:18.334774 28131 raft_consensus.cc:2454] T 6f19592b330546b2b4935646ff7b0304 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:18.334913 28391 raft_consensus.cc:2454] T 6f19592b330546b2b4935646ff7b0304 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:18.335194 27938 leader_election.cc:302] T 6f19592b330546b2b4935646ff7b0304 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 833582ff68c94269be364ebd187a75db; no voters:
I20240503 08:12:18.335492 29403 raft_consensus.cc:2781] T 6f19592b330546b2b4935646ff7b0304 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:18.335614 29403 raft_consensus.cc:686] T 6f19592b330546b2b4935646ff7b0304 P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Becoming Leader. State: Replica: 833582ff68c94269be364ebd187a75db, State: Running, Role: LEADER
I20240503 08:12:18.335882 29403 consensus_queue.cc:237] T 6f19592b330546b2b4935646ff7b0304 P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:18.338676 27859 catalog_manager.cc:5484] T 6f19592b330546b2b4935646ff7b0304 P 833582ff68c94269be364ebd187a75db reported cstate change: term changed from 0 to 1, leader changed from <none> to 833582ff68c94269be364ebd187a75db (127.20.230.65). New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
W20240503 08:12:18.688628 29530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:18.689002 29530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:18.699957 29530 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240503 08:12:18.747140 29403 consensus_queue.cc:1035] T 1856d5641d3f48698f5401a1d157369d P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:18.759800 29526 consensus_queue.cc:1035] T 1856d5641d3f48698f5401a1d157369d P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:18.808074 29526 consensus_queue.cc:1035] T c19f22d2bc0d47e3a8217b2c97760348 P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:18.823805 29403 consensus_queue.cc:1035] T c19f22d2bc0d47e3a8217b2c97760348 P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:18.853329 28131 raft_consensus.cc:1261] T 6f19592b330546b2b4935646ff7b0304 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:18.853984 28391 raft_consensus.cc:1261] T 6f19592b330546b2b4935646ff7b0304 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:18.853924 29559 consensus_queue.cc:1035] T 6f19592b330546b2b4935646ff7b0304 P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:18.855846 29559 consensus_queue.cc:1035] T 6f19592b330546b2b4935646ff7b0304 P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 220.456 ms
time per row: 0.110228 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 08:12:19.942440 27859 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:19.951287 27982 tablet_service.cc:1467] Processing CreateTablet for tablet df90a0d9d338416b81e253fa23542cca (DEFAULT_TABLE table=rebalance_test_table_7 [id=f65dccb36e67468c8e28e95e66b5e4ef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:19.951472 28242 tablet_service.cc:1467] Processing CreateTablet for tablet df90a0d9d338416b81e253fa23542cca (DEFAULT_TABLE table=rebalance_test_table_7 [id=f65dccb36e67468c8e28e95e66b5e4ef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:19.951741 27982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df90a0d9d338416b81e253fa23542cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:19.951821 28242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df90a0d9d338416b81e253fa23542cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:19.951934 28111 tablet_service.cc:1467] Processing CreateTablet for tablet df90a0d9d338416b81e253fa23542cca (DEFAULT_TABLE table=rebalance_test_table_7 [id=f65dccb36e67468c8e28e95e66b5e4ef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:19.952334 28111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df90a0d9d338416b81e253fa23542cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:19.954957 28243 tablet_service.cc:1467] Processing CreateTablet for tablet 8cc6ae30aa60495d8a4364a11d5bef79 (DEFAULT_TABLE table=rebalance_test_table_7 [id=f65dccb36e67468c8e28e95e66b5e4ef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:19.955322 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cc6ae30aa60495d8a4364a11d5bef79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:19.959072 28112 tablet_service.cc:1467] Processing CreateTablet for tablet f46393257ce9471891698c36a07546ad (DEFAULT_TABLE table=rebalance_test_table_7 [id=f65dccb36e67468c8e28e95e66b5e4ef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:19.959463 28112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f46393257ce9471891698c36a07546ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:19.960049 28241 tablet_service.cc:1467] Processing CreateTablet for tablet f46393257ce9471891698c36a07546ad (DEFAULT_TABLE table=rebalance_test_table_7 [id=f65dccb36e67468c8e28e95e66b5e4ef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:19.960157 28372 tablet_service.cc:1467] Processing CreateTablet for tablet f46393257ce9471891698c36a07546ad (DEFAULT_TABLE table=rebalance_test_table_7 [id=f65dccb36e67468c8e28e95e66b5e4ef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:19.960388 28241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f46393257ce9471891698c36a07546ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:19.960508 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f46393257ce9471891698c36a07546ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:19.962213 28373 tablet_service.cc:1467] Processing CreateTablet for tablet 8cc6ae30aa60495d8a4364a11d5bef79 (DEFAULT_TABLE table=rebalance_test_table_7 [id=f65dccb36e67468c8e28e95e66b5e4ef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:19.962556 28373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cc6ae30aa60495d8a4364a11d5bef79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:19.966357 27983 tablet_service.cc:1467] Processing CreateTablet for tablet 8cc6ae30aa60495d8a4364a11d5bef79 (DEFAULT_TABLE table=rebalance_test_table_7 [id=f65dccb36e67468c8e28e95e66b5e4ef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:19.966755 27983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cc6ae30aa60495d8a4364a11d5bef79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:19.968246 29593 tablet_bootstrap.cc:492] T df90a0d9d338416b81e253fa23542cca P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:19.970075 29593 tablet_bootstrap.cc:654] T df90a0d9d338416b81e253fa23542cca P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:19.974346 29594 tablet_bootstrap.cc:492] T 8cc6ae30aa60495d8a4364a11d5bef79 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:19.974444 29597 tablet_bootstrap.cc:492] T f46393257ce9471891698c36a07546ad P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:19.976702 29594 tablet_bootstrap.cc:654] T 8cc6ae30aa60495d8a4364a11d5bef79 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:19.977030 29597 tablet_bootstrap.cc:654] T f46393257ce9471891698c36a07546ad P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:19.983062 29596 tablet_bootstrap.cc:492] T 8cc6ae30aa60495d8a4364a11d5bef79 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:19.983590 29593 tablet_bootstrap.cc:492] T df90a0d9d338416b81e253fa23542cca P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:19.983722 29593 ts_tablet_manager.cc:1367] T df90a0d9d338416b81e253fa23542cca P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.003s
I20240503 08:12:19.984094 29594 tablet_bootstrap.cc:492] T 8cc6ae30aa60495d8a4364a11d5bef79 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:19.984246 29594 ts_tablet_manager.cc:1367] T 8cc6ae30aa60495d8a4364a11d5bef79 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20240503 08:12:19.984463 29593 raft_consensus.cc:348] T df90a0d9d338416b81e253fa23542cca P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:19.984606 29596 tablet_bootstrap.cc:654] T 8cc6ae30aa60495d8a4364a11d5bef79 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:19.984622 29593 raft_consensus.cc:374] T df90a0d9d338416b81e253fa23542cca P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:19.984699 29593 raft_consensus.cc:729] T df90a0d9d338416b81e253fa23542cca P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:19.984900 29593 consensus_queue.cc:260] T df90a0d9d338416b81e253fa23542cca P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:19.984925 29594 raft_consensus.cc:348] T 8cc6ae30aa60495d8a4364a11d5bef79 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:19.985177 29594 raft_consensus.cc:374] T 8cc6ae30aa60495d8a4364a11d5bef79 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:19.985317 29594 raft_consensus.cc:729] T 8cc6ae30aa60495d8a4364a11d5bef79 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:19.985605 29594 consensus_queue.cc:260] T 8cc6ae30aa60495d8a4364a11d5bef79 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:19.986366 29594 ts_tablet_manager.cc:1398] T 8cc6ae30aa60495d8a4364a11d5bef79 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:19.986526 29596 tablet_bootstrap.cc:492] T 8cc6ae30aa60495d8a4364a11d5bef79 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:19.986675 29594 tablet_bootstrap.cc:492] T f46393257ce9471891698c36a07546ad P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:19.986701 29596 ts_tablet_manager.cc:1367] T 8cc6ae30aa60495d8a4364a11d5bef79 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:12:19.987346 29593 ts_tablet_manager.cc:1398] T df90a0d9d338416b81e253fa23542cca P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 08:12:19.987387 29596 raft_consensus.cc:348] T 8cc6ae30aa60495d8a4364a11d5bef79 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:19.987654 29596 raft_consensus.cc:374] T 8cc6ae30aa60495d8a4364a11d5bef79 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:19.987795 29596 raft_consensus.cc:729] T 8cc6ae30aa60495d8a4364a11d5bef79 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:19.987735 29593 tablet_bootstrap.cc:492] T 8cc6ae30aa60495d8a4364a11d5bef79 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:19.988037 29596 consensus_queue.cc:260] T 8cc6ae30aa60495d8a4364a11d5bef79 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:19.988230 29594 tablet_bootstrap.cc:654] T f46393257ce9471891698c36a07546ad P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:19.989544 29596 ts_tablet_manager.cc:1398] T 8cc6ae30aa60495d8a4364a11d5bef79 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:12:19.989843 29596 tablet_bootstrap.cc:492] T f46393257ce9471891698c36a07546ad P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:19.992076 29596 tablet_bootstrap.cc:654] T f46393257ce9471891698c36a07546ad P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:19.993564 29593 tablet_bootstrap.cc:654] T 8cc6ae30aa60495d8a4364a11d5bef79 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:19.996019 29597 tablet_bootstrap.cc:492] T f46393257ce9471891698c36a07546ad P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:19.996243 29597 ts_tablet_manager.cc:1367] T f46393257ce9471891698c36a07546ad P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20240503 08:12:19.997112 29597 raft_consensus.cc:348] T f46393257ce9471891698c36a07546ad P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:19.997396 29597 raft_consensus.cc:374] T f46393257ce9471891698c36a07546ad P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:19.997521 29597 raft_consensus.cc:729] T f46393257ce9471891698c36a07546ad P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:19.997857 29597 consensus_queue.cc:260] T f46393257ce9471891698c36a07546ad P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:19.999750 29587 raft_consensus.cc:482] T 8cc6ae30aa60495d8a4364a11d5bef79 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:19.999883 29587 raft_consensus.cc:504] T 8cc6ae30aa60495d8a4364a11d5bef79 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:20.000041 29593 tablet_bootstrap.cc:492] T 8cc6ae30aa60495d8a4364a11d5bef79 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:20.000192 29593 ts_tablet_manager.cc:1367] T 8cc6ae30aa60495d8a4364a11d5bef79 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.003s
I20240503 08:12:20.000510 29587 leader_election.cc:288] T 8cc6ae30aa60495d8a4364a11d5bef79 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:20.000888 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cc6ae30aa60495d8a4364a11d5bef79" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
W20240503 08:12:20.001806 28199 leader_election.cc:341] T 8cc6ae30aa60495d8a4364a11d5bef79 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:20.002687 29597 ts_tablet_manager.cc:1398] T f46393257ce9471891698c36a07546ad P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:12:20.003433 29593 raft_consensus.cc:348] T 8cc6ae30aa60495d8a4364a11d5bef79 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:20.003656 29597 tablet_bootstrap.cc:492] T df90a0d9d338416b81e253fa23542cca P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:20.004120 29593 raft_consensus.cc:374] T 8cc6ae30aa60495d8a4364a11d5bef79 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:20.004344 29593 raft_consensus.cc:729] T 8cc6ae30aa60495d8a4364a11d5bef79 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:20.004793 29593 consensus_queue.cc:260] T 8cc6ae30aa60495d8a4364a11d5bef79 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:20.005915 29593 ts_tablet_manager.cc:1398] T 8cc6ae30aa60495d8a4364a11d5bef79 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20240503 08:12:20.006964 29596 tablet_bootstrap.cc:492] T f46393257ce9471891698c36a07546ad P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:20.007129 29596 ts_tablet_manager.cc:1367] T f46393257ce9471891698c36a07546ad P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.005s
I20240503 08:12:20.007279 29597 tablet_bootstrap.cc:654] T df90a0d9d338416b81e253fa23542cca P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:20.007897 29596 raft_consensus.cc:348] T f46393257ce9471891698c36a07546ad P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:20.008108 29596 raft_consensus.cc:374] T f46393257ce9471891698c36a07546ad P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:20.008185 29596 raft_consensus.cc:729] T f46393257ce9471891698c36a07546ad P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:20.008416 29596 consensus_queue.cc:260] T f46393257ce9471891698c36a07546ad P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:20.009744 29594 tablet_bootstrap.cc:492] T f46393257ce9471891698c36a07546ad P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:20.009996 29594 ts_tablet_manager.cc:1367] T f46393257ce9471891698c36a07546ad P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.016s
I20240503 08:12:20.010840 29596 ts_tablet_manager.cc:1398] T f46393257ce9471891698c36a07546ad P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 08:12:20.010810 29594 raft_consensus.cc:348] T f46393257ce9471891698c36a07546ad P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:20.011138 29594 raft_consensus.cc:374] T f46393257ce9471891698c36a07546ad P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:20.011265 29594 raft_consensus.cc:729] T f46393257ce9471891698c36a07546ad P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:20.011567 29594 consensus_queue.cc:260] T f46393257ce9471891698c36a07546ad P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:20.012295 29594 ts_tablet_manager.cc:1398] T f46393257ce9471891698c36a07546ad P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:12:20.013401 29594 tablet_bootstrap.cc:492] T df90a0d9d338416b81e253fa23542cca P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:20.015254 29594 tablet_bootstrap.cc:654] T df90a0d9d338416b81e253fa23542cca P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:20.016578 29597 tablet_bootstrap.cc:492] T df90a0d9d338416b81e253fa23542cca P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:20.016791 29597 ts_tablet_manager.cc:1367] T df90a0d9d338416b81e253fa23542cca P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240503 08:12:20.017614 29597 raft_consensus.cc:348] T df90a0d9d338416b81e253fa23542cca P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:20.017892 29597 raft_consensus.cc:374] T df90a0d9d338416b81e253fa23542cca P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:20.017994 29597 raft_consensus.cc:729] T df90a0d9d338416b81e253fa23542cca P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:20.018291 29597 consensus_queue.cc:260] T df90a0d9d338416b81e253fa23542cca P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:20.018867 29597 ts_tablet_manager.cc:1398] T df90a0d9d338416b81e253fa23542cca P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:12:20.018956 28392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cc6ae30aa60495d8a4364a11d5bef79" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:20.019176 28392 raft_consensus.cc:2454] T 8cc6ae30aa60495d8a4364a11d5bef79 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:12:20.019742 28199 leader_election.cc:302] T 8cc6ae30aa60495d8a4364a11d5bef79 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, d09a11a03989421b9f28019dc1c14cc2; no voters: 833582ff68c94269be364ebd187a75db
I20240503 08:12:20.020045 29587 raft_consensus.cc:2781] T 8cc6ae30aa60495d8a4364a11d5bef79 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:20.020073 29594 tablet_bootstrap.cc:492] T df90a0d9d338416b81e253fa23542cca P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:20.020181 29587 raft_consensus.cc:482] T 8cc6ae30aa60495d8a4364a11d5bef79 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:20.020242 29594 ts_tablet_manager.cc:1367] T df90a0d9d338416b81e253fa23542cca P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240503 08:12:20.020289 29587 raft_consensus.cc:3037] T 8cc6ae30aa60495d8a4364a11d5bef79 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:20.021049 29594 raft_consensus.cc:348] T df90a0d9d338416b81e253fa23542cca P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:20.021338 29594 raft_consensus.cc:374] T df90a0d9d338416b81e253fa23542cca P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:20.021488 29594 raft_consensus.cc:729] T df90a0d9d338416b81e253fa23542cca P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:20.021800 29594 consensus_queue.cc:260] T df90a0d9d338416b81e253fa23542cca P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:20.022567 29587 raft_consensus.cc:504] T 8cc6ae30aa60495d8a4364a11d5bef79 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:20.023102 29587 leader_election.cc:288] T 8cc6ae30aa60495d8a4364a11d5bef79 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Requested vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:20.023519 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cc6ae30aa60495d8a4364a11d5bef79" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:12:20.023702 28002 raft_consensus.cc:3037] T 8cc6ae30aa60495d8a4364a11d5bef79 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:20.023986 28392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cc6ae30aa60495d8a4364a11d5bef79" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:12:20.024189 28392 raft_consensus.cc:3037] T 8cc6ae30aa60495d8a4364a11d5bef79 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:20.026204 28002 raft_consensus.cc:2454] T 8cc6ae30aa60495d8a4364a11d5bef79 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:20.026561 28199 leader_election.cc:302] T 8cc6ae30aa60495d8a4364a11d5bef79 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:20.026808 29587 raft_consensus.cc:2781] T 8cc6ae30aa60495d8a4364a11d5bef79 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:20.026921 29587 raft_consensus.cc:686] T 8cc6ae30aa60495d8a4364a11d5bef79 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Becoming Leader. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Running, Role: LEADER
I20240503 08:12:20.026978 28392 raft_consensus.cc:2454] T 8cc6ae30aa60495d8a4364a11d5bef79 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:20.027206 29587 consensus_queue.cc:237] T 8cc6ae30aa60495d8a4364a11d5bef79 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:20.028937 29594 ts_tablet_manager.cc:1398] T df90a0d9d338416b81e253fa23542cca P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.009s user 0.000s sys 0.008s
I20240503 08:12:20.029997 27856 catalog_manager.cc:5484] T 8cc6ae30aa60495d8a4364a11d5bef79 P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: term changed from 0 to 1, leader changed from <none> to aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67). New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:12:20.112556 29525 raft_consensus.cc:482] T df90a0d9d338416b81e253fa23542cca P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:20.112720 29525 raft_consensus.cc:504] T df90a0d9d338416b81e253fa23542cca P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:20.113307 29525 leader_election.cc:288] T df90a0d9d338416b81e253fa23542cca P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:12:20.113691 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df90a0d9d338416b81e253fa23542cca" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:12:20.113746 28261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df90a0d9d338416b81e253fa23542cca" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:12:20.113934 28002 raft_consensus.cc:2454] T df90a0d9d338416b81e253fa23542cca P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:20.113988 28261 raft_consensus.cc:2454] T df90a0d9d338416b81e253fa23542cca P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:20.114339 28069 leader_election.cc:302] T df90a0d9d338416b81e253fa23542cca P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 833582ff68c94269be364ebd187a75db; no voters:
I20240503 08:12:20.114607 29525 raft_consensus.cc:2781] T df90a0d9d338416b81e253fa23542cca P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:20.114720 29525 raft_consensus.cc:482] T df90a0d9d338416b81e253fa23542cca P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:20.114799 29525 raft_consensus.cc:3037] T df90a0d9d338416b81e253fa23542cca P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:20.116595 29525 raft_consensus.cc:504] T df90a0d9d338416b81e253fa23542cca P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:20.117090 29525 leader_election.cc:288] T df90a0d9d338416b81e253fa23542cca P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Requested vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:12:20.117434 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df90a0d9d338416b81e253fa23542cca" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:12:20.117534 28261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df90a0d9d338416b81e253fa23542cca" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:12:20.117627 28002 raft_consensus.cc:3037] T df90a0d9d338416b81e253fa23542cca P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:20.117698 28261 raft_consensus.cc:3037] T df90a0d9d338416b81e253fa23542cca P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:20.119417 28002 raft_consensus.cc:2454] T df90a0d9d338416b81e253fa23542cca P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:12:20.119558 28261 raft_consensus.cc:2454] T df90a0d9d338416b81e253fa23542cca P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:12:20.119766 28069 leader_election.cc:302] T df90a0d9d338416b81e253fa23542cca P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 833582ff68c94269be364ebd187a75db; no voters:
I20240503 08:12:20.120041 29525 raft_consensus.cc:2781] T df90a0d9d338416b81e253fa23542cca P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:20.120437 29525 raft_consensus.cc:686] T df90a0d9d338416b81e253fa23542cca P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Becoming Leader. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Running, Role: LEADER
I20240503 08:12:20.120733 29525 consensus_queue.cc:237] T df90a0d9d338416b81e253fa23542cca P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:20.123286 27856 catalog_manager.cc:5484] T df90a0d9d338416b81e253fa23542cca P 1b411c6bb25549368041773d7b710d48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b411c6bb25549368041773d7b710d48 (127.20.230.66). New cstate: current_term: 1 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:12:20.183633 29608 raft_consensus.cc:482] T f46393257ce9471891698c36a07546ad P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:20.183790 29608 raft_consensus.cc:504] T f46393257ce9471891698c36a07546ad P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:20.184381 29608 leader_election.cc:288] T f46393257ce9471891698c36a07546ad P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:20.184772 28133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46393257ce9471891698c36a07546ad" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:12:20.184808 28392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46393257ce9471891698c36a07546ad" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:20.184993 28392 raft_consensus.cc:2454] T f46393257ce9471891698c36a07546ad P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:12:20.185007 28133 raft_consensus.cc:2454] T f46393257ce9471891698c36a07546ad P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:12:20.185340 28199 leader_election.cc:302] T f46393257ce9471891698c36a07546ad P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:20.185604 29608 raft_consensus.cc:2781] T f46393257ce9471891698c36a07546ad P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:20.185724 29608 raft_consensus.cc:482] T f46393257ce9471891698c36a07546ad P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:20.185873 29608 raft_consensus.cc:3037] T f46393257ce9471891698c36a07546ad P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:20.187785 29608 raft_consensus.cc:504] T f46393257ce9471891698c36a07546ad P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:20.188251 29608 leader_election.cc:288] T f46393257ce9471891698c36a07546ad P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Requested vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:20.188692 28133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46393257ce9471891698c36a07546ad" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:12:20.188676 28392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46393257ce9471891698c36a07546ad" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:12:20.188886 28392 raft_consensus.cc:3037] T f46393257ce9471891698c36a07546ad P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:20.188899 28133 raft_consensus.cc:3037] T f46393257ce9471891698c36a07546ad P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:20.190851 28133 raft_consensus.cc:2454] T f46393257ce9471891698c36a07546ad P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:20.190944 28392 raft_consensus.cc:2454] T f46393257ce9471891698c36a07546ad P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:20.191334 28198 leader_election.cc:302] T f46393257ce9471891698c36a07546ad P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:20.191608 29608 raft_consensus.cc:2781] T f46393257ce9471891698c36a07546ad P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:20.191743 29608 raft_consensus.cc:686] T f46393257ce9471891698c36a07546ad P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Becoming Leader. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Running, Role: LEADER
I20240503 08:12:20.192021 29608 consensus_queue.cc:237] T f46393257ce9471891698c36a07546ad P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:20.194720 27856 catalog_manager.cc:5484] T f46393257ce9471891698c36a07546ad P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: term changed from 0 to 1, leader changed from <none> to aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67). New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:12:20.512410 29608 consensus_queue.cc:1035] T 8cc6ae30aa60495d8a4364a11d5bef79 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:20.517742 29608 consensus_queue.cc:1035] T 8cc6ae30aa60495d8a4364a11d5bef79 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:20.551635 29613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:20.552059 29613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:20.559715 29613 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240503 08:12:20.653800 29610 consensus_queue.cc:1035] T df90a0d9d338416b81e253fa23542cca P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:20.658838 29610 consensus_queue.cc:1035] T df90a0d9d338416b81e253fa23542cca P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:20.662846 29608 consensus_queue.cc:1035] T f46393257ce9471891698c36a07546ad P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:20.683732 29587 consensus_queue.cc:1035] T f46393257ce9471891698c36a07546ad P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 280.052 ms
time per row: 0.140026 ms
I20240503 08:12:20.984501 27862 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:20.992697 28242 tablet_service.cc:1467] Processing CreateTablet for tablet b8560620faf54b51ba4c92d8cf95be2d (DEFAULT_TABLE table=rebalance_test_table_8 [id=53655b23fc8b423298fafe1b2c13a1f7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:20.992795 27983 tablet_service.cc:1467] Processing CreateTablet for tablet b8560620faf54b51ba4c92d8cf95be2d (DEFAULT_TABLE table=rebalance_test_table_8 [id=53655b23fc8b423298fafe1b2c13a1f7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:20.993148 28242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8560620faf54b51ba4c92d8cf95be2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:20.993147 27983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8560620faf54b51ba4c92d8cf95be2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:20.993446 28111 tablet_service.cc:1467] Processing CreateTablet for tablet b8560620faf54b51ba4c92d8cf95be2d (DEFAULT_TABLE table=rebalance_test_table_8 [id=53655b23fc8b423298fafe1b2c13a1f7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:20.993909 28111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8560620faf54b51ba4c92d8cf95be2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:20.994233 28112 tablet_service.cc:1467] Processing CreateTablet for tablet 582128d9dc884a9ab5ecc3e1c90e535e (DEFAULT_TABLE table=rebalance_test_table_8 [id=53655b23fc8b423298fafe1b2c13a1f7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:20.994319 28243 tablet_service.cc:1467] Processing CreateTablet for tablet 2fbd9559f45645fd8df45491295224ba (DEFAULT_TABLE table=rebalance_test_table_8 [id=53655b23fc8b423298fafe1b2c13a1f7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:20.994598 28112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 582128d9dc884a9ab5ecc3e1c90e535e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:20.994709 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fbd9559f45645fd8df45491295224ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:21.005607 28373 tablet_service.cc:1467] Processing CreateTablet for tablet 582128d9dc884a9ab5ecc3e1c90e535e (DEFAULT_TABLE table=rebalance_test_table_8 [id=53655b23fc8b423298fafe1b2c13a1f7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:21.005738 28372 tablet_service.cc:1467] Processing CreateTablet for tablet 2fbd9559f45645fd8df45491295224ba (DEFAULT_TABLE table=rebalance_test_table_8 [id=53655b23fc8b423298fafe1b2c13a1f7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:21.006124 28373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 582128d9dc884a9ab5ecc3e1c90e535e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:21.006544 29680 tablet_bootstrap.cc:492] T 2fbd9559f45645fd8df45491295224ba P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:21.006620 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fbd9559f45645fd8df45491295224ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:21.008285 29680 tablet_bootstrap.cc:654] T 2fbd9559f45645fd8df45491295224ba P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:21.011093 27982 tablet_service.cc:1467] Processing CreateTablet for tablet 582128d9dc884a9ab5ecc3e1c90e535e (DEFAULT_TABLE table=rebalance_test_table_8 [id=53655b23fc8b423298fafe1b2c13a1f7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:21.011507 27982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 582128d9dc884a9ab5ecc3e1c90e535e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:21.017539 27981 tablet_service.cc:1467] Processing CreateTablet for tablet 2fbd9559f45645fd8df45491295224ba (DEFAULT_TABLE table=rebalance_test_table_8 [id=53655b23fc8b423298fafe1b2c13a1f7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:21.017928 27981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fbd9559f45645fd8df45491295224ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:21.019944 29679 tablet_bootstrap.cc:492] T b8560620faf54b51ba4c92d8cf95be2d P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:21.021868 29682 tablet_bootstrap.cc:492] T 2fbd9559f45645fd8df45491295224ba P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:21.023509 29682 tablet_bootstrap.cc:654] T 2fbd9559f45645fd8df45491295224ba P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:21.023584 29679 tablet_bootstrap.cc:654] T b8560620faf54b51ba4c92d8cf95be2d P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:21.024842 29678 tablet_bootstrap.cc:492] T b8560620faf54b51ba4c92d8cf95be2d P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:21.025730 29680 tablet_bootstrap.cc:492] T 2fbd9559f45645fd8df45491295224ba P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:21.025884 29679 tablet_bootstrap.cc:492] T b8560620faf54b51ba4c92d8cf95be2d P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:21.025916 29680 ts_tablet_manager.cc:1367] T 2fbd9559f45645fd8df45491295224ba P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.006s
I20240503 08:12:21.026114 29679 ts_tablet_manager.cc:1367] T b8560620faf54b51ba4c92d8cf95be2d P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20240503 08:12:21.026726 29680 raft_consensus.cc:348] T 2fbd9559f45645fd8df45491295224ba P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:21.027040 29680 raft_consensus.cc:374] T 2fbd9559f45645fd8df45491295224ba P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:21.027189 29678 tablet_bootstrap.cc:654] T b8560620faf54b51ba4c92d8cf95be2d P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:21.027194 29680 raft_consensus.cc:729] T 2fbd9559f45645fd8df45491295224ba P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:21.026877 29679 raft_consensus.cc:348] T b8560620faf54b51ba4c92d8cf95be2d P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:21.027501 29679 raft_consensus.cc:374] T b8560620faf54b51ba4c92d8cf95be2d P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:21.027455 29680 consensus_queue.cc:260] T 2fbd9559f45645fd8df45491295224ba P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:21.027638 29679 raft_consensus.cc:729] T b8560620faf54b51ba4c92d8cf95be2d P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:21.027890 29679 consensus_queue.cc:260] T b8560620faf54b51ba4c92d8cf95be2d P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:21.028029 29682 tablet_bootstrap.cc:492] T 2fbd9559f45645fd8df45491295224ba P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:21.028121 29680 ts_tablet_manager.cc:1398] T 2fbd9559f45645fd8df45491295224ba P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:21.028168 29682 ts_tablet_manager.cc:1367] T 2fbd9559f45645fd8df45491295224ba P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 08:12:21.028653 29679 ts_tablet_manager.cc:1398] T b8560620faf54b51ba4c92d8cf95be2d P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:12:21.028992 29679 tablet_bootstrap.cc:492] T 582128d9dc884a9ab5ecc3e1c90e535e P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:21.028932 29682 raft_consensus.cc:348] T 2fbd9559f45645fd8df45491295224ba P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:21.029198 29682 raft_consensus.cc:374] T 2fbd9559f45645fd8df45491295224ba P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:21.029348 29682 raft_consensus.cc:729] T 2fbd9559f45645fd8df45491295224ba P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:21.029657 29682 consensus_queue.cc:260] T 2fbd9559f45645fd8df45491295224ba P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:21.030380 29682 ts_tablet_manager.cc:1398] T 2fbd9559f45645fd8df45491295224ba P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:21.031044 29679 tablet_bootstrap.cc:654] T 582128d9dc884a9ab5ecc3e1c90e535e P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:21.038201 29682 tablet_bootstrap.cc:492] T 582128d9dc884a9ab5ecc3e1c90e535e P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:21.040572 29682 tablet_bootstrap.cc:654] T 582128d9dc884a9ab5ecc3e1c90e535e P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:21.044625 29678 tablet_bootstrap.cc:492] T b8560620faf54b51ba4c92d8cf95be2d P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:21.044869 29678 ts_tablet_manager.cc:1367] T b8560620faf54b51ba4c92d8cf95be2d P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.006s
I20240503 08:12:21.046005 29680 tablet_bootstrap.cc:492] T b8560620faf54b51ba4c92d8cf95be2d P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:21.046001 29678 raft_consensus.cc:348] T b8560620faf54b51ba4c92d8cf95be2d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:21.046465 29678 raft_consensus.cc:374] T b8560620faf54b51ba4c92d8cf95be2d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:21.047919 29680 tablet_bootstrap.cc:654] T b8560620faf54b51ba4c92d8cf95be2d P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:21.046658 29678 raft_consensus.cc:729] T b8560620faf54b51ba4c92d8cf95be2d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:21.048599 29678 consensus_queue.cc:260] T b8560620faf54b51ba4c92d8cf95be2d P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:21.048961 29682 tablet_bootstrap.cc:492] T 582128d9dc884a9ab5ecc3e1c90e535e P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:21.049176 29682 ts_tablet_manager.cc:1367] T 582128d9dc884a9ab5ecc3e1c90e535e P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20240503 08:12:21.050068 29682 raft_consensus.cc:348] T 582128d9dc884a9ab5ecc3e1c90e535e P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:21.050364 29682 raft_consensus.cc:374] T 582128d9dc884a9ab5ecc3e1c90e535e P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:21.050498 29682 raft_consensus.cc:729] T 582128d9dc884a9ab5ecc3e1c90e535e P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:21.050839 29682 consensus_queue.cc:260] T 582128d9dc884a9ab5ecc3e1c90e535e P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:21.051795 29680 tablet_bootstrap.cc:492] T b8560620faf54b51ba4c92d8cf95be2d P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:21.052067 29680 ts_tablet_manager.cc:1367] T b8560620faf54b51ba4c92d8cf95be2d P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20240503 08:12:21.052269 29682 ts_tablet_manager.cc:1398] T 582128d9dc884a9ab5ecc3e1c90e535e P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:21.052939 29680 raft_consensus.cc:348] T b8560620faf54b51ba4c92d8cf95be2d P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:21.053184 29680 raft_consensus.cc:374] T b8560620faf54b51ba4c92d8cf95be2d P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:21.053259 29680 raft_consensus.cc:729] T b8560620faf54b51ba4c92d8cf95be2d P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:21.053632 29680 consensus_queue.cc:260] T b8560620faf54b51ba4c92d8cf95be2d P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:21.054049 29678 ts_tablet_manager.cc:1398] T b8560620faf54b51ba4c92d8cf95be2d P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 08:12:21.055227 29680 ts_tablet_manager.cc:1398] T b8560620faf54b51ba4c92d8cf95be2d P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:21.060918 29678 tablet_bootstrap.cc:492] T 582128d9dc884a9ab5ecc3e1c90e535e P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:21.061879 29679 tablet_bootstrap.cc:492] T 582128d9dc884a9ab5ecc3e1c90e535e P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:21.062129 29679 ts_tablet_manager.cc:1367] T 582128d9dc884a9ab5ecc3e1c90e535e P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.033s user 0.019s sys 0.012s
I20240503 08:12:21.062498 29678 tablet_bootstrap.cc:654] T 582128d9dc884a9ab5ecc3e1c90e535e P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:21.063015 29679 raft_consensus.cc:348] T 582128d9dc884a9ab5ecc3e1c90e535e P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:21.063334 29679 raft_consensus.cc:374] T 582128d9dc884a9ab5ecc3e1c90e535e P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:21.063555 29679 raft_consensus.cc:729] T 582128d9dc884a9ab5ecc3e1c90e535e P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:21.063863 29679 consensus_queue.cc:260] T 582128d9dc884a9ab5ecc3e1c90e535e P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:21.064483 29679 ts_tablet_manager.cc:1398] T 582128d9dc884a9ab5ecc3e1c90e535e P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:21.065392 29678 tablet_bootstrap.cc:492] T 582128d9dc884a9ab5ecc3e1c90e535e P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:21.066053 29678 ts_tablet_manager.cc:1367] T 582128d9dc884a9ab5ecc3e1c90e535e P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 08:12:21.067545 29678 raft_consensus.cc:348] T 582128d9dc884a9ab5ecc3e1c90e535e P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:21.067824 29678 raft_consensus.cc:374] T 582128d9dc884a9ab5ecc3e1c90e535e P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:21.067945 29678 raft_consensus.cc:729] T 582128d9dc884a9ab5ecc3e1c90e535e P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:21.068308 29678 consensus_queue.cc:260] T 582128d9dc884a9ab5ecc3e1c90e535e P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:21.069808 29678 ts_tablet_manager.cc:1398] T 582128d9dc884a9ab5ecc3e1c90e535e P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:21.070124 29678 tablet_bootstrap.cc:492] T 2fbd9559f45645fd8df45491295224ba P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:21.071750 29678 tablet_bootstrap.cc:654] T 2fbd9559f45645fd8df45491295224ba P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:21.073549 29678 tablet_bootstrap.cc:492] T 2fbd9559f45645fd8df45491295224ba P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:21.073683 29678 ts_tablet_manager.cc:1367] T 2fbd9559f45645fd8df45491295224ba P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:12:21.074301 29678 raft_consensus.cc:348] T 2fbd9559f45645fd8df45491295224ba P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:21.074456 29678 raft_consensus.cc:374] T 2fbd9559f45645fd8df45491295224ba P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:21.074543 29678 raft_consensus.cc:729] T 2fbd9559f45645fd8df45491295224ba P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:21.074762 29678 consensus_queue.cc:260] T 2fbd9559f45645fd8df45491295224ba P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:21.075225 29678 ts_tablet_manager.cc:1398] T 2fbd9559f45645fd8df45491295224ba P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:12:21.121557 29615 raft_consensus.cc:482] T b8560620faf54b51ba4c92d8cf95be2d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:21.121738 29615 raft_consensus.cc:504] T b8560620faf54b51ba4c92d8cf95be2d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:21.122550 29615 leader_election.cc:288] T b8560620faf54b51ba4c92d8cf95be2d P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:21.123013 28261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8560620faf54b51ba4c92d8cf95be2d" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:12:21.123117 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8560620faf54b51ba4c92d8cf95be2d" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:12:21.123235 28261 raft_consensus.cc:2454] T b8560620faf54b51ba4c92d8cf95be2d P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:21.123354 28132 raft_consensus.cc:2454] T b8560620faf54b51ba4c92d8cf95be2d P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:21.123732 27937 leader_election.cc:302] T b8560620faf54b51ba4c92d8cf95be2d P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:21.123988 29615 raft_consensus.cc:2781] T b8560620faf54b51ba4c92d8cf95be2d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:21.124099 29615 raft_consensus.cc:482] T b8560620faf54b51ba4c92d8cf95be2d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:21.124184 29615 raft_consensus.cc:3037] T b8560620faf54b51ba4c92d8cf95be2d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:21.126132 29615 raft_consensus.cc:504] T b8560620faf54b51ba4c92d8cf95be2d P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:21.126729 29615 leader_election.cc:288] T b8560620faf54b51ba4c92d8cf95be2d P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:21.127148 28261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8560620faf54b51ba4c92d8cf95be2d" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:12:21.127274 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8560620faf54b51ba4c92d8cf95be2d" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:12:21.127347 28261 raft_consensus.cc:3037] T b8560620faf54b51ba4c92d8cf95be2d P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:21.127470 28132 raft_consensus.cc:3037] T b8560620faf54b51ba4c92d8cf95be2d P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:21.129225 28261 raft_consensus.cc:2454] T b8560620faf54b51ba4c92d8cf95be2d P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:21.129323 28132 raft_consensus.cc:2454] T b8560620faf54b51ba4c92d8cf95be2d P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:21.129859 27938 leader_election.cc:302] T b8560620faf54b51ba4c92d8cf95be2d P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 833582ff68c94269be364ebd187a75db; no voters:
I20240503 08:12:21.130180 29615 raft_consensus.cc:2781] T b8560620faf54b51ba4c92d8cf95be2d P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:21.130563 29615 raft_consensus.cc:686] T b8560620faf54b51ba4c92d8cf95be2d P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Becoming Leader. State: Replica: 833582ff68c94269be364ebd187a75db, State: Running, Role: LEADER
I20240503 08:12:21.130884 29615 consensus_queue.cc:237] T b8560620faf54b51ba4c92d8cf95be2d P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:21.134073 27854 catalog_manager.cc:5484] T b8560620faf54b51ba4c92d8cf95be2d P 833582ff68c94269be364ebd187a75db reported cstate change: term changed from 0 to 1, leader changed from <none> to 833582ff68c94269be364ebd187a75db (127.20.230.65). New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:12:21.180771 29615 raft_consensus.cc:482] T 582128d9dc884a9ab5ecc3e1c90e535e P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:21.181025 29615 raft_consensus.cc:504] T 582128d9dc884a9ab5ecc3e1c90e535e P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:21.181699 29615 leader_election.cc:288] T 582128d9dc884a9ab5ecc3e1c90e535e P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:21.182279 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "582128d9dc884a9ab5ecc3e1c90e535e" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:12:21.182363 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "582128d9dc884a9ab5ecc3e1c90e535e" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:21.182493 28132 raft_consensus.cc:2454] T 582128d9dc884a9ab5ecc3e1c90e535e P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:21.182605 28391 raft_consensus.cc:2454] T 582128d9dc884a9ab5ecc3e1c90e535e P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:21.182910 27938 leader_election.cc:302] T 582128d9dc884a9ab5ecc3e1c90e535e P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 833582ff68c94269be364ebd187a75db; no voters:
I20240503 08:12:21.183205 29615 raft_consensus.cc:2781] T 582128d9dc884a9ab5ecc3e1c90e535e P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:21.183310 29615 raft_consensus.cc:482] T 582128d9dc884a9ab5ecc3e1c90e535e P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:21.183420 29615 raft_consensus.cc:3037] T 582128d9dc884a9ab5ecc3e1c90e535e P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:21.185220 29615 raft_consensus.cc:504] T 582128d9dc884a9ab5ecc3e1c90e535e P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:21.185850 29615 leader_election.cc:288] T 582128d9dc884a9ab5ecc3e1c90e535e P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Requested vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:21.186261 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "582128d9dc884a9ab5ecc3e1c90e535e" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:12:21.186347 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "582128d9dc884a9ab5ecc3e1c90e535e" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:12:21.186460 28132 raft_consensus.cc:3037] T 582128d9dc884a9ab5ecc3e1c90e535e P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:21.186555 28391 raft_consensus.cc:3037] T 582128d9dc884a9ab5ecc3e1c90e535e P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:21.188516 28132 raft_consensus.cc:2454] T 582128d9dc884a9ab5ecc3e1c90e535e P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:21.188668 28391 raft_consensus.cc:2454] T 582128d9dc884a9ab5ecc3e1c90e535e P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:21.189021 27938 leader_election.cc:302] T 582128d9dc884a9ab5ecc3e1c90e535e P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 833582ff68c94269be364ebd187a75db; no voters:
I20240503 08:12:21.189327 29615 raft_consensus.cc:2781] T 582128d9dc884a9ab5ecc3e1c90e535e P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:21.189499 29615 raft_consensus.cc:686] T 582128d9dc884a9ab5ecc3e1c90e535e P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Becoming Leader. State: Replica: 833582ff68c94269be364ebd187a75db, State: Running, Role: LEADER
I20240503 08:12:21.189756 29615 consensus_queue.cc:237] T 582128d9dc884a9ab5ecc3e1c90e535e P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:21.192735 27854 catalog_manager.cc:5484] T 582128d9dc884a9ab5ecc3e1c90e535e P 833582ff68c94269be364ebd187a75db reported cstate change: term changed from 0 to 1, leader changed from <none> to 833582ff68c94269be364ebd187a75db (127.20.230.65). New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:12:21.424949 29488 raft_consensus.cc:482] T 2fbd9559f45645fd8df45491295224ba P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:21.425117 29488 raft_consensus.cc:504] T 2fbd9559f45645fd8df45491295224ba P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:21.425937 29488 leader_election.cc:288] T 2fbd9559f45645fd8df45491295224ba P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:12:21.426276 28261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fbd9559f45645fd8df45491295224ba" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:12:21.426501 28261 raft_consensus.cc:2454] T 2fbd9559f45645fd8df45491295224ba P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:12:21.426990 28327 leader_election.cc:302] T 2fbd9559f45645fd8df45491295224ba P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:21.427255 29488 raft_consensus.cc:2781] T 2fbd9559f45645fd8df45491295224ba P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:21.427371 29488 raft_consensus.cc:482] T 2fbd9559f45645fd8df45491295224ba P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:21.427460 29488 raft_consensus.cc:3037] T 2fbd9559f45645fd8df45491295224ba P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:21.430048 29488 raft_consensus.cc:504] T 2fbd9559f45645fd8df45491295224ba P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:21.430706 29488 leader_election.cc:288] T 2fbd9559f45645fd8df45491295224ba P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:12:21.430902 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fbd9559f45645fd8df45491295224ba" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:12:21.431089 28262 raft_consensus.cc:3037] T 2fbd9559f45645fd8df45491295224ba P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:21.433058 28262 raft_consensus.cc:2454] T 2fbd9559f45645fd8df45491295224ba P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:12:21.433399 28327 leader_election.cc:302] T 2fbd9559f45645fd8df45491295224ba P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:21.433861 29488 raft_consensus.cc:2781] T 2fbd9559f45645fd8df45491295224ba P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:21.433866 28001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fbd9559f45645fd8df45491295224ba" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:12:21.433923 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fbd9559f45645fd8df45491295224ba" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:12:21.434010 29488 raft_consensus.cc:686] T 2fbd9559f45645fd8df45491295224ba P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Becoming Leader. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Running, Role: LEADER
I20240503 08:12:21.434087 28001 raft_consensus.cc:2454] T 2fbd9559f45645fd8df45491295224ba P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:12:21.434491 29488 consensus_queue.cc:237] T 2fbd9559f45645fd8df45491295224ba P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:21.437155 27854 catalog_manager.cc:5484] T 2fbd9559f45645fd8df45491295224ba P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68). New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:12:21.659960 29691 consensus_queue.cc:1035] T b8560620faf54b51ba4c92d8cf95be2d P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:21.665889 29691 consensus_queue.cc:1035] T b8560620faf54b51ba4c92d8cf95be2d P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:21.762251 29691 consensus_queue.cc:1035] T 582128d9dc884a9ab5ecc3e1c90e535e P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:21.766161 29691 consensus_queue.cc:1035] T 582128d9dc884a9ab5ecc3e1c90e535e P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:21.796696 29698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:21.797107 29698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:21.804903 29698 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240503 08:12:21.874123 28003 raft_consensus.cc:3037] T 2fbd9559f45645fd8df45491295224ba P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:21.878028 29488 consensus_queue.cc:1035] T 2fbd9559f45645fd8df45491295224ba P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:21.991783 29488 consensus_queue.cc:1035] T 2fbd9559f45645fd8df45491295224ba P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:22.077962 27938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.66:37143, user_credentials={real_user=slave}} blocked reactor thread for 76419.1us
INSERT report
rows total: 2000
time total: 272.017 ms
time per row: 0.136009 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 08:12:23.088441 27857 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:23.097364 28372 tablet_service.cc:1467] Processing CreateTablet for tablet 599cb18c58a74e93a3e0334cc43a73d4 (DEFAULT_TABLE table=rebalance_test_table_9 [id=eeeb4ab7272f4686b013f5d8f74fc917]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:23.097632 28242 tablet_service.cc:1467] Processing CreateTablet for tablet 599cb18c58a74e93a3e0334cc43a73d4 (DEFAULT_TABLE table=rebalance_test_table_9 [id=eeeb4ab7272f4686b013f5d8f74fc917]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:23.097812 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 599cb18c58a74e93a3e0334cc43a73d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:23.098080 28242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 599cb18c58a74e93a3e0334cc43a73d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:23.098651 28111 tablet_service.cc:1467] Processing CreateTablet for tablet 599cb18c58a74e93a3e0334cc43a73d4 (DEFAULT_TABLE table=rebalance_test_table_9 [id=eeeb4ab7272f4686b013f5d8f74fc917]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:23.099030 28111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 599cb18c58a74e93a3e0334cc43a73d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:23.099749 28241 tablet_service.cc:1467] Processing CreateTablet for tablet c68032ba399547e3bd272f424927fbce (DEFAULT_TABLE table=rebalance_test_table_9 [id=eeeb4ab7272f4686b013f5d8f74fc917]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:23.100077 28241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c68032ba399547e3bd272f424927fbce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:23.102136 28373 tablet_service.cc:1467] Processing CreateTablet for tablet 07866de372e541ec9dca8a3d42f8b0ab (DEFAULT_TABLE table=rebalance_test_table_9 [id=eeeb4ab7272f4686b013f5d8f74fc917]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:23.102540 28373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07866de372e541ec9dca8a3d42f8b0ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:23.102571 27983 tablet_service.cc:1467] Processing CreateTablet for tablet c68032ba399547e3bd272f424927fbce (DEFAULT_TABLE table=rebalance_test_table_9 [id=eeeb4ab7272f4686b013f5d8f74fc917]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:23.102923 27983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c68032ba399547e3bd272f424927fbce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:23.104951 28243 tablet_service.cc:1467] Processing CreateTablet for tablet 07866de372e541ec9dca8a3d42f8b0ab (DEFAULT_TABLE table=rebalance_test_table_9 [id=eeeb4ab7272f4686b013f5d8f74fc917]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:23.105280 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07866de372e541ec9dca8a3d42f8b0ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:23.106009 28112 tablet_service.cc:1467] Processing CreateTablet for tablet 07866de372e541ec9dca8a3d42f8b0ab (DEFAULT_TABLE table=rebalance_test_table_9 [id=eeeb4ab7272f4686b013f5d8f74fc917]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:23.106346 28112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07866de372e541ec9dca8a3d42f8b0ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:23.107863 29760 tablet_bootstrap.cc:492] T 599cb18c58a74e93a3e0334cc43a73d4 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:23.109450 29760 tablet_bootstrap.cc:654] T 599cb18c58a74e93a3e0334cc43a73d4 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:23.109810 28113 tablet_service.cc:1467] Processing CreateTablet for tablet c68032ba399547e3bd272f424927fbce (DEFAULT_TABLE table=rebalance_test_table_9 [id=eeeb4ab7272f4686b013f5d8f74fc917]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:23.110309 28113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c68032ba399547e3bd272f424927fbce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:23.120700 29761 tablet_bootstrap.cc:492] T 07866de372e541ec9dca8a3d42f8b0ab P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:23.121002 29760 tablet_bootstrap.cc:492] T 599cb18c58a74e93a3e0334cc43a73d4 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:23.121150 29760 ts_tablet_manager.cc:1367] T 599cb18c58a74e93a3e0334cc43a73d4 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20240503 08:12:23.122334 29760 raft_consensus.cc:348] T 599cb18c58a74e93a3e0334cc43a73d4 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:23.122435 29761 tablet_bootstrap.cc:654] T 07866de372e541ec9dca8a3d42f8b0ab P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:23.122491 29760 raft_consensus.cc:374] T 599cb18c58a74e93a3e0334cc43a73d4 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:23.122576 29760 raft_consensus.cc:729] T 599cb18c58a74e93a3e0334cc43a73d4 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:23.122781 29760 consensus_queue.cc:260] T 599cb18c58a74e93a3e0334cc43a73d4 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:23.123263 29760 ts_tablet_manager.cc:1398] T 599cb18c58a74e93a3e0334cc43a73d4 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:12:23.123512 29760 tablet_bootstrap.cc:492] T 07866de372e541ec9dca8a3d42f8b0ab P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:23.125067 29763 tablet_bootstrap.cc:492] T c68032ba399547e3bd272f424927fbce P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:23.127573 29763 tablet_bootstrap.cc:654] T c68032ba399547e3bd272f424927fbce P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:23.128856 29759 tablet_bootstrap.cc:492] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:23.129642 29760 tablet_bootstrap.cc:654] T 07866de372e541ec9dca8a3d42f8b0ab P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:23.130643 29761 tablet_bootstrap.cc:492] T 07866de372e541ec9dca8a3d42f8b0ab P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:23.130656 29759 tablet_bootstrap.cc:654] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:23.130847 29761 ts_tablet_manager.cc:1367] T 07866de372e541ec9dca8a3d42f8b0ab P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.005s
I20240503 08:12:23.131676 29761 raft_consensus.cc:348] T 07866de372e541ec9dca8a3d42f8b0ab P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:23.131956 29761 raft_consensus.cc:374] T 07866de372e541ec9dca8a3d42f8b0ab P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:23.132099 29761 raft_consensus.cc:729] T 07866de372e541ec9dca8a3d42f8b0ab P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:23.132447 29761 consensus_queue.cc:260] T 07866de372e541ec9dca8a3d42f8b0ab P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:23.133105 29761 ts_tablet_manager.cc:1398] T 07866de372e541ec9dca8a3d42f8b0ab P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:23.133433 29761 tablet_bootstrap.cc:492] T 599cb18c58a74e93a3e0334cc43a73d4 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:23.133901 29763 tablet_bootstrap.cc:492] T c68032ba399547e3bd272f424927fbce P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:23.134083 29763 ts_tablet_manager.cc:1367] T c68032ba399547e3bd272f424927fbce P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240503 08:12:23.135046 29761 tablet_bootstrap.cc:654] T 599cb18c58a74e93a3e0334cc43a73d4 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:23.135385 29763 raft_consensus.cc:348] T c68032ba399547e3bd272f424927fbce P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:23.135659 29763 raft_consensus.cc:374] T c68032ba399547e3bd272f424927fbce P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:23.135787 29763 raft_consensus.cc:729] T c68032ba399547e3bd272f424927fbce P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:23.136058 29763 consensus_queue.cc:260] T c68032ba399547e3bd272f424927fbce P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:23.136801 29763 ts_tablet_manager.cc:1398] T c68032ba399547e3bd272f424927fbce P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:12:23.139896 29759 tablet_bootstrap.cc:492] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:23.140384 29759 ts_tablet_manager.cc:1367] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.006s
I20240503 08:12:23.141378 29759 raft_consensus.cc:348] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:23.141530 29760 tablet_bootstrap.cc:492] T 07866de372e541ec9dca8a3d42f8b0ab P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:23.141600 29759 raft_consensus.cc:374] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:23.141669 29760 ts_tablet_manager.cc:1367] T 07866de372e541ec9dca8a3d42f8b0ab P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.008s
I20240503 08:12:23.142071 29759 raft_consensus.cc:729] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:23.142325 29759 consensus_queue.cc:260] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:23.142398 29760 raft_consensus.cc:348] T 07866de372e541ec9dca8a3d42f8b0ab P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:23.142557 29760 raft_consensus.cc:374] T 07866de372e541ec9dca8a3d42f8b0ab P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:23.142648 29760 raft_consensus.cc:729] T 07866de372e541ec9dca8a3d42f8b0ab P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:23.142889 29761 tablet_bootstrap.cc:492] T 599cb18c58a74e93a3e0334cc43a73d4 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:23.142850 29760 consensus_queue.cc:260] T 07866de372e541ec9dca8a3d42f8b0ab P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:23.143165 29761 ts_tablet_manager.cc:1367] T 599cb18c58a74e93a3e0334cc43a73d4 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20240503 08:12:23.143326 29760 ts_tablet_manager.cc:1398] T 07866de372e541ec9dca8a3d42f8b0ab P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:12:23.143950 29759 ts_tablet_manager.cc:1398] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:12:23.144246 29759 tablet_bootstrap.cc:492] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:23.144210 29761 raft_consensus.cc:348] T 599cb18c58a74e93a3e0334cc43a73d4 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:23.144483 29761 raft_consensus.cc:374] T 599cb18c58a74e93a3e0334cc43a73d4 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:23.144593 29761 raft_consensus.cc:729] T 599cb18c58a74e93a3e0334cc43a73d4 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:23.144883 29761 consensus_queue.cc:260] T 599cb18c58a74e93a3e0334cc43a73d4 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:23.145035 29760 tablet_bootstrap.cc:492] T c68032ba399547e3bd272f424927fbce P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:23.145694 29759 tablet_bootstrap.cc:654] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:23.147931 29761 ts_tablet_manager.cc:1398] T 599cb18c58a74e93a3e0334cc43a73d4 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 08:12:23.148591 29760 tablet_bootstrap.cc:654] T c68032ba399547e3bd272f424927fbce P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:23.148952 29761 tablet_bootstrap.cc:492] T c68032ba399547e3bd272f424927fbce P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:23.150521 29761 tablet_bootstrap.cc:654] T c68032ba399547e3bd272f424927fbce P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:23.154518 29759 tablet_bootstrap.cc:492] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:23.154735 29759 ts_tablet_manager.cc:1367] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20240503 08:12:23.155467 29759 raft_consensus.cc:348] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:23.155774 29759 raft_consensus.cc:374] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:23.155885 29759 raft_consensus.cc:729] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:23.157577 29734 raft_consensus.cc:482] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:23.157728 29734 raft_consensus.cc:504] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:23.158421 29734 leader_election.cc:288] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:23.158990 28133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599cb18c58a74e93a3e0334cc43a73d4" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:12:23.159027 29759 consensus_queue.cc:260] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:23.159191 28133 raft_consensus.cc:2454] T 599cb18c58a74e93a3e0334cc43a73d4 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:12:23.160091 28328 leader_election.cc:302] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:23.161010 29734 raft_consensus.cc:2781] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:23.161123 29734 raft_consensus.cc:482] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:23.161211 29734 raft_consensus.cc:3037] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:23.163671 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599cb18c58a74e93a3e0334cc43a73d4" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:12:23.163895 28262 raft_consensus.cc:2454] T 599cb18c58a74e93a3e0334cc43a73d4 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:12:23.164086 29734 raft_consensus.cc:504] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:23.165071 28133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599cb18c58a74e93a3e0334cc43a73d4" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:12:23.165256 28133 raft_consensus.cc:3037] T 599cb18c58a74e93a3e0334cc43a73d4 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:23.166548 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599cb18c58a74e93a3e0334cc43a73d4" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:12:23.166730 28262 raft_consensus.cc:3037] T 599cb18c58a74e93a3e0334cc43a73d4 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:23.167405 29734 leader_election.cc:288] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:23.167598 28133 raft_consensus.cc:2454] T 599cb18c58a74e93a3e0334cc43a73d4 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:12:23.169262 28262 raft_consensus.cc:2454] T 599cb18c58a74e93a3e0334cc43a73d4 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:12:23.170132 28327 leader_election.cc:302] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:23.171432 29760 tablet_bootstrap.cc:492] T c68032ba399547e3bd272f424927fbce P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:23.171605 29760 ts_tablet_manager.cc:1367] T c68032ba399547e3bd272f424927fbce P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.010s
I20240503 08:12:23.171715 29734 raft_consensus.cc:2781] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:23.172349 29760 raft_consensus.cc:348] T c68032ba399547e3bd272f424927fbce P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:23.172585 29760 raft_consensus.cc:374] T c68032ba399547e3bd272f424927fbce P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:23.172894 29760 raft_consensus.cc:729] T c68032ba399547e3bd272f424927fbce P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:23.173138 29760 consensus_queue.cc:260] T c68032ba399547e3bd272f424927fbce P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:23.174458 29760 ts_tablet_manager.cc:1398] T c68032ba399547e3bd272f424927fbce P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 08:12:23.180604 29759 ts_tablet_manager.cc:1398] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.026s user 0.000s sys 0.002s
I20240503 08:12:23.184180 29734 raft_consensus.cc:686] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Becoming Leader. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Running, Role: LEADER
I20240503 08:12:23.184445 29761 tablet_bootstrap.cc:492] T c68032ba399547e3bd272f424927fbce P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:23.184432 29734 consensus_queue.cc:237] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:23.184564 29761 ts_tablet_manager.cc:1367] T c68032ba399547e3bd272f424927fbce P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.036s user 0.015s sys 0.019s
I20240503 08:12:23.185109 29761 raft_consensus.cc:348] T c68032ba399547e3bd272f424927fbce P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:23.185297 29761 raft_consensus.cc:374] T c68032ba399547e3bd272f424927fbce P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:23.185447 29761 raft_consensus.cc:729] T c68032ba399547e3bd272f424927fbce P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:23.185688 29761 consensus_queue.cc:260] T c68032ba399547e3bd272f424927fbce P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:23.186208 29761 ts_tablet_manager.cc:1398] T c68032ba399547e3bd272f424927fbce P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:23.186993 27857 catalog_manager.cc:5484] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68). New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:12:23.353612 29587 raft_consensus.cc:482] T 07866de372e541ec9dca8a3d42f8b0ab P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:23.353817 29587 raft_consensus.cc:504] T 07866de372e541ec9dca8a3d42f8b0ab P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:23.354409 29587 leader_election.cc:288] T 07866de372e541ec9dca8a3d42f8b0ab P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:23.354810 28133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07866de372e541ec9dca8a3d42f8b0ab" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:12:23.354806 28393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07866de372e541ec9dca8a3d42f8b0ab" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:23.355017 28133 raft_consensus.cc:2454] T 07866de372e541ec9dca8a3d42f8b0ab P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:12:23.355008 28393 raft_consensus.cc:2454] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:12:23.355363 28198 leader_election.cc:302] T 07866de372e541ec9dca8a3d42f8b0ab P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:23.355598 29587 raft_consensus.cc:2781] T 07866de372e541ec9dca8a3d42f8b0ab P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:23.355707 29587 raft_consensus.cc:482] T 07866de372e541ec9dca8a3d42f8b0ab P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:23.355798 29587 raft_consensus.cc:3037] T 07866de372e541ec9dca8a3d42f8b0ab P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:23.357676 29587 raft_consensus.cc:504] T 07866de372e541ec9dca8a3d42f8b0ab P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:23.358215 29587 leader_election.cc:288] T 07866de372e541ec9dca8a3d42f8b0ab P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Requested vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:23.358587 28133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07866de372e541ec9dca8a3d42f8b0ab" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:12:23.358673 28393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07866de372e541ec9dca8a3d42f8b0ab" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:12:23.358777 28133 raft_consensus.cc:3037] T 07866de372e541ec9dca8a3d42f8b0ab P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:23.358907 28393 raft_consensus.cc:3037] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:23.360577 28133 raft_consensus.cc:2454] T 07866de372e541ec9dca8a3d42f8b0ab P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:23.360663 28393 raft_consensus.cc:2454] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:23.360911 28198 leader_election.cc:302] T 07866de372e541ec9dca8a3d42f8b0ab P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:23.361284 29587 raft_consensus.cc:2781] T 07866de372e541ec9dca8a3d42f8b0ab P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:23.361434 29587 raft_consensus.cc:686] T 07866de372e541ec9dca8a3d42f8b0ab P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Becoming Leader. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Running, Role: LEADER
I20240503 08:12:23.361861 29587 consensus_queue.cc:237] T 07866de372e541ec9dca8a3d42f8b0ab P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:23.364941 27854 catalog_manager.cc:5484] T 07866de372e541ec9dca8a3d42f8b0ab P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: term changed from 0 to 1, leader changed from <none> to aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67). New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:12:23.385730 29742 raft_consensus.cc:482] T c68032ba399547e3bd272f424927fbce P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:23.385892 29742 raft_consensus.cc:504] T c68032ba399547e3bd272f424927fbce P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:23.386566 29742 leader_election.cc:288] T c68032ba399547e3bd272f424927fbce P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:23.386890 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c68032ba399547e3bd272f424927fbce" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:12:23.387037 28133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c68032ba399547e3bd272f424927fbce" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:12:23.387115 28262 raft_consensus.cc:2454] T c68032ba399547e3bd272f424927fbce P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:23.387238 28133 raft_consensus.cc:2454] T c68032ba399547e3bd272f424927fbce P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:23.387502 27937 leader_election.cc:302] T c68032ba399547e3bd272f424927fbce P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:23.387807 29778 raft_consensus.cc:2781] T c68032ba399547e3bd272f424927fbce P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:23.387899 29778 raft_consensus.cc:482] T c68032ba399547e3bd272f424927fbce P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:23.387992 29778 raft_consensus.cc:3037] T c68032ba399547e3bd272f424927fbce P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:23.389757 29778 raft_consensus.cc:504] T c68032ba399547e3bd272f424927fbce P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:23.390301 29778 leader_election.cc:288] T c68032ba399547e3bd272f424927fbce P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:23.390600 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c68032ba399547e3bd272f424927fbce" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:12:23.390754 28262 raft_consensus.cc:3037] T c68032ba399547e3bd272f424927fbce P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:23.391172 28133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c68032ba399547e3bd272f424927fbce" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:12:23.391350 28133 raft_consensus.cc:3037] T c68032ba399547e3bd272f424927fbce P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:23.393347 28262 raft_consensus.cc:2454] T c68032ba399547e3bd272f424927fbce P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:23.393666 27937 leader_election.cc:302] T c68032ba399547e3bd272f424927fbce P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:23.393956 29778 raft_consensus.cc:2781] T c68032ba399547e3bd272f424927fbce P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:23.393951 28133 raft_consensus.cc:2454] T c68032ba399547e3bd272f424927fbce P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:23.394105 29778 raft_consensus.cc:686] T c68032ba399547e3bd272f424927fbce P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Becoming Leader. State: Replica: 833582ff68c94269be364ebd187a75db, State: Running, Role: LEADER
I20240503 08:12:23.394361 29778 consensus_queue.cc:237] T c68032ba399547e3bd272f424927fbce P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:23.397182 27854 catalog_manager.cc:5484] T c68032ba399547e3bd272f424927fbce P 833582ff68c94269be364ebd187a75db reported cstate change: term changed from 0 to 1, leader changed from <none> to 833582ff68c94269be364ebd187a75db (127.20.230.65). New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:12:23.607892 29734 consensus_queue.cc:1035] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:23.614552 29734 consensus_queue.cc:1035] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:23.725100 29780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:23.725365 29780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:23.732565 29780 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240503 08:12:23.830514 29770 consensus_queue.cc:1035] T 07866de372e541ec9dca8a3d42f8b0ab P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:23.860720 29778 consensus_queue.cc:1035] T c68032ba399547e3bd272f424927fbce P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:23.874357 29742 consensus_queue.cc:1035] T c68032ba399547e3bd272f424927fbce P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:23.893453 29823 consensus_queue.cc:1035] T 07866de372e541ec9dca8a3d42f8b0ab P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 215.177 ms
time per row: 0.107589 ms
I20240503 08:12:24.087393 27854 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:24.096026 28372 tablet_service.cc:1467] Processing CreateTablet for tablet cdc2d9cfc64d42ff9a9b30bb04f4433e (DEFAULT_TABLE table=rebalance_test_table_10 [id=2d7e406e0e184a5faee52035a9d992d7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:24.096483 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdc2d9cfc64d42ff9a9b30bb04f4433e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:24.096809 28373 tablet_service.cc:1467] Processing CreateTablet for tablet 5c293c45cb5e45bebc486f0418a612ed (DEFAULT_TABLE table=rebalance_test_table_10 [id=2d7e406e0e184a5faee52035a9d992d7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:24.097182 28373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c293c45cb5e45bebc486f0418a612ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:24.097544 27983 tablet_service.cc:1467] Processing CreateTablet for tablet cdc2d9cfc64d42ff9a9b30bb04f4433e (DEFAULT_TABLE table=rebalance_test_table_10 [id=2d7e406e0e184a5faee52035a9d992d7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:24.097568 27982 tablet_service.cc:1467] Processing CreateTablet for tablet f9d5fb461eca4d7ca0b2f0e81e9823bc (DEFAULT_TABLE table=rebalance_test_table_10 [id=2d7e406e0e184a5faee52035a9d992d7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:24.098003 27983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdc2d9cfc64d42ff9a9b30bb04f4433e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:24.098076 27981 tablet_service.cc:1467] Processing CreateTablet for tablet 5c293c45cb5e45bebc486f0418a612ed (DEFAULT_TABLE table=rebalance_test_table_10 [id=2d7e406e0e184a5faee52035a9d992d7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:24.098434 27981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c293c45cb5e45bebc486f0418a612ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:24.102370 28371 tablet_service.cc:1467] Processing CreateTablet for tablet f9d5fb461eca4d7ca0b2f0e81e9823bc (DEFAULT_TABLE table=rebalance_test_table_10 [id=2d7e406e0e184a5faee52035a9d992d7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:24.102463 28112 tablet_service.cc:1467] Processing CreateTablet for tablet f9d5fb461eca4d7ca0b2f0e81e9823bc (DEFAULT_TABLE table=rebalance_test_table_10 [id=2d7e406e0e184a5faee52035a9d992d7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:24.102694 28371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9d5fb461eca4d7ca0b2f0e81e9823bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:24.102824 28112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9d5fb461eca4d7ca0b2f0e81e9823bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:24.103276 28241 tablet_service.cc:1467] Processing CreateTablet for tablet 5c293c45cb5e45bebc486f0418a612ed (DEFAULT_TABLE table=rebalance_test_table_10 [id=2d7e406e0e184a5faee52035a9d992d7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:24.103642 28241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c293c45cb5e45bebc486f0418a612ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:24.104051 27982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9d5fb461eca4d7ca0b2f0e81e9823bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:24.107200 28113 tablet_service.cc:1467] Processing CreateTablet for tablet cdc2d9cfc64d42ff9a9b30bb04f4433e (DEFAULT_TABLE table=rebalance_test_table_10 [id=2d7e406e0e184a5faee52035a9d992d7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:24.107596 28113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdc2d9cfc64d42ff9a9b30bb04f4433e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:24.114176 29847 tablet_bootstrap.cc:492] T f9d5fb461eca4d7ca0b2f0e81e9823bc P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:24.114333 29848 tablet_bootstrap.cc:492] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:24.115926 29847 tablet_bootstrap.cc:654] T f9d5fb461eca4d7ca0b2f0e81e9823bc P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:24.115926 29848 tablet_bootstrap.cc:654] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:24.122778 29851 tablet_bootstrap.cc:492] T 5c293c45cb5e45bebc486f0418a612ed P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:24.125941 29851 tablet_bootstrap.cc:654] T 5c293c45cb5e45bebc486f0418a612ed P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:24.126523 29846 tablet_bootstrap.cc:492] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:24.127797 29847 tablet_bootstrap.cc:492] T f9d5fb461eca4d7ca0b2f0e81e9823bc P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:24.127957 29846 tablet_bootstrap.cc:654] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:24.128093 29847 ts_tablet_manager.cc:1367] T f9d5fb461eca4d7ca0b2f0e81e9823bc P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20240503 08:12:24.128943 29847 raft_consensus.cc:348] T f9d5fb461eca4d7ca0b2f0e81e9823bc P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:24.129165 29847 raft_consensus.cc:374] T f9d5fb461eca4d7ca0b2f0e81e9823bc P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:24.129293 29847 raft_consensus.cc:729] T f9d5fb461eca4d7ca0b2f0e81e9823bc P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:24.129613 29847 consensus_queue.cc:260] T f9d5fb461eca4d7ca0b2f0e81e9823bc P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:24.130313 29847 ts_tablet_manager.cc:1398] T f9d5fb461eca4d7ca0b2f0e81e9823bc P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:24.130643 29847 tablet_bootstrap.cc:492] T 5c293c45cb5e45bebc486f0418a612ed P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:24.130671 29848 tablet_bootstrap.cc:492] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:24.130780 29848 ts_tablet_manager.cc:1367] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.011s
I20240503 08:12:24.131311 29846 tablet_bootstrap.cc:492] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:24.131445 29846 ts_tablet_manager.cc:1367] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:12:24.131620 29848 raft_consensus.cc:348] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:24.131875 29848 raft_consensus.cc:374] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:24.132012 29848 raft_consensus.cc:729] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:24.132153 29847 tablet_bootstrap.cc:654] T 5c293c45cb5e45bebc486f0418a612ed P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:24.132192 29846 raft_consensus.cc:348] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:24.132310 29848 consensus_queue.cc:260] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:24.132414 29846 raft_consensus.cc:374] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:24.132515 29846 raft_consensus.cc:729] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:24.133666 29848 ts_tablet_manager.cc:1398] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 08:12:24.133996 29848 tablet_bootstrap.cc:492] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:24.135771 29848 tablet_bootstrap.cc:654] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:24.135735 29846 consensus_queue.cc:260] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:24.136855 29851 tablet_bootstrap.cc:492] T 5c293c45cb5e45bebc486f0418a612ed P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:24.137035 29851 ts_tablet_manager.cc:1367] T 5c293c45cb5e45bebc486f0418a612ed P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.003s
I20240503 08:12:24.137847 29851 raft_consensus.cc:348] T 5c293c45cb5e45bebc486f0418a612ed P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:24.138094 29851 raft_consensus.cc:374] T 5c293c45cb5e45bebc486f0418a612ed P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:24.138223 29851 raft_consensus.cc:729] T 5c293c45cb5e45bebc486f0418a612ed P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:24.138485 29851 consensus_queue.cc:260] T 5c293c45cb5e45bebc486f0418a612ed P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:24.139367 29851 ts_tablet_manager.cc:1398] T 5c293c45cb5e45bebc486f0418a612ed P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:24.139765 29846 ts_tablet_manager.cc:1398] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 08:12:24.140096 29846 tablet_bootstrap.cc:492] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:24.141220 29846 tablet_bootstrap.cc:654] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:24.143049 29846 tablet_bootstrap.cc:492] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:24.143232 29846 ts_tablet_manager.cc:1367] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:12:24.144042 29846 raft_consensus.cc:348] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:24.144289 29846 raft_consensus.cc:374] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:24.144404 29846 raft_consensus.cc:729] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:24.144713 29846 consensus_queue.cc:260] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:24.144945 29848 tablet_bootstrap.cc:492] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:24.145131 29848 ts_tablet_manager.cc:1367] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.009s
I20240503 08:12:24.145233 29846 ts_tablet_manager.cc:1398] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:12:24.145511 29846 tablet_bootstrap.cc:492] T 5c293c45cb5e45bebc486f0418a612ed P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:24.145835 29848 raft_consensus.cc:348] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:24.146092 29848 raft_consensus.cc:374] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:24.146209 29848 raft_consensus.cc:729] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:24.146497 29848 consensus_queue.cc:260] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:24.146778 29847 tablet_bootstrap.cc:492] T 5c293c45cb5e45bebc486f0418a612ed P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:24.146921 29847 ts_tablet_manager.cc:1367] T 5c293c45cb5e45bebc486f0418a612ed P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.000s
I20240503 08:12:24.146962 29846 tablet_bootstrap.cc:654] T 5c293c45cb5e45bebc486f0418a612ed P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:24.147781 29848 ts_tablet_manager.cc:1398] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:24.147694 29847 raft_consensus.cc:348] T 5c293c45cb5e45bebc486f0418a612ed P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:24.147886 29847 raft_consensus.cc:374] T 5c293c45cb5e45bebc486f0418a612ed P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:24.147961 29847 raft_consensus.cc:729] T 5c293c45cb5e45bebc486f0418a612ed P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:24.148192 29847 consensus_queue.cc:260] T 5c293c45cb5e45bebc486f0418a612ed P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:24.148741 29847 ts_tablet_manager.cc:1398] T 5c293c45cb5e45bebc486f0418a612ed P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:24.149015 29847 tablet_bootstrap.cc:492] T cdc2d9cfc64d42ff9a9b30bb04f4433e P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:24.150249 29847 tablet_bootstrap.cc:654] T cdc2d9cfc64d42ff9a9b30bb04f4433e P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:24.152082 29847 tablet_bootstrap.cc:492] T cdc2d9cfc64d42ff9a9b30bb04f4433e P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:24.152201 29847 ts_tablet_manager.cc:1367] T cdc2d9cfc64d42ff9a9b30bb04f4433e P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:24.152812 29847 raft_consensus.cc:348] T cdc2d9cfc64d42ff9a9b30bb04f4433e P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:24.152946 29847 raft_consensus.cc:374] T cdc2d9cfc64d42ff9a9b30bb04f4433e P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:24.153033 29847 raft_consensus.cc:729] T cdc2d9cfc64d42ff9a9b30bb04f4433e P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:24.153220 29847 consensus_queue.cc:260] T cdc2d9cfc64d42ff9a9b30bb04f4433e P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:24.153656 29847 ts_tablet_manager.cc:1398] T cdc2d9cfc64d42ff9a9b30bb04f4433e P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 08:12:24.154670 29846 tablet_bootstrap.cc:492] T 5c293c45cb5e45bebc486f0418a612ed P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:24.154788 29846 ts_tablet_manager.cc:1367] T 5c293c45cb5e45bebc486f0418a612ed P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.008s
I20240503 08:12:24.155443 29846 raft_consensus.cc:348] T 5c293c45cb5e45bebc486f0418a612ed P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:24.155622 29846 raft_consensus.cc:374] T 5c293c45cb5e45bebc486f0418a612ed P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:24.155712 29846 raft_consensus.cc:729] T 5c293c45cb5e45bebc486f0418a612ed P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:24.155902 29846 consensus_queue.cc:260] T 5c293c45cb5e45bebc486f0418a612ed P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:24.156332 29846 ts_tablet_manager.cc:1398] T 5c293c45cb5e45bebc486f0418a612ed P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 08:12:24.157886 29742 raft_consensus.cc:482] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:24.157991 29742 raft_consensus.cc:504] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:24.158699 29742 leader_election.cc:288] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:24.158943 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc2d9cfc64d42ff9a9b30bb04f4433e" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:24.159106 28391 raft_consensus.cc:2454] T cdc2d9cfc64d42ff9a9b30bb04f4433e P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:24.159418 27939 leader_election.cc:302] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:24.159663 29742 raft_consensus.cc:2781] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:24.159662 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc2d9cfc64d42ff9a9b30bb04f4433e" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:12:24.159785 29742 raft_consensus.cc:482] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:24.159842 28132 raft_consensus.cc:2454] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:24.159874 29742 raft_consensus.cc:3037] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:24.161615 29742 raft_consensus.cc:504] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:24.162165 29742 leader_election.cc:288] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Requested vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:24.162473 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc2d9cfc64d42ff9a9b30bb04f4433e" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:12:24.162495 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc2d9cfc64d42ff9a9b30bb04f4433e" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:12:24.162616 28132 raft_consensus.cc:3037] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:24.162619 28391 raft_consensus.cc:3037] T cdc2d9cfc64d42ff9a9b30bb04f4433e P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:24.164395 28391 raft_consensus.cc:2454] T cdc2d9cfc64d42ff9a9b30bb04f4433e P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:24.164881 27939 leader_election.cc:302] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:24.164979 28132 raft_consensus.cc:2454] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:24.165135 29742 raft_consensus.cc:2781] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:24.165298 29742 raft_consensus.cc:686] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Becoming Leader. State: Replica: 833582ff68c94269be364ebd187a75db, State: Running, Role: LEADER
I20240503 08:12:24.165551 29742 consensus_queue.cc:237] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:24.168128 27855 catalog_manager.cc:5484] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 833582ff68c94269be364ebd187a75db reported cstate change: term changed from 0 to 1, leader changed from <none> to 833582ff68c94269be364ebd187a75db (127.20.230.65). New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:12:24.252244 29610 raft_consensus.cc:482] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:24.252413 29610 raft_consensus.cc:504] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:24.253216 29610 leader_election.cc:288] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:24.253630 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9d5fb461eca4d7ca0b2f0e81e9823bc" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:12:24.253703 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9d5fb461eca4d7ca0b2f0e81e9823bc" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:24.253901 28003 raft_consensus.cc:2454] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:24.253960 28391 raft_consensus.cc:2454] T f9d5fb461eca4d7ca0b2f0e81e9823bc P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:24.254338 28069 leader_election.cc:302] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:24.254593 29610 raft_consensus.cc:2781] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:24.254707 29610 raft_consensus.cc:482] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:24.254789 29610 raft_consensus.cc:3037] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:24.256659 29610 raft_consensus.cc:504] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:24.257119 29610 leader_election.cc:288] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Requested vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:24.257515 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9d5fb461eca4d7ca0b2f0e81e9823bc" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:12:24.257527 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9d5fb461eca4d7ca0b2f0e81e9823bc" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:12:24.257676 28003 raft_consensus.cc:3037] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:24.257692 28391 raft_consensus.cc:3037] T f9d5fb461eca4d7ca0b2f0e81e9823bc P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:24.259442 28003 raft_consensus.cc:2454] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:12:24.259745 28391 raft_consensus.cc:2454] T f9d5fb461eca4d7ca0b2f0e81e9823bc P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:12:24.259770 28069 leader_election.cc:302] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 833582ff68c94269be364ebd187a75db; no voters:
I20240503 08:12:24.260027 29610 raft_consensus.cc:2781] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:24.260432 29610 raft_consensus.cc:686] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Becoming Leader. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Running, Role: LEADER
I20240503 08:12:24.260689 29610 consensus_queue.cc:237] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:24.263473 27855 catalog_manager.cc:5484] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 1b411c6bb25549368041773d7b710d48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b411c6bb25549368041773d7b710d48 (127.20.230.66). New cstate: current_term: 1 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:12:24.345665 29840 raft_consensus.cc:482] T 5c293c45cb5e45bebc486f0418a612ed P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:24.345845 29840 raft_consensus.cc:504] T 5c293c45cb5e45bebc486f0418a612ed P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:24.346612 29840 leader_election.cc:288] T 5c293c45cb5e45bebc486f0418a612ed P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:12:24.347136 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c293c45cb5e45bebc486f0418a612ed" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:12:24.347226 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c293c45cb5e45bebc486f0418a612ed" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:12:24.347385 28262 raft_consensus.cc:2454] T 5c293c45cb5e45bebc486f0418a612ed P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:12:24.347421 28003 raft_consensus.cc:2454] T 5c293c45cb5e45bebc486f0418a612ed P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:12:24.347769 28329 leader_election.cc:302] T 5c293c45cb5e45bebc486f0418a612ed P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:24.348002 29840 raft_consensus.cc:2781] T 5c293c45cb5e45bebc486f0418a612ed P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:24.348110 29840 raft_consensus.cc:482] T 5c293c45cb5e45bebc486f0418a612ed P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:24.348176 29840 raft_consensus.cc:3037] T 5c293c45cb5e45bebc486f0418a612ed P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:24.350273 29840 raft_consensus.cc:504] T 5c293c45cb5e45bebc486f0418a612ed P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:24.350723 29840 leader_election.cc:288] T 5c293c45cb5e45bebc486f0418a612ed P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:12:24.351115 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c293c45cb5e45bebc486f0418a612ed" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:12:24.351208 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c293c45cb5e45bebc486f0418a612ed" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:12:24.351326 28262 raft_consensus.cc:3037] T 5c293c45cb5e45bebc486f0418a612ed P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:24.351414 28003 raft_consensus.cc:3037] T 5c293c45cb5e45bebc486f0418a612ed P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:24.354069 28003 raft_consensus.cc:2454] T 5c293c45cb5e45bebc486f0418a612ed P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:12:24.354104 28262 raft_consensus.cc:2454] T 5c293c45cb5e45bebc486f0418a612ed P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:12:24.354434 28329 leader_election.cc:302] T 5c293c45cb5e45bebc486f0418a612ed P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:24.354710 29840 raft_consensus.cc:2781] T 5c293c45cb5e45bebc486f0418a612ed P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:24.354843 29840 raft_consensus.cc:686] T 5c293c45cb5e45bebc486f0418a612ed P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Becoming Leader. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Running, Role: LEADER
I20240503 08:12:24.355075 29840 consensus_queue.cc:237] T 5c293c45cb5e45bebc486f0418a612ed P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:24.357828 27855 catalog_manager.cc:5484] T 5c293c45cb5e45bebc486f0418a612ed P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68). New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:12:24.628602 29742 consensus_queue.cc:1035] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:24.647462 29833 consensus_queue.cc:1035] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:24.695006 29610 consensus_queue.cc:1035] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:24.705476 29610 consensus_queue.cc:1035] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:24.744614 29863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:24.744954 29863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:24.752431 29863 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
I20240503 08:12:24.933701 28003 raft_consensus.cc:1261] T 5c293c45cb5e45bebc486f0418a612ed P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:24.963760 29908 consensus_queue.cc:1035] T 5c293c45cb5e45bebc486f0418a612ed P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:24.965116 28262 raft_consensus.cc:1261] T 5c293c45cb5e45bebc486f0418a612ed P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:24.971169 29840 consensus_queue.cc:1035] T 5c293c45cb5e45bebc486f0418a612ed P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 297.494 ms
time per row: 0.148747 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 08:12:26.055805 27860 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:26.064664 28113 tablet_service.cc:1467] Processing CreateTablet for tablet 3d0498256b5f4b8fb1681759429f10ed (DEFAULT_TABLE table=rebalance_test_table_11 [id=e00af55ec1c448bdb2c617816b10a90f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:26.064911 27983 tablet_service.cc:1467] Processing CreateTablet for tablet 3d0498256b5f4b8fb1681759429f10ed (DEFAULT_TABLE table=rebalance_test_table_11 [id=e00af55ec1c448bdb2c617816b10a90f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:26.065121 28113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d0498256b5f4b8fb1681759429f10ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:26.065337 27983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d0498256b5f4b8fb1681759429f10ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:26.065908 28372 tablet_service.cc:1467] Processing CreateTablet for tablet 3042212758bf4aef8be5c5d12d10113c (DEFAULT_TABLE table=rebalance_test_table_11 [id=e00af55ec1c448bdb2c617816b10a90f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:26.066143 28243 tablet_service.cc:1467] Processing CreateTablet for tablet 3042212758bf4aef8be5c5d12d10113c (DEFAULT_TABLE table=rebalance_test_table_11 [id=e00af55ec1c448bdb2c617816b10a90f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:26.066277 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3042212758bf4aef8be5c5d12d10113c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:26.066504 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3042212758bf4aef8be5c5d12d10113c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:26.070099 28241 tablet_service.cc:1467] Processing CreateTablet for tablet 3d0498256b5f4b8fb1681759429f10ed (DEFAULT_TABLE table=rebalance_test_table_11 [id=e00af55ec1c448bdb2c617816b10a90f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:26.070420 28241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d0498256b5f4b8fb1681759429f10ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:26.071146 27981 tablet_service.cc:1467] Processing CreateTablet for tablet 3042212758bf4aef8be5c5d12d10113c (DEFAULT_TABLE table=rebalance_test_table_11 [id=e00af55ec1c448bdb2c617816b10a90f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:26.071482 27981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3042212758bf4aef8be5c5d12d10113c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:26.076584 28242 tablet_service.cc:1467] Processing CreateTablet for tablet 8fc87f7f3c724cd4a270583ddebd87aa (DEFAULT_TABLE table=rebalance_test_table_11 [id=e00af55ec1c448bdb2c617816b10a90f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:26.076858 29936 tablet_bootstrap.cc:492] T 3d0498256b5f4b8fb1681759429f10ed P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:26.076906 28242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fc87f7f3c724cd4a270583ddebd87aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:26.077101 29935 tablet_bootstrap.cc:492] T 3d0498256b5f4b8fb1681759429f10ed P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:26.078816 29936 tablet_bootstrap.cc:654] T 3d0498256b5f4b8fb1681759429f10ed P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:26.079386 29938 tablet_bootstrap.cc:492] T 3d0498256b5f4b8fb1681759429f10ed P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:26.079550 29935 tablet_bootstrap.cc:654] T 3d0498256b5f4b8fb1681759429f10ed P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:26.080332 29937 tablet_bootstrap.cc:492] T 3042212758bf4aef8be5c5d12d10113c P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:26.081858 29937 tablet_bootstrap.cc:654] T 3042212758bf4aef8be5c5d12d10113c P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:26.081811 29938 tablet_bootstrap.cc:654] T 3d0498256b5f4b8fb1681759429f10ed P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:26.083129 28113 tablet_service.cc:1467] Processing CreateTablet for tablet 8fc87f7f3c724cd4a270583ddebd87aa (DEFAULT_TABLE table=rebalance_test_table_11 [id=e00af55ec1c448bdb2c617816b10a90f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:26.083511 28113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fc87f7f3c724cd4a270583ddebd87aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:26.083129 28372 tablet_service.cc:1467] Processing CreateTablet for tablet 8fc87f7f3c724cd4a270583ddebd87aa (DEFAULT_TABLE table=rebalance_test_table_11 [id=e00af55ec1c448bdb2c617816b10a90f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:26.083766 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fc87f7f3c724cd4a270583ddebd87aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:26.096519 29936 tablet_bootstrap.cc:492] T 3d0498256b5f4b8fb1681759429f10ed P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:26.097306 29936 ts_tablet_manager.cc:1367] T 3d0498256b5f4b8fb1681759429f10ed P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.000s
I20240503 08:12:26.098464 29936 raft_consensus.cc:348] T 3d0498256b5f4b8fb1681759429f10ed P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:26.098723 29936 raft_consensus.cc:374] T 3d0498256b5f4b8fb1681759429f10ed P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:26.098934 29936 raft_consensus.cc:729] T 3d0498256b5f4b8fb1681759429f10ed P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:26.099181 29936 consensus_queue.cc:260] T 3d0498256b5f4b8fb1681759429f10ed P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:26.099959 29936 ts_tablet_manager.cc:1398] T 3d0498256b5f4b8fb1681759429f10ed P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:26.100984 29936 tablet_bootstrap.cc:492] T 3042212758bf4aef8be5c5d12d10113c P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:26.102293 29935 tablet_bootstrap.cc:492] T 3d0498256b5f4b8fb1681759429f10ed P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:26.102452 29936 tablet_bootstrap.cc:654] T 3042212758bf4aef8be5c5d12d10113c P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:26.102461 29935 ts_tablet_manager.cc:1367] T 3d0498256b5f4b8fb1681759429f10ed P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.004s
I20240503 08:12:26.103232 29935 raft_consensus.cc:348] T 3d0498256b5f4b8fb1681759429f10ed P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:26.103494 29935 raft_consensus.cc:374] T 3d0498256b5f4b8fb1681759429f10ed P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:26.103636 29935 raft_consensus.cc:729] T 3d0498256b5f4b8fb1681759429f10ed P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:26.103911 29935 consensus_queue.cc:260] T 3d0498256b5f4b8fb1681759429f10ed P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:26.106771 29936 tablet_bootstrap.cc:492] T 3042212758bf4aef8be5c5d12d10113c P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:26.106786 29935 ts_tablet_manager.cc:1398] T 3d0498256b5f4b8fb1681759429f10ed P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 08:12:26.106976 29936 ts_tablet_manager.cc:1367] T 3042212758bf4aef8be5c5d12d10113c P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 08:12:26.107074 29935 tablet_bootstrap.cc:492] T 8fc87f7f3c724cd4a270583ddebd87aa P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:26.107079 29937 tablet_bootstrap.cc:492] T 3042212758bf4aef8be5c5d12d10113c P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:26.107200 29937 ts_tablet_manager.cc:1367] T 3042212758bf4aef8be5c5d12d10113c P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.007s
I20240503 08:12:26.107676 29936 raft_consensus.cc:348] T 3042212758bf4aef8be5c5d12d10113c P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:26.107931 29936 raft_consensus.cc:374] T 3042212758bf4aef8be5c5d12d10113c P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:26.107923 29937 raft_consensus.cc:348] T 3042212758bf4aef8be5c5d12d10113c P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:26.108044 29936 raft_consensus.cc:729] T 3042212758bf4aef8be5c5d12d10113c P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:26.108108 29937 raft_consensus.cc:374] T 3042212758bf4aef8be5c5d12d10113c P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:26.108189 29937 raft_consensus.cc:729] T 3042212758bf4aef8be5c5d12d10113c P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:26.108552 29937 consensus_queue.cc:260] T 3042212758bf4aef8be5c5d12d10113c P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:26.109170 29937 ts_tablet_manager.cc:1398] T 3042212758bf4aef8be5c5d12d10113c P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:26.109297 29935 tablet_bootstrap.cc:654] T 8fc87f7f3c724cd4a270583ddebd87aa P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:26.109496 29937 tablet_bootstrap.cc:492] T 8fc87f7f3c724cd4a270583ddebd87aa P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:26.110005 29936 consensus_queue.cc:260] T 3042212758bf4aef8be5c5d12d10113c P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:26.110553 29936 ts_tablet_manager.cc:1398] T 3042212758bf4aef8be5c5d12d10113c P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:12:26.111260 29937 tablet_bootstrap.cc:654] T 8fc87f7f3c724cd4a270583ddebd87aa P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:26.113992 29938 tablet_bootstrap.cc:492] T 3d0498256b5f4b8fb1681759429f10ed P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:26.114145 29938 ts_tablet_manager.cc:1367] T 3d0498256b5f4b8fb1681759429f10ed P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.035s user 0.009s sys 0.009s
I20240503 08:12:26.114908 29938 raft_consensus.cc:348] T 3d0498256b5f4b8fb1681759429f10ed P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:26.115126 29938 raft_consensus.cc:374] T 3d0498256b5f4b8fb1681759429f10ed P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:26.115211 29938 raft_consensus.cc:729] T 3d0498256b5f4b8fb1681759429f10ed P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:26.115422 29938 consensus_queue.cc:260] T 3d0498256b5f4b8fb1681759429f10ed P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:26.115962 29938 ts_tablet_manager.cc:1398] T 3d0498256b5f4b8fb1681759429f10ed P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:12:26.116233 29938 tablet_bootstrap.cc:492] T 8fc87f7f3c724cd4a270583ddebd87aa P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:26.118005 29938 tablet_bootstrap.cc:654] T 8fc87f7f3c724cd4a270583ddebd87aa P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:26.119760 29935 tablet_bootstrap.cc:492] T 8fc87f7f3c724cd4a270583ddebd87aa P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:26.119920 29935 ts_tablet_manager.cc:1367] T 8fc87f7f3c724cd4a270583ddebd87aa P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.001s
I20240503 08:12:26.120622 29935 raft_consensus.cc:348] T 8fc87f7f3c724cd4a270583ddebd87aa P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:26.120846 29935 raft_consensus.cc:374] T 8fc87f7f3c724cd4a270583ddebd87aa P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:26.120934 29935 raft_consensus.cc:729] T 8fc87f7f3c724cd4a270583ddebd87aa P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:26.121147 29935 consensus_queue.cc:260] T 8fc87f7f3c724cd4a270583ddebd87aa P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:26.122267 29935 ts_tablet_manager.cc:1398] T 8fc87f7f3c724cd4a270583ddebd87aa P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:26.123250 29937 tablet_bootstrap.cc:492] T 8fc87f7f3c724cd4a270583ddebd87aa P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:26.123400 29937 ts_tablet_manager.cc:1367] T 8fc87f7f3c724cd4a270583ddebd87aa P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20240503 08:12:26.123966 29937 raft_consensus.cc:348] T 8fc87f7f3c724cd4a270583ddebd87aa P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:26.124125 29938 tablet_bootstrap.cc:492] T 8fc87f7f3c724cd4a270583ddebd87aa P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:26.124140 29937 raft_consensus.cc:374] T 8fc87f7f3c724cd4a270583ddebd87aa P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:26.124248 29937 raft_consensus.cc:729] T 8fc87f7f3c724cd4a270583ddebd87aa P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:26.124300 29938 ts_tablet_manager.cc:1367] T 8fc87f7f3c724cd4a270583ddebd87aa P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240503 08:12:26.124456 29937 consensus_queue.cc:260] T 8fc87f7f3c724cd4a270583ddebd87aa P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:26.124886 29937 ts_tablet_manager.cc:1398] T 8fc87f7f3c724cd4a270583ddebd87aa P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:12:26.125344 29938 raft_consensus.cc:348] T 8fc87f7f3c724cd4a270583ddebd87aa P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:26.125547 29938 raft_consensus.cc:374] T 8fc87f7f3c724cd4a270583ddebd87aa P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:26.125631 29938 raft_consensus.cc:729] T 8fc87f7f3c724cd4a270583ddebd87aa P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:26.126056 29938 consensus_queue.cc:260] T 8fc87f7f3c724cd4a270583ddebd87aa P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:26.126597 29938 ts_tablet_manager.cc:1398] T 8fc87f7f3c724cd4a270583ddebd87aa P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:26.126871 29938 tablet_bootstrap.cc:492] T 3042212758bf4aef8be5c5d12d10113c P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:26.128341 29938 tablet_bootstrap.cc:654] T 3042212758bf4aef8be5c5d12d10113c P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:26.130252 29938 tablet_bootstrap.cc:492] T 3042212758bf4aef8be5c5d12d10113c P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:26.130357 29938 ts_tablet_manager.cc:1367] T 3042212758bf4aef8be5c5d12d10113c P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:26.130894 29938 raft_consensus.cc:348] T 3042212758bf4aef8be5c5d12d10113c P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:26.131075 29938 raft_consensus.cc:374] T 3042212758bf4aef8be5c5d12d10113c P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:26.131176 29938 raft_consensus.cc:729] T 3042212758bf4aef8be5c5d12d10113c P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:26.131381 29938 consensus_queue.cc:260] T 3042212758bf4aef8be5c5d12d10113c P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:26.131868 29938 ts_tablet_manager.cc:1398] T 3042212758bf4aef8be5c5d12d10113c P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:12:26.214097 29929 raft_consensus.cc:482] T 3d0498256b5f4b8fb1681759429f10ed P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:26.214299 29929 raft_consensus.cc:504] T 3d0498256b5f4b8fb1681759429f10ed P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:26.215389 28133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d0498256b5f4b8fb1681759429f10ed" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:12:26.215453 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d0498256b5f4b8fb1681759429f10ed" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:12:26.215637 28133 raft_consensus.cc:2454] T 3d0498256b5f4b8fb1681759429f10ed P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:26.215669 28262 raft_consensus.cc:2454] T 3d0498256b5f4b8fb1681759429f10ed P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:26.215071 29929 leader_election.cc:288] T 3d0498256b5f4b8fb1681759429f10ed P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:12:26.216075 27937 leader_election.cc:302] T 3d0498256b5f4b8fb1681759429f10ed P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:26.216428 29929 raft_consensus.cc:2781] T 3d0498256b5f4b8fb1681759429f10ed P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:26.216531 29929 raft_consensus.cc:482] T 3d0498256b5f4b8fb1681759429f10ed P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:26.216614 29929 raft_consensus.cc:3037] T 3d0498256b5f4b8fb1681759429f10ed P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:26.218458 29929 raft_consensus.cc:504] T 3d0498256b5f4b8fb1681759429f10ed P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:26.218989 29929 leader_election.cc:288] T 3d0498256b5f4b8fb1681759429f10ed P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Requested vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:12:26.219297 28133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d0498256b5f4b8fb1681759429f10ed" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:12:26.219380 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d0498256b5f4b8fb1681759429f10ed" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:12:26.219456 28133 raft_consensus.cc:3037] T 3d0498256b5f4b8fb1681759429f10ed P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:26.219558 28262 raft_consensus.cc:3037] T 3d0498256b5f4b8fb1681759429f10ed P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:26.221185 28133 raft_consensus.cc:2454] T 3d0498256b5f4b8fb1681759429f10ed P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:26.221311 28262 raft_consensus.cc:2454] T 3d0498256b5f4b8fb1681759429f10ed P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:26.221539 27938 leader_election.cc:302] T 3d0498256b5f4b8fb1681759429f10ed P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 833582ff68c94269be364ebd187a75db; no voters:
I20240503 08:12:26.221813 29929 raft_consensus.cc:2781] T 3d0498256b5f4b8fb1681759429f10ed P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:26.221944 29929 raft_consensus.cc:686] T 3d0498256b5f4b8fb1681759429f10ed P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Becoming Leader. State: Replica: 833582ff68c94269be364ebd187a75db, State: Running, Role: LEADER
I20240503 08:12:26.222198 29929 consensus_queue.cc:237] T 3d0498256b5f4b8fb1681759429f10ed P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:26.224807 27856 catalog_manager.cc:5484] T 3d0498256b5f4b8fb1681759429f10ed P 833582ff68c94269be364ebd187a75db reported cstate change: term changed from 0 to 1, leader changed from <none> to 833582ff68c94269be364ebd187a75db (127.20.230.65). New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:12:26.257753 29929 raft_consensus.cc:482] T 3042212758bf4aef8be5c5d12d10113c P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:26.257956 29929 raft_consensus.cc:504] T 3042212758bf4aef8be5c5d12d10113c P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:26.258469 29929 leader_election.cc:288] T 3042212758bf4aef8be5c5d12d10113c P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:26.258862 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3042212758bf4aef8be5c5d12d10113c" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:12:26.258910 28393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3042212758bf4aef8be5c5d12d10113c" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:26.259068 28262 raft_consensus.cc:2454] T 3042212758bf4aef8be5c5d12d10113c P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:26.259102 28393 raft_consensus.cc:2454] T 3042212758bf4aef8be5c5d12d10113c P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:26.259397 27937 leader_election.cc:302] T 3042212758bf4aef8be5c5d12d10113c P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:26.259657 29929 raft_consensus.cc:2781] T 3042212758bf4aef8be5c5d12d10113c P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:26.259738 29929 raft_consensus.cc:482] T 3042212758bf4aef8be5c5d12d10113c P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:26.259810 29929 raft_consensus.cc:3037] T 3042212758bf4aef8be5c5d12d10113c P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:26.261858 29929 raft_consensus.cc:504] T 3042212758bf4aef8be5c5d12d10113c P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:26.262362 29929 leader_election.cc:288] T 3042212758bf4aef8be5c5d12d10113c P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:26.262665 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3042212758bf4aef8be5c5d12d10113c" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:12:26.262750 28393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3042212758bf4aef8be5c5d12d10113c" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:12:26.262898 28262 raft_consensus.cc:3037] T 3042212758bf4aef8be5c5d12d10113c P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:26.262918 28393 raft_consensus.cc:3037] T 3042212758bf4aef8be5c5d12d10113c P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:26.264699 28262 raft_consensus.cc:2454] T 3042212758bf4aef8be5c5d12d10113c P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:26.264721 28393 raft_consensus.cc:2454] T 3042212758bf4aef8be5c5d12d10113c P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:26.265039 27939 leader_election.cc:302] T 3042212758bf4aef8be5c5d12d10113c P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:26.265259 29929 raft_consensus.cc:2781] T 3042212758bf4aef8be5c5d12d10113c P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:26.265405 29929 raft_consensus.cc:686] T 3042212758bf4aef8be5c5d12d10113c P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Becoming Leader. State: Replica: 833582ff68c94269be364ebd187a75db, State: Running, Role: LEADER
I20240503 08:12:26.265630 29929 consensus_queue.cc:237] T 3042212758bf4aef8be5c5d12d10113c P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:26.268543 27856 catalog_manager.cc:5484] T 3042212758bf4aef8be5c5d12d10113c P 833582ff68c94269be364ebd187a75db reported cstate change: term changed from 0 to 1, leader changed from <none> to 833582ff68c94269be364ebd187a75db (127.20.230.65). New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:12:26.350337 29930 raft_consensus.cc:482] T 8fc87f7f3c724cd4a270583ddebd87aa P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:26.350499 29930 raft_consensus.cc:504] T 8fc87f7f3c724cd4a270583ddebd87aa P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:26.351303 29930 leader_election.cc:288] T 8fc87f7f3c724cd4a270583ddebd87aa P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:26.351784 28133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc87f7f3c724cd4a270583ddebd87aa" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:12:26.351768 28393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc87f7f3c724cd4a270583ddebd87aa" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:26.351997 28133 raft_consensus.cc:2454] T 8fc87f7f3c724cd4a270583ddebd87aa P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:12:26.352017 28393 raft_consensus.cc:2454] T 8fc87f7f3c724cd4a270583ddebd87aa P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:12:26.352368 28198 leader_election.cc:302] T 8fc87f7f3c724cd4a270583ddebd87aa P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:26.352607 29930 raft_consensus.cc:2781] T 8fc87f7f3c724cd4a270583ddebd87aa P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:26.352710 29930 raft_consensus.cc:482] T 8fc87f7f3c724cd4a270583ddebd87aa P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:26.352792 29930 raft_consensus.cc:3037] T 8fc87f7f3c724cd4a270583ddebd87aa P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:26.354743 29930 raft_consensus.cc:504] T 8fc87f7f3c724cd4a270583ddebd87aa P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:26.355230 29930 leader_election.cc:288] T 8fc87f7f3c724cd4a270583ddebd87aa P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Requested vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:26.355613 28393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc87f7f3c724cd4a270583ddebd87aa" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:12:26.355871 28393 raft_consensus.cc:3037] T 8fc87f7f3c724cd4a270583ddebd87aa P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:26.355999 28133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc87f7f3c724cd4a270583ddebd87aa" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:12:26.356199 28133 raft_consensus.cc:3037] T 8fc87f7f3c724cd4a270583ddebd87aa P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:26.358589 28393 raft_consensus.cc:2454] T 8fc87f7f3c724cd4a270583ddebd87aa P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:26.358824 28133 raft_consensus.cc:2454] T 8fc87f7f3c724cd4a270583ddebd87aa P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:26.358968 28199 leader_election.cc:302] T 8fc87f7f3c724cd4a270583ddebd87aa P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:26.359221 29770 raft_consensus.cc:2781] T 8fc87f7f3c724cd4a270583ddebd87aa P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:26.359380 29770 raft_consensus.cc:686] T 8fc87f7f3c724cd4a270583ddebd87aa P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Becoming Leader. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Running, Role: LEADER
I20240503 08:12:26.359625 29770 consensus_queue.cc:237] T 8fc87f7f3c724cd4a270583ddebd87aa P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:26.362421 27856 catalog_manager.cc:5484] T 8fc87f7f3c724cd4a270583ddebd87aa P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: term changed from 0 to 1, leader changed from <none> to aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67). New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:12:26.646845 29946 consensus_queue.cc:1035] T 3042212758bf4aef8be5c5d12d10113c P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:26.667230 29946 consensus_queue.cc:1035] T 3042212758bf4aef8be5c5d12d10113c P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:26.684289 29946 consensus_queue.cc:1035] T 3d0498256b5f4b8fb1681759429f10ed P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:26.689172 29929 consensus_queue.cc:1035] T 3d0498256b5f4b8fb1681759429f10ed P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:26.701253 29953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:26.701661 29953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:26.709062 29953 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
I20240503 08:12:26.853510 28393 raft_consensus.cc:1261] T 8fc87f7f3c724cd4a270583ddebd87aa P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:26.856124 29770 consensus_queue.cc:1035] T 8fc87f7f3c724cd4a270583ddebd87aa P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:26.874950 28132 raft_consensus.cc:1261] T 8fc87f7f3c724cd4a270583ddebd87aa P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:26.875579 29770 consensus_queue.cc:1035] T 8fc87f7f3c724cd4a270583ddebd87aa P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 258.854 ms
time per row: 0.129427 ms
I20240503 08:12:27.125072 27856 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalance_test_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:27.133505 27983 tablet_service.cc:1467] Processing CreateTablet for tablet 095d76252eb341c6906963c317224285 (DEFAULT_TABLE table=rebalance_test_table_12 [id=bce17fe5bc2f48b1b1cada053caa5e61]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:27.133672 28113 tablet_service.cc:1467] Processing CreateTablet for tablet 095d76252eb341c6906963c317224285 (DEFAULT_TABLE table=rebalance_test_table_12 [id=bce17fe5bc2f48b1b1cada053caa5e61]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:27.133960 27983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 095d76252eb341c6906963c317224285. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:27.134148 28113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 095d76252eb341c6906963c317224285. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:27.135353 28241 tablet_service.cc:1467] Processing CreateTablet for tablet 095d76252eb341c6906963c317224285 (DEFAULT_TABLE table=rebalance_test_table_12 [id=bce17fe5bc2f48b1b1cada053caa5e61]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:27.135723 28241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 095d76252eb341c6906963c317224285. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:27.136430 28371 tablet_service.cc:1467] Processing CreateTablet for tablet 13cd6e71728040f5a939f212ccb75f0d (DEFAULT_TABLE table=rebalance_test_table_12 [id=bce17fe5bc2f48b1b1cada053caa5e61]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:27.136828 28371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13cd6e71728040f5a939f212ccb75f0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:27.139456 30015 tablet_bootstrap.cc:492] T 095d76252eb341c6906963c317224285 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:27.139423 27982 tablet_service.cc:1467] Processing CreateTablet for tablet 13cd6e71728040f5a939f212ccb75f0d (DEFAULT_TABLE table=rebalance_test_table_12 [id=bce17fe5bc2f48b1b1cada053caa5e61]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:27.139770 27982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13cd6e71728040f5a939f212ccb75f0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:27.141100 30015 tablet_bootstrap.cc:654] T 095d76252eb341c6906963c317224285 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:27.141360 27981 tablet_service.cc:1467] Processing CreateTablet for tablet 333d4108d2ef4a809f5b53304b340899 (DEFAULT_TABLE table=rebalance_test_table_12 [id=bce17fe5bc2f48b1b1cada053caa5e61]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:27.141798 27981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 333d4108d2ef4a809f5b53304b340899. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:27.141722 28243 tablet_service.cc:1467] Processing CreateTablet for tablet 13cd6e71728040f5a939f212ccb75f0d (DEFAULT_TABLE table=rebalance_test_table_12 [id=bce17fe5bc2f48b1b1cada053caa5e61]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:27.142045 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13cd6e71728040f5a939f212ccb75f0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:27.146126 28112 tablet_service.cc:1467] Processing CreateTablet for tablet 333d4108d2ef4a809f5b53304b340899 (DEFAULT_TABLE table=rebalance_test_table_12 [id=bce17fe5bc2f48b1b1cada053caa5e61]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:27.146495 28112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 333d4108d2ef4a809f5b53304b340899. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:27.150152 28372 tablet_service.cc:1467] Processing CreateTablet for tablet 333d4108d2ef4a809f5b53304b340899 (DEFAULT_TABLE table=rebalance_test_table_12 [id=bce17fe5bc2f48b1b1cada053caa5e61]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:27.150383 30018 tablet_bootstrap.cc:492] T 13cd6e71728040f5a939f212ccb75f0d P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:27.150511 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 333d4108d2ef4a809f5b53304b340899. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:27.151575 30019 tablet_bootstrap.cc:492] T 13cd6e71728040f5a939f212ccb75f0d P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:27.152030 30018 tablet_bootstrap.cc:654] T 13cd6e71728040f5a939f212ccb75f0d P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:27.152920 30019 tablet_bootstrap.cc:654] T 13cd6e71728040f5a939f212ccb75f0d P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:27.153838 30015 tablet_bootstrap.cc:492] T 095d76252eb341c6906963c317224285 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:27.153959 30015 ts_tablet_manager.cc:1367] T 095d76252eb341c6906963c317224285 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.007s
I20240503 08:12:27.154076 30018 tablet_bootstrap.cc:492] T 13cd6e71728040f5a939f212ccb75f0d P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:27.154219 30018 ts_tablet_manager.cc:1367] T 13cd6e71728040f5a939f212ccb75f0d P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:12:27.154853 30015 raft_consensus.cc:348] T 095d76252eb341c6906963c317224285 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:27.155126 30015 raft_consensus.cc:374] T 095d76252eb341c6906963c317224285 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:27.155045 30018 raft_consensus.cc:348] T 13cd6e71728040f5a939f212ccb75f0d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:27.155218 30018 raft_consensus.cc:374] T 13cd6e71728040f5a939f212ccb75f0d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:27.155301 30015 raft_consensus.cc:729] T 095d76252eb341c6906963c317224285 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:27.155364 30018 raft_consensus.cc:729] T 13cd6e71728040f5a939f212ccb75f0d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:27.155578 30015 consensus_queue.cc:260] T 095d76252eb341c6906963c317224285 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:27.155741 30018 consensus_queue.cc:260] T 13cd6e71728040f5a939f212ccb75f0d P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:27.156281 30015 ts_tablet_manager.cc:1398] T 095d76252eb341c6906963c317224285 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:27.156425 30018 ts_tablet_manager.cc:1398] T 13cd6e71728040f5a939f212ccb75f0d P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:27.156708 30018 tablet_bootstrap.cc:492] T 333d4108d2ef4a809f5b53304b340899 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:27.158195 30018 tablet_bootstrap.cc:654] T 333d4108d2ef4a809f5b53304b340899 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:27.160229 30018 tablet_bootstrap.cc:492] T 333d4108d2ef4a809f5b53304b340899 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:27.160382 30018 ts_tablet_manager.cc:1367] T 333d4108d2ef4a809f5b53304b340899 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:12:27.161084 30018 raft_consensus.cc:348] T 333d4108d2ef4a809f5b53304b340899 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:27.161337 30018 raft_consensus.cc:374] T 333d4108d2ef4a809f5b53304b340899 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:27.161454 30018 raft_consensus.cc:729] T 333d4108d2ef4a809f5b53304b340899 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:27.161795 30018 consensus_queue.cc:260] T 333d4108d2ef4a809f5b53304b340899 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:27.162319 30018 ts_tablet_manager.cc:1398] T 333d4108d2ef4a809f5b53304b340899 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:27.162695 30018 tablet_bootstrap.cc:492] T 095d76252eb341c6906963c317224285 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:27.164139 30018 tablet_bootstrap.cc:654] T 095d76252eb341c6906963c317224285 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:27.165206 30016 tablet_bootstrap.cc:492] T 095d76252eb341c6906963c317224285 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:27.166967 30016 tablet_bootstrap.cc:654] T 095d76252eb341c6906963c317224285 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:27.170482 30015 tablet_bootstrap.cc:492] T 333d4108d2ef4a809f5b53304b340899 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:27.170629 30016 tablet_bootstrap.cc:492] T 095d76252eb341c6906963c317224285 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:27.170768 30016 ts_tablet_manager.cc:1367] T 095d76252eb341c6906963c317224285 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 08:12:27.171491 30016 raft_consensus.cc:348] T 095d76252eb341c6906963c317224285 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:27.171686 30016 raft_consensus.cc:374] T 095d76252eb341c6906963c317224285 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:27.171777 30016 raft_consensus.cc:729] T 095d76252eb341c6906963c317224285 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:27.171897 30015 tablet_bootstrap.cc:654] T 333d4108d2ef4a809f5b53304b340899 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:27.171993 30016 consensus_queue.cc:260] T 095d76252eb341c6906963c317224285 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:27.172183 30019 tablet_bootstrap.cc:492] T 13cd6e71728040f5a939f212ccb75f0d P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:27.172367 30019 ts_tablet_manager.cc:1367] T 13cd6e71728040f5a939f212ccb75f0d P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20240503 08:12:27.172451 30016 ts_tablet_manager.cc:1398] T 095d76252eb341c6906963c317224285 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:27.172662 30016 tablet_bootstrap.cc:492] T 13cd6e71728040f5a939f212ccb75f0d P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:27.173096 30019 raft_consensus.cc:348] T 13cd6e71728040f5a939f212ccb75f0d P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:27.173291 30019 raft_consensus.cc:374] T 13cd6e71728040f5a939f212ccb75f0d P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:27.173416 30019 raft_consensus.cc:729] T 13cd6e71728040f5a939f212ccb75f0d P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:27.174010 30016 tablet_bootstrap.cc:654] T 13cd6e71728040f5a939f212ccb75f0d P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:27.173657 30019 consensus_queue.cc:260] T 13cd6e71728040f5a939f212ccb75f0d P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:27.175235 30019 ts_tablet_manager.cc:1398] T 13cd6e71728040f5a939f212ccb75f0d P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:12:27.175527 30019 tablet_bootstrap.cc:492] T 333d4108d2ef4a809f5b53304b340899 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:27.176602 30015 tablet_bootstrap.cc:492] T 333d4108d2ef4a809f5b53304b340899 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:27.176625 30018 tablet_bootstrap.cc:492] T 095d76252eb341c6906963c317224285 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:27.176734 30018 ts_tablet_manager.cc:1367] T 095d76252eb341c6906963c317224285 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.005s
I20240503 08:12:27.176752 30015 ts_tablet_manager.cc:1367] T 333d4108d2ef4a809f5b53304b340899 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20240503 08:12:27.177398 30015 raft_consensus.cc:348] T 333d4108d2ef4a809f5b53304b340899 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:27.177603 30015 raft_consensus.cc:374] T 333d4108d2ef4a809f5b53304b340899 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:27.177676 30015 raft_consensus.cc:729] T 333d4108d2ef4a809f5b53304b340899 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:27.177922 30015 consensus_queue.cc:260] T 333d4108d2ef4a809f5b53304b340899 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:27.178336 30015 ts_tablet_manager.cc:1398] T 333d4108d2ef4a809f5b53304b340899 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:27.178910 30018 raft_consensus.cc:348] T 095d76252eb341c6906963c317224285 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:27.179031 30018 raft_consensus.cc:374] T 095d76252eb341c6906963c317224285 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:27.179090 30018 raft_consensus.cc:729] T 095d76252eb341c6906963c317224285 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:27.179296 30018 consensus_queue.cc:260] T 095d76252eb341c6906963c317224285 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:27.179802 30018 ts_tablet_manager.cc:1398] T 095d76252eb341c6906963c317224285 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 08:12:27.181262 30019 tablet_bootstrap.cc:654] T 333d4108d2ef4a809f5b53304b340899 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:27.184305 30016 tablet_bootstrap.cc:492] T 13cd6e71728040f5a939f212ccb75f0d P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:27.184473 30016 ts_tablet_manager.cc:1367] T 13cd6e71728040f5a939f212ccb75f0d P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.004s
I20240503 08:12:27.185109 30016 raft_consensus.cc:348] T 13cd6e71728040f5a939f212ccb75f0d P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:27.185269 30016 raft_consensus.cc:374] T 13cd6e71728040f5a939f212ccb75f0d P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:27.185339 30016 raft_consensus.cc:729] T 13cd6e71728040f5a939f212ccb75f0d P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:27.185564 30016 consensus_queue.cc:260] T 13cd6e71728040f5a939f212ccb75f0d P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:27.186061 30016 ts_tablet_manager.cc:1398] T 13cd6e71728040f5a939f212ccb75f0d P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:27.187815 30019 tablet_bootstrap.cc:492] T 333d4108d2ef4a809f5b53304b340899 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:27.187973 30019 ts_tablet_manager.cc:1367] T 333d4108d2ef4a809f5b53304b340899 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.003s
I20240503 08:12:27.188557 30019 raft_consensus.cc:348] T 333d4108d2ef4a809f5b53304b340899 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:27.188711 30019 raft_consensus.cc:374] T 333d4108d2ef4a809f5b53304b340899 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:27.188776 30019 raft_consensus.cc:729] T 333d4108d2ef4a809f5b53304b340899 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:27.188962 30019 consensus_queue.cc:260] T 333d4108d2ef4a809f5b53304b340899 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:27.189429 30019 ts_tablet_manager.cc:1398] T 333d4108d2ef4a809f5b53304b340899 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 08:12:27.236174 29942 raft_consensus.cc:482] T 13cd6e71728040f5a939f212ccb75f0d P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:27.236332 29942 raft_consensus.cc:504] T 13cd6e71728040f5a939f212ccb75f0d P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:27.237051 29942 leader_election.cc:288] T 13cd6e71728040f5a939f212ccb75f0d P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:12:27.237390 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13cd6e71728040f5a939f212ccb75f0d" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:12:27.237473 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13cd6e71728040f5a939f212ccb75f0d" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:12:27.237589 28003 raft_consensus.cc:2454] T 13cd6e71728040f5a939f212ccb75f0d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:12:27.237672 28262 raft_consensus.cc:2454] T 13cd6e71728040f5a939f212ccb75f0d P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:12:27.238023 28329 leader_election.cc:302] T 13cd6e71728040f5a939f212ccb75f0d P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:27.238286 29942 raft_consensus.cc:2781] T 13cd6e71728040f5a939f212ccb75f0d P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:27.238387 29942 raft_consensus.cc:482] T 13cd6e71728040f5a939f212ccb75f0d P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:27.238471 29942 raft_consensus.cc:3037] T 13cd6e71728040f5a939f212ccb75f0d P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:27.240321 29942 raft_consensus.cc:504] T 13cd6e71728040f5a939f212ccb75f0d P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:27.240847 29942 leader_election.cc:288] T 13cd6e71728040f5a939f212ccb75f0d P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Requested vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:12:27.241212 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13cd6e71728040f5a939f212ccb75f0d" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:12:27.241380 28003 raft_consensus.cc:3037] T 13cd6e71728040f5a939f212ccb75f0d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:27.241313 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13cd6e71728040f5a939f212ccb75f0d" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:12:27.241533 28262 raft_consensus.cc:3037] T 13cd6e71728040f5a939f212ccb75f0d P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:27.243762 28003 raft_consensus.cc:2454] T 13cd6e71728040f5a939f212ccb75f0d P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:12:27.243917 28262 raft_consensus.cc:2454] T 13cd6e71728040f5a939f212ccb75f0d P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:12:27.244096 28329 leader_election.cc:302] T 13cd6e71728040f5a939f212ccb75f0d P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:27.244479 29942 raft_consensus.cc:2781] T 13cd6e71728040f5a939f212ccb75f0d P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:27.244858 29942 raft_consensus.cc:686] T 13cd6e71728040f5a939f212ccb75f0d P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Becoming Leader. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Running, Role: LEADER
I20240503 08:12:27.245180 29942 consensus_queue.cc:237] T 13cd6e71728040f5a939f212ccb75f0d P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:27.247768 27859 catalog_manager.cc:5484] T 13cd6e71728040f5a939f212ccb75f0d P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68). New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:12:27.474675 29929 raft_consensus.cc:482] T 095d76252eb341c6906963c317224285 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:27.474864 29929 raft_consensus.cc:504] T 095d76252eb341c6906963c317224285 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:27.475646 29929 leader_election.cc:288] T 095d76252eb341c6906963c317224285 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:12:27.476007 28131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "095d76252eb341c6906963c317224285" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:12:27.476119 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "095d76252eb341c6906963c317224285" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:12:27.476254 28131 raft_consensus.cc:2454] T 095d76252eb341c6906963c317224285 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:27.476351 28262 raft_consensus.cc:2454] T 095d76252eb341c6906963c317224285 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:27.476691 27938 leader_election.cc:302] T 095d76252eb341c6906963c317224285 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 833582ff68c94269be364ebd187a75db; no voters:
I20240503 08:12:27.476948 29929 raft_consensus.cc:2781] T 095d76252eb341c6906963c317224285 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:27.477035 29929 raft_consensus.cc:482] T 095d76252eb341c6906963c317224285 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:27.477131 29929 raft_consensus.cc:3037] T 095d76252eb341c6906963c317224285 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:27.479168 29929 raft_consensus.cc:504] T 095d76252eb341c6906963c317224285 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:27.479785 29929 leader_election.cc:288] T 095d76252eb341c6906963c317224285 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Requested vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:12:27.479933 28131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "095d76252eb341c6906963c317224285" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:12:27.480044 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "095d76252eb341c6906963c317224285" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:12:27.480123 28131 raft_consensus.cc:3037] T 095d76252eb341c6906963c317224285 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:27.480232 28262 raft_consensus.cc:3037] T 095d76252eb341c6906963c317224285 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:27.481918 28262 raft_consensus.cc:2454] T 095d76252eb341c6906963c317224285 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:27.482247 27937 leader_election.cc:302] T 095d76252eb341c6906963c317224285 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:27.482458 29929 raft_consensus.cc:2781] T 095d76252eb341c6906963c317224285 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:27.482678 29929 raft_consensus.cc:686] T 095d76252eb341c6906963c317224285 P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Becoming Leader. State: Replica: 833582ff68c94269be364ebd187a75db, State: Running, Role: LEADER
I20240503 08:12:27.482817 28131 raft_consensus.cc:2454] T 095d76252eb341c6906963c317224285 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:27.482924 29929 consensus_queue.cc:237] T 095d76252eb341c6906963c317224285 P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:27.486300 27859 catalog_manager.cc:5484] T 095d76252eb341c6906963c317224285 P 833582ff68c94269be364ebd187a75db reported cstate change: term changed from 0 to 1, leader changed from <none> to 833582ff68c94269be364ebd187a75db (127.20.230.65). New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:12:27.523533 29942 raft_consensus.cc:482] T 333d4108d2ef4a809f5b53304b340899 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:27.523679 29942 raft_consensus.cc:504] T 333d4108d2ef4a809f5b53304b340899 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:27.524247 29942 leader_election.cc:288] T 333d4108d2ef4a809f5b53304b340899 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:27.524601 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "333d4108d2ef4a809f5b53304b340899" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:12:27.524636 28131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "333d4108d2ef4a809f5b53304b340899" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:12:27.524760 28003 raft_consensus.cc:2454] T 333d4108d2ef4a809f5b53304b340899 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:12:27.524801 28131 raft_consensus.cc:2454] T 333d4108d2ef4a809f5b53304b340899 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:12:27.525094 28329 leader_election.cc:302] T 333d4108d2ef4a809f5b53304b340899 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:27.525359 29942 raft_consensus.cc:2781] T 333d4108d2ef4a809f5b53304b340899 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:27.525466 29942 raft_consensus.cc:482] T 333d4108d2ef4a809f5b53304b340899 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:27.525557 29942 raft_consensus.cc:3037] T 333d4108d2ef4a809f5b53304b340899 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:27.527237 29942 raft_consensus.cc:504] T 333d4108d2ef4a809f5b53304b340899 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:27.527740 29942 leader_election.cc:288] T 333d4108d2ef4a809f5b53304b340899 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Requested vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:27.527994 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "333d4108d2ef4a809f5b53304b340899" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:12:27.528137 28003 raft_consensus.cc:3037] T 333d4108d2ef4a809f5b53304b340899 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:27.528117 28131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "333d4108d2ef4a809f5b53304b340899" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:12:27.528530 28131 raft_consensus.cc:3037] T 333d4108d2ef4a809f5b53304b340899 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:27.529912 28003 raft_consensus.cc:2454] T 333d4108d2ef4a809f5b53304b340899 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:12:27.530228 28329 leader_election.cc:302] T 333d4108d2ef4a809f5b53304b340899 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:27.530382 28131 raft_consensus.cc:2454] T 333d4108d2ef4a809f5b53304b340899 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:12:27.530436 29942 raft_consensus.cc:2781] T 333d4108d2ef4a809f5b53304b340899 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:27.530588 29942 raft_consensus.cc:686] T 333d4108d2ef4a809f5b53304b340899 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Becoming Leader. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Running, Role: LEADER
I20240503 08:12:27.530975 29942 consensus_queue.cc:237] T 333d4108d2ef4a809f5b53304b340899 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:27.533684 27859 catalog_manager.cc:5484] T 333d4108d2ef4a809f5b53304b340899 P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68). New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:12:27.678766 30029 consensus_queue.cc:1035] T 13cd6e71728040f5a939f212ccb75f0d P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:27.683997 29942 consensus_queue.cc:1035] T 13cd6e71728040f5a939f212ccb75f0d P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:27.914645 30035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:27.914924 30035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:27.922206 30035 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_12'
I20240503 08:12:27.977936 29942 consensus_queue.cc:1035] T 333d4108d2ef4a809f5b53304b340899 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:27.987025 30029 consensus_queue.cc:1035] T 333d4108d2ef4a809f5b53304b340899 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:28.004396 30032 consensus_queue.cc:1035] T 095d76252eb341c6906963c317224285 P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:28.019986 30032 consensus_queue.cc:1035] T 095d76252eb341c6906963c317224285 P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 237.7 ms
time per row: 0.11885 ms
I20240503 08:12:28.306883 27856 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalance_test_table_13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:28.315222 28241 tablet_service.cc:1467] Processing CreateTablet for tablet 7083414bc16c4d499c97b93ca4e3f631 (DEFAULT_TABLE table=rebalance_test_table_13 [id=6e8538506685489f8a1e931a2a781746]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:28.315358 28112 tablet_service.cc:1467] Processing CreateTablet for tablet 7083414bc16c4d499c97b93ca4e3f631 (DEFAULT_TABLE table=rebalance_test_table_13 [id=6e8538506685489f8a1e931a2a781746]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:28.315704 28241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7083414bc16c4d499c97b93ca4e3f631. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:28.315778 28112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7083414bc16c4d499c97b93ca4e3f631. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:28.315706 28243 tablet_service.cc:1467] Processing CreateTablet for tablet 7a101bee3af34166848ee7fa7c733892 (DEFAULT_TABLE table=rebalance_test_table_13 [id=6e8538506685489f8a1e931a2a781746]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:28.316186 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a101bee3af34166848ee7fa7c733892. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:28.317674 28372 tablet_service.cc:1467] Processing CreateTablet for tablet 7083414bc16c4d499c97b93ca4e3f631 (DEFAULT_TABLE table=rebalance_test_table_13 [id=6e8538506685489f8a1e931a2a781746]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:28.318099 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7083414bc16c4d499c97b93ca4e3f631. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:28.318245 28371 tablet_service.cc:1467] Processing CreateTablet for tablet 7a101bee3af34166848ee7fa7c733892 (DEFAULT_TABLE table=rebalance_test_table_13 [id=6e8538506685489f8a1e931a2a781746]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:28.318583 28371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a101bee3af34166848ee7fa7c733892. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:28.319144 27983 tablet_service.cc:1467] Processing CreateTablet for tablet ca38671a760d4534ab903579e0487772 (DEFAULT_TABLE table=rebalance_test_table_13 [id=6e8538506685489f8a1e931a2a781746]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:28.319546 27983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca38671a760d4534ab903579e0487772. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:28.319607 28242 tablet_service.cc:1467] Processing CreateTablet for tablet ca38671a760d4534ab903579e0487772 (DEFAULT_TABLE table=rebalance_test_table_13 [id=6e8538506685489f8a1e931a2a781746]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:28.319918 28242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca38671a760d4534ab903579e0487772. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:28.324339 28113 tablet_service.cc:1467] Processing CreateTablet for tablet 7a101bee3af34166848ee7fa7c733892 (DEFAULT_TABLE table=rebalance_test_table_13 [id=6e8538506685489f8a1e931a2a781746]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:28.324573 28111 tablet_service.cc:1467] Processing CreateTablet for tablet ca38671a760d4534ab903579e0487772 (DEFAULT_TABLE table=rebalance_test_table_13 [id=6e8538506685489f8a1e931a2a781746]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:28.324754 28113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a101bee3af34166848ee7fa7c733892. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:28.324965 28111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca38671a760d4534ab903579e0487772. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:28.339473 30100 tablet_bootstrap.cc:492] T 7083414bc16c4d499c97b93ca4e3f631 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:28.340260 30101 tablet_bootstrap.cc:492] T ca38671a760d4534ab903579e0487772 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:28.341310 30100 tablet_bootstrap.cc:654] T 7083414bc16c4d499c97b93ca4e3f631 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:28.341395 30099 tablet_bootstrap.cc:492] T 7083414bc16c4d499c97b93ca4e3f631 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:28.341899 30102 tablet_bootstrap.cc:492] T ca38671a760d4534ab903579e0487772 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:28.342779 30101 tablet_bootstrap.cc:654] T ca38671a760d4534ab903579e0487772 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:28.343045 30100 tablet_bootstrap.cc:492] T 7083414bc16c4d499c97b93ca4e3f631 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:28.343204 30100 ts_tablet_manager.cc:1367] T 7083414bc16c4d499c97b93ca4e3f631 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:12:28.343586 30102 tablet_bootstrap.cc:654] T ca38671a760d4534ab903579e0487772 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:28.343910 30100 raft_consensus.cc:348] T 7083414bc16c4d499c97b93ca4e3f631 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:28.344107 30100 raft_consensus.cc:374] T 7083414bc16c4d499c97b93ca4e3f631 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:28.344209 30100 raft_consensus.cc:729] T 7083414bc16c4d499c97b93ca4e3f631 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:28.344435 30100 consensus_queue.cc:260] T 7083414bc16c4d499c97b93ca4e3f631 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:28.345580 30099 tablet_bootstrap.cc:654] T 7083414bc16c4d499c97b93ca4e3f631 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:28.346374 30100 ts_tablet_manager.cc:1398] T 7083414bc16c4d499c97b93ca4e3f631 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:12:28.346685 30100 tablet_bootstrap.cc:492] T ca38671a760d4534ab903579e0487772 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:28.348134 30100 tablet_bootstrap.cc:654] T ca38671a760d4534ab903579e0487772 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:28.350173 30101 tablet_bootstrap.cc:492] T ca38671a760d4534ab903579e0487772 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:28.350395 30101 ts_tablet_manager.cc:1367] T ca38671a760d4534ab903579e0487772 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240503 08:12:28.351259 30101 raft_consensus.cc:348] T ca38671a760d4534ab903579e0487772 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:28.351516 30101 raft_consensus.cc:374] T ca38671a760d4534ab903579e0487772 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:28.351666 30101 raft_consensus.cc:729] T ca38671a760d4534ab903579e0487772 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:28.352035 30102 tablet_bootstrap.cc:492] T ca38671a760d4534ab903579e0487772 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:28.352170 30102 ts_tablet_manager.cc:1367] T ca38671a760d4534ab903579e0487772 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20240503 08:12:28.352056 30101 consensus_queue.cc:260] T ca38671a760d4534ab903579e0487772 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:28.352625 30101 ts_tablet_manager.cc:1398] T ca38671a760d4534ab903579e0487772 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:28.352916 30102 raft_consensus.cc:348] T ca38671a760d4534ab903579e0487772 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:28.353121 30102 raft_consensus.cc:374] T ca38671a760d4534ab903579e0487772 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:28.353262 30102 raft_consensus.cc:729] T ca38671a760d4534ab903579e0487772 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:28.353677 30102 consensus_queue.cc:260] T ca38671a760d4534ab903579e0487772 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:28.354360 30102 ts_tablet_manager.cc:1398] T ca38671a760d4534ab903579e0487772 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:28.354668 30102 tablet_bootstrap.cc:492] T 7a101bee3af34166848ee7fa7c733892 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:28.355909 30102 tablet_bootstrap.cc:654] T 7a101bee3af34166848ee7fa7c733892 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:28.356339 30100 tablet_bootstrap.cc:492] T ca38671a760d4534ab903579e0487772 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:28.356484 30100 ts_tablet_manager.cc:1367] T ca38671a760d4534ab903579e0487772 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240503 08:12:28.357141 30100 raft_consensus.cc:348] T ca38671a760d4534ab903579e0487772 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:28.357336 30100 raft_consensus.cc:374] T ca38671a760d4534ab903579e0487772 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:28.357410 30100 raft_consensus.cc:729] T ca38671a760d4534ab903579e0487772 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:28.357645 30100 consensus_queue.cc:260] T ca38671a760d4534ab903579e0487772 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:28.357669 30099 tablet_bootstrap.cc:492] T 7083414bc16c4d499c97b93ca4e3f631 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:28.357892 30102 tablet_bootstrap.cc:492] T 7a101bee3af34166848ee7fa7c733892 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:28.358028 30102 ts_tablet_manager.cc:1367] T 7a101bee3af34166848ee7fa7c733892 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20240503 08:12:28.358049 30099 ts_tablet_manager.cc:1367] T 7083414bc16c4d499c97b93ca4e3f631 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20240503 08:12:28.358212 30100 ts_tablet_manager.cc:1398] T ca38671a760d4534ab903579e0487772 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:28.358600 30100 tablet_bootstrap.cc:492] T 7a101bee3af34166848ee7fa7c733892 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:28.358664 30102 raft_consensus.cc:348] T 7a101bee3af34166848ee7fa7c733892 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:28.358855 30102 raft_consensus.cc:374] T 7a101bee3af34166848ee7fa7c733892 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:28.358933 30102 raft_consensus.cc:729] T 7a101bee3af34166848ee7fa7c733892 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:28.359069 30099 raft_consensus.cc:348] T 7083414bc16c4d499c97b93ca4e3f631 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:28.359151 30102 consensus_queue.cc:260] T 7a101bee3af34166848ee7fa7c733892 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:28.359344 30099 raft_consensus.cc:374] T 7083414bc16c4d499c97b93ca4e3f631 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:28.359470 30099 raft_consensus.cc:729] T 7083414bc16c4d499c97b93ca4e3f631 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:28.359652 30102 ts_tablet_manager.cc:1398] T 7a101bee3af34166848ee7fa7c733892 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:12:28.359902 30102 tablet_bootstrap.cc:492] T 7083414bc16c4d499c97b93ca4e3f631 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:28.359804 30099 consensus_queue.cc:260] T 7083414bc16c4d499c97b93ca4e3f631 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:28.360642 30099 ts_tablet_manager.cc:1398] T 7083414bc16c4d499c97b93ca4e3f631 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:28.360962 30099 tablet_bootstrap.cc:492] T 7a101bee3af34166848ee7fa7c733892 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:28.361714 30102 tablet_bootstrap.cc:654] T 7083414bc16c4d499c97b93ca4e3f631 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:28.362246 30100 tablet_bootstrap.cc:654] T 7a101bee3af34166848ee7fa7c733892 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:28.362653 30099 tablet_bootstrap.cc:654] T 7a101bee3af34166848ee7fa7c733892 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:28.370613 30099 tablet_bootstrap.cc:492] T 7a101bee3af34166848ee7fa7c733892 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:28.370838 30099 ts_tablet_manager.cc:1367] T 7a101bee3af34166848ee7fa7c733892 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20240503 08:12:28.371599 30099 raft_consensus.cc:348] T 7a101bee3af34166848ee7fa7c733892 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:28.371901 30099 raft_consensus.cc:374] T 7a101bee3af34166848ee7fa7c733892 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:28.372012 30099 raft_consensus.cc:729] T 7a101bee3af34166848ee7fa7c733892 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:28.372328 30099 consensus_queue.cc:260] T 7a101bee3af34166848ee7fa7c733892 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:28.372913 30099 ts_tablet_manager.cc:1398] T 7a101bee3af34166848ee7fa7c733892 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:12:28.378338 30112 raft_consensus.cc:482] T 7083414bc16c4d499c97b93ca4e3f631 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:28.378584 30112 raft_consensus.cc:504] T 7083414bc16c4d499c97b93ca4e3f631 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:28.379575 30112 leader_election.cc:288] T 7083414bc16c4d499c97b93ca4e3f631 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:28.379845 28261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7083414bc16c4d499c97b93ca4e3f631" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:12:28.380059 30102 tablet_bootstrap.cc:492] T 7083414bc16c4d499c97b93ca4e3f631 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:28.380196 30102 ts_tablet_manager.cc:1367] T 7083414bc16c4d499c97b93ca4e3f631 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.007s
W20240503 08:12:28.380595 28067 leader_election.cc:341] T 7083414bc16c4d499c97b93ca4e3f631 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:28.380889 30102 raft_consensus.cc:348] T 7083414bc16c4d499c97b93ca4e3f631 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:28.380916 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7083414bc16c4d499c97b93ca4e3f631" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:28.381042 30102 raft_consensus.cc:374] T 7083414bc16c4d499c97b93ca4e3f631 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:28.381088 28391 raft_consensus.cc:2454] T 7083414bc16c4d499c97b93ca4e3f631 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:28.381127 30102 raft_consensus.cc:729] T 7083414bc16c4d499c97b93ca4e3f631 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:28.381317 30102 consensus_queue.cc:260] T 7083414bc16c4d499c97b93ca4e3f631 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:28.381734 30102 ts_tablet_manager.cc:1398] T 7083414bc16c4d499c97b93ca4e3f631 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:12:28.381800 28069 leader_election.cc:302] T 7083414bc16c4d499c97b93ca4e3f631 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, d09a11a03989421b9f28019dc1c14cc2; no voters: aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:12:28.382320 30112 raft_consensus.cc:2781] T 7083414bc16c4d499c97b93ca4e3f631 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:28.382359 30100 tablet_bootstrap.cc:492] T 7a101bee3af34166848ee7fa7c733892 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:28.382504 30112 raft_consensus.cc:482] T 7083414bc16c4d499c97b93ca4e3f631 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:28.382553 30100 ts_tablet_manager.cc:1367] T 7a101bee3af34166848ee7fa7c733892 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.000s
I20240503 08:12:28.382668 30112 raft_consensus.cc:3037] T 7083414bc16c4d499c97b93ca4e3f631 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:28.383329 30100 raft_consensus.cc:348] T 7a101bee3af34166848ee7fa7c733892 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:28.383512 30100 raft_consensus.cc:374] T 7a101bee3af34166848ee7fa7c733892 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:28.383605 30100 raft_consensus.cc:729] T 7a101bee3af34166848ee7fa7c733892 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:28.383832 30100 consensus_queue.cc:260] T 7a101bee3af34166848ee7fa7c733892 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:28.384374 30100 ts_tablet_manager.cc:1398] T 7a101bee3af34166848ee7fa7c733892 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:28.385486 30112 raft_consensus.cc:504] T 7083414bc16c4d499c97b93ca4e3f631 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:28.386149 30112 leader_election.cc:288] T 7083414bc16c4d499c97b93ca4e3f631 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:28.386796 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7083414bc16c4d499c97b93ca4e3f631" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:12:28.386806 28261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7083414bc16c4d499c97b93ca4e3f631" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:12:28.386945 28391 raft_consensus.cc:3037] T 7083414bc16c4d499c97b93ca4e3f631 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:28.386982 28261 raft_consensus.cc:3037] T 7083414bc16c4d499c97b93ca4e3f631 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:28.389142 28391 raft_consensus.cc:2454] T 7083414bc16c4d499c97b93ca4e3f631 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:12:28.389480 28261 raft_consensus.cc:2454] T 7083414bc16c4d499c97b93ca4e3f631 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:12:28.389464 28069 leader_election.cc:302] T 7083414bc16c4d499c97b93ca4e3f631 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:28.389755 30112 raft_consensus.cc:2781] T 7083414bc16c4d499c97b93ca4e3f631 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:28.389959 30112 raft_consensus.cc:686] T 7083414bc16c4d499c97b93ca4e3f631 P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Becoming Leader. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Running, Role: LEADER
I20240503 08:12:28.390197 30112 consensus_queue.cc:237] T 7083414bc16c4d499c97b93ca4e3f631 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:28.393074 27859 catalog_manager.cc:5484] T 7083414bc16c4d499c97b93ca4e3f631 P 1b411c6bb25549368041773d7b710d48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b411c6bb25549368041773d7b710d48 (127.20.230.66). New cstate: current_term: 1 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:12:28.426831 30112 raft_consensus.cc:482] T 7a101bee3af34166848ee7fa7c733892 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:28.427182 30034 raft_consensus.cc:482] T ca38671a760d4534ab903579e0487772 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:28.427119 30112 raft_consensus.cc:504] T 7a101bee3af34166848ee7fa7c733892 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:28.427350 30034 raft_consensus.cc:504] T ca38671a760d4534ab903579e0487772 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:28.427839 30112 leader_election.cc:288] T 7a101bee3af34166848ee7fa7c733892 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:28.428232 30034 leader_election.cc:288] T ca38671a760d4534ab903579e0487772 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:12:28.428524 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a101bee3af34166848ee7fa7c733892" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:28.428575 28261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a101bee3af34166848ee7fa7c733892" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:12:28.428697 28133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca38671a760d4534ab903579e0487772" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:12:28.428735 28261 raft_consensus.cc:2454] T 7a101bee3af34166848ee7fa7c733892 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:28.428674 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca38671a760d4534ab903579e0487772" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:12:28.428761 28391 raft_consensus.cc:2454] T 7a101bee3af34166848ee7fa7c733892 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:28.428890 28133 raft_consensus.cc:2454] T ca38671a760d4534ab903579e0487772 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:12:28.428905 28003 raft_consensus.cc:2454] T ca38671a760d4534ab903579e0487772 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:12:28.429152 28067 leader_election.cc:302] T 7a101bee3af34166848ee7fa7c733892 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:28.429265 28198 leader_election.cc:302] T ca38671a760d4534ab903579e0487772 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:28.429438 30112 raft_consensus.cc:2781] T 7a101bee3af34166848ee7fa7c733892 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:28.429504 30034 raft_consensus.cc:2781] T ca38671a760d4534ab903579e0487772 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:28.429536 30112 raft_consensus.cc:482] T 7a101bee3af34166848ee7fa7c733892 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:28.429608 30034 raft_consensus.cc:482] T ca38671a760d4534ab903579e0487772 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:28.429631 30112 raft_consensus.cc:3037] T 7a101bee3af34166848ee7fa7c733892 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:28.429692 30034 raft_consensus.cc:3037] T ca38671a760d4534ab903579e0487772 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:28.432262 30112 raft_consensus.cc:504] T 7a101bee3af34166848ee7fa7c733892 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:28.432268 30034 raft_consensus.cc:504] T ca38671a760d4534ab903579e0487772 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:28.432833 30034 leader_election.cc:288] T ca38671a760d4534ab903579e0487772 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Requested vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:12:28.432833 30112 leader_election.cc:288] T 7a101bee3af34166848ee7fa7c733892 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:28.433153 28261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a101bee3af34166848ee7fa7c733892" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:12:28.433313 28261 raft_consensus.cc:3037] T 7a101bee3af34166848ee7fa7c733892 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:28.433298 28133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca38671a760d4534ab903579e0487772" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:12:28.433485 28133 raft_consensus.cc:3037] T ca38671a760d4534ab903579e0487772 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:28.433668 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a101bee3af34166848ee7fa7c733892" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:12:28.433893 28391 raft_consensus.cc:3037] T 7a101bee3af34166848ee7fa7c733892 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:28.433866 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca38671a760d4534ab903579e0487772" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:12:28.434048 28003 raft_consensus.cc:3037] T ca38671a760d4534ab903579e0487772 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:28.435791 28261 raft_consensus.cc:2454] T 7a101bee3af34166848ee7fa7c733892 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:12:28.436064 28133 raft_consensus.cc:2454] T ca38671a760d4534ab903579e0487772 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:28.436358 28391 raft_consensus.cc:2454] T 7a101bee3af34166848ee7fa7c733892 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:12:28.436573 28198 leader_election.cc:302] T ca38671a760d4534ab903579e0487772 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:28.436611 28003 raft_consensus.cc:2454] T ca38671a760d4534ab903579e0487772 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:28.436902 28067 leader_election.cc:302] T 7a101bee3af34166848ee7fa7c733892 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:28.436930 30034 raft_consensus.cc:2781] T ca38671a760d4534ab903579e0487772 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:28.437121 30034 raft_consensus.cc:686] T ca38671a760d4534ab903579e0487772 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Becoming Leader. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Running, Role: LEADER
I20240503 08:12:28.437209 30112 raft_consensus.cc:2781] T 7a101bee3af34166848ee7fa7c733892 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:28.437436 30112 raft_consensus.cc:686] T 7a101bee3af34166848ee7fa7c733892 P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Becoming Leader. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Running, Role: LEADER
I20240503 08:12:28.437700 30112 consensus_queue.cc:237] T 7a101bee3af34166848ee7fa7c733892 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:28.437472 30034 consensus_queue.cc:237] T ca38671a760d4534ab903579e0487772 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:28.441205 27859 catalog_manager.cc:5484] T 7a101bee3af34166848ee7fa7c733892 P 1b411c6bb25549368041773d7b710d48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b411c6bb25549368041773d7b710d48 (127.20.230.66). New cstate: current_term: 1 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:12:28.441749 27856 catalog_manager.cc:5484] T ca38671a760d4534ab903579e0487772 P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: term changed from 0 to 1, leader changed from <none> to aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67). New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
W20240503 08:12:28.785584 30117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:28.785929 30117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:28.793567 30117 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:12:28.813580 30112 consensus_queue.cc:1035] T 7083414bc16c4d499c97b93ca4e3f631 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:28.819945 30041 consensus_queue.cc:1035] T 7083414bc16c4d499c97b93ca4e3f631 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_13'
I20240503 08:12:28.855376 29770 consensus_queue.cc:1035] T ca38671a760d4534ab903579e0487772 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:28.876168 29770 consensus_queue.cc:1035] T ca38671a760d4534ab903579e0487772 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:28.923489 30139 meta_cache.cc:1097] Time spent processing table locations response: real 0.054s user 0.002s sys 0.000s
W20240503 08:12:28.923575 30140 meta_cache.cc:1261] Time spent looking up entry by key: real 0.053s user 0.000s sys 0.007s
W20240503 08:12:28.923779 30141 meta_cache.cc:1261] Time spent looking up entry by key: real 0.054s user 0.002s sys 0.001s
W20240503 08:12:28.924046 30138 meta_cache.cc:1261] Time spent looking up entry by key: real 0.053s user 0.000s sys 0.003s
W20240503 08:12:28.923962 30144 meta_cache.cc:1261] Time spent looking up entry by key: real 0.054s user 0.003s sys 0.000s
W20240503 08:12:28.924201 30139 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.20.230.126:39173, user_credentials={real_user=slave}} blocked reactor thread for 54432.6us
I20240503 08:12:28.972235 30041 consensus_queue.cc:1035] T 7a101bee3af34166848ee7fa7c733892 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:28.995690 30112 consensus_queue.cc:1035] T 7a101bee3af34166848ee7fa7c733892 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 255.485 ms
time per row: 0.127743 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 08:12:30.094796 27859 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalance_test_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:30.103220 28112 tablet_service.cc:1467] Processing CreateTablet for tablet f1f2318ab9a24643a4007dc51769ae9c (DEFAULT_TABLE table=rebalance_test_table_14 [id=d2bc25b059a84de8a7559b7f9dcebadf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:30.103423 28372 tablet_service.cc:1467] Processing CreateTablet for tablet f1f2318ab9a24643a4007dc51769ae9c (DEFAULT_TABLE table=rebalance_test_table_14 [id=d2bc25b059a84de8a7559b7f9dcebadf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:30.103668 28112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1f2318ab9a24643a4007dc51769ae9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:30.103821 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1f2318ab9a24643a4007dc51769ae9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:30.104060 27983 tablet_service.cc:1467] Processing CreateTablet for tablet f1f2318ab9a24643a4007dc51769ae9c (DEFAULT_TABLE table=rebalance_test_table_14 [id=d2bc25b059a84de8a7559b7f9dcebadf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:30.104435 27983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1f2318ab9a24643a4007dc51769ae9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:30.107012 28113 tablet_service.cc:1467] Processing CreateTablet for tablet afc92633cbb644be92ab6480743761c1 (DEFAULT_TABLE table=rebalance_test_table_14 [id=d2bc25b059a84de8a7559b7f9dcebadf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:30.107368 28113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afc92633cbb644be92ab6480743761c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:30.108134 27982 tablet_service.cc:1467] Processing CreateTablet for tablet 53af98acddc84bd3b69e00db733379a8 (DEFAULT_TABLE table=rebalance_test_table_14 [id=d2bc25b059a84de8a7559b7f9dcebadf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:30.108471 27982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53af98acddc84bd3b69e00db733379a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:30.108829 28242 tablet_service.cc:1467] Processing CreateTablet for tablet 53af98acddc84bd3b69e00db733379a8 (DEFAULT_TABLE table=rebalance_test_table_14 [id=d2bc25b059a84de8a7559b7f9dcebadf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:30.109210 28242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53af98acddc84bd3b69e00db733379a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:30.113543 30179 tablet_bootstrap.cc:492] T f1f2318ab9a24643a4007dc51769ae9c P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:30.114849 27981 tablet_service.cc:1467] Processing CreateTablet for tablet afc92633cbb644be92ab6480743761c1 (DEFAULT_TABLE table=rebalance_test_table_14 [id=d2bc25b059a84de8a7559b7f9dcebadf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:30.114867 28371 tablet_service.cc:1467] Processing CreateTablet for tablet 53af98acddc84bd3b69e00db733379a8 (DEFAULT_TABLE table=rebalance_test_table_14 [id=d2bc25b059a84de8a7559b7f9dcebadf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:30.115108 28373 tablet_service.cc:1467] Processing CreateTablet for tablet afc92633cbb644be92ab6480743761c1 (DEFAULT_TABLE table=rebalance_test_table_14 [id=d2bc25b059a84de8a7559b7f9dcebadf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:30.115212 27981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afc92633cbb644be92ab6480743761c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:30.115218 28371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53af98acddc84bd3b69e00db733379a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:30.115465 28373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afc92633cbb644be92ab6480743761c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:30.117710 30179 tablet_bootstrap.cc:654] T f1f2318ab9a24643a4007dc51769ae9c P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:30.124142 30180 tablet_bootstrap.cc:492] T f1f2318ab9a24643a4007dc51769ae9c P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:30.125581 30178 tablet_bootstrap.cc:492] T f1f2318ab9a24643a4007dc51769ae9c P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:30.126029 30180 tablet_bootstrap.cc:654] T f1f2318ab9a24643a4007dc51769ae9c P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:30.127460 30178 tablet_bootstrap.cc:654] T f1f2318ab9a24643a4007dc51769ae9c P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:30.131234 30183 tablet_bootstrap.cc:492] T 53af98acddc84bd3b69e00db733379a8 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:30.133325 30183 tablet_bootstrap.cc:654] T 53af98acddc84bd3b69e00db733379a8 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:30.135161 30179 tablet_bootstrap.cc:492] T f1f2318ab9a24643a4007dc51769ae9c P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:30.135288 30179 ts_tablet_manager.cc:1367] T f1f2318ab9a24643a4007dc51769ae9c P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.011s
I20240503 08:12:30.135583 30180 tablet_bootstrap.cc:492] T f1f2318ab9a24643a4007dc51769ae9c P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:30.135726 30180 ts_tablet_manager.cc:1367] T f1f2318ab9a24643a4007dc51769ae9c P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20240503 08:12:30.135886 30179 raft_consensus.cc:348] T f1f2318ab9a24643a4007dc51769ae9c P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:30.136039 30179 raft_consensus.cc:374] T f1f2318ab9a24643a4007dc51769ae9c P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:30.136155 30179 raft_consensus.cc:729] T f1f2318ab9a24643a4007dc51769ae9c P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:30.136353 30179 consensus_queue.cc:260] T f1f2318ab9a24643a4007dc51769ae9c P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:30.136400 30180 raft_consensus.cc:348] T f1f2318ab9a24643a4007dc51769ae9c P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:30.136667 30180 raft_consensus.cc:374] T f1f2318ab9a24643a4007dc51769ae9c P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:30.136788 30178 tablet_bootstrap.cc:492] T f1f2318ab9a24643a4007dc51769ae9c P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:30.136797 30180 raft_consensus.cc:729] T f1f2318ab9a24643a4007dc51769ae9c P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:30.136835 30179 ts_tablet_manager.cc:1398] T f1f2318ab9a24643a4007dc51769ae9c P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:12:30.136947 30178 ts_tablet_manager.cc:1367] T f1f2318ab9a24643a4007dc51769ae9c P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20240503 08:12:30.137229 30180 consensus_queue.cc:260] T f1f2318ab9a24643a4007dc51769ae9c P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:30.137825 30180 ts_tablet_manager.cc:1398] T f1f2318ab9a24643a4007dc51769ae9c P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 08:12:30.138370 30180 tablet_bootstrap.cc:492] T 53af98acddc84bd3b69e00db733379a8 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:30.138733 30179 tablet_bootstrap.cc:492] T afc92633cbb644be92ab6480743761c1 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:30.140290 30179 tablet_bootstrap.cc:654] T afc92633cbb644be92ab6480743761c1 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:30.140612 30180 tablet_bootstrap.cc:654] T 53af98acddc84bd3b69e00db733379a8 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:30.141564 30183 tablet_bootstrap.cc:492] T 53af98acddc84bd3b69e00db733379a8 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:30.141700 30183 ts_tablet_manager.cc:1367] T 53af98acddc84bd3b69e00db733379a8 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240503 08:12:30.142613 30183 raft_consensus.cc:348] T 53af98acddc84bd3b69e00db733379a8 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:30.142812 30183 raft_consensus.cc:374] T 53af98acddc84bd3b69e00db733379a8 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:30.142890 30183 raft_consensus.cc:729] T 53af98acddc84bd3b69e00db733379a8 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:30.143153 30183 consensus_queue.cc:260] T 53af98acddc84bd3b69e00db733379a8 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:30.147280 30160 raft_consensus.cc:482] T f1f2318ab9a24643a4007dc51769ae9c P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:30.147459 30160 raft_consensus.cc:504] T f1f2318ab9a24643a4007dc51769ae9c P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:30.146565 30183 ts_tablet_manager.cc:1398] T 53af98acddc84bd3b69e00db733379a8 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 08:12:30.148113 30160 leader_election.cc:288] T f1f2318ab9a24643a4007dc51769ae9c P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:12:30.148725 28393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1f2318ab9a24643a4007dc51769ae9c" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:30.149288 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1f2318ab9a24643a4007dc51769ae9c" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:12:30.149485 28003 raft_consensus.cc:2454] T f1f2318ab9a24643a4007dc51769ae9c P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:30.149936 28069 leader_election.cc:302] T f1f2318ab9a24643a4007dc51769ae9c P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 833582ff68c94269be364ebd187a75db; no voters:
I20240503 08:12:30.150219 30160 raft_consensus.cc:2781] T f1f2318ab9a24643a4007dc51769ae9c P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:30.150314 30160 raft_consensus.cc:482] T f1f2318ab9a24643a4007dc51769ae9c P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:30.150417 30160 raft_consensus.cc:3037] T f1f2318ab9a24643a4007dc51769ae9c P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:30.151331 30179 tablet_bootstrap.cc:492] T afc92633cbb644be92ab6480743761c1 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:30.151484 30179 ts_tablet_manager.cc:1367] T afc92633cbb644be92ab6480743761c1 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.008s
I20240503 08:12:30.151589 30180 tablet_bootstrap.cc:492] T 53af98acddc84bd3b69e00db733379a8 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:30.151734 30180 ts_tablet_manager.cc:1367] T 53af98acddc84bd3b69e00db733379a8 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.007s
I20240503 08:12:30.152076 30179 raft_consensus.cc:348] T afc92633cbb644be92ab6480743761c1 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:30.152271 30179 raft_consensus.cc:374] T afc92633cbb644be92ab6480743761c1 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:30.152348 30179 raft_consensus.cc:729] T afc92633cbb644be92ab6480743761c1 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:30.152374 30180 raft_consensus.cc:348] T 53af98acddc84bd3b69e00db733379a8 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:30.152562 30180 raft_consensus.cc:374] T 53af98acddc84bd3b69e00db733379a8 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:30.152628 30180 raft_consensus.cc:729] T 53af98acddc84bd3b69e00db733379a8 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:30.152576 30179 consensus_queue.cc:260] T afc92633cbb644be92ab6480743761c1 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:30.152829 30180 consensus_queue.cc:260] T 53af98acddc84bd3b69e00db733379a8 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:30.152984 30160 raft_consensus.cc:504] T f1f2318ab9a24643a4007dc51769ae9c P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:30.153429 30180 ts_tablet_manager.cc:1398] T 53af98acddc84bd3b69e00db733379a8 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:30.153944 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1f2318ab9a24643a4007dc51769ae9c" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:12:30.154047 30180 tablet_bootstrap.cc:492] T afc92633cbb644be92ab6480743761c1 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:30.154071 28001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1f2318ab9a24643a4007dc51769ae9c" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:12:30.154256 28001 raft_consensus.cc:3037] T f1f2318ab9a24643a4007dc51769ae9c P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:30.154774 30179 ts_tablet_manager.cc:1398] T afc92633cbb644be92ab6480743761c1 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:12:30.155620 30180 tablet_bootstrap.cc:654] T afc92633cbb644be92ab6480743761c1 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:30.157521 28001 raft_consensus.cc:2454] T f1f2318ab9a24643a4007dc51769ae9c P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:12:30.157980 28069 leader_election.cc:302] T f1f2318ab9a24643a4007dc51769ae9c P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 833582ff68c94269be364ebd187a75db; no voters: d09a11a03989421b9f28019dc1c14cc2
I20240503 08:12:30.158217 30188 raft_consensus.cc:2781] T f1f2318ab9a24643a4007dc51769ae9c P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:30.158563 30188 raft_consensus.cc:686] T f1f2318ab9a24643a4007dc51769ae9c P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Becoming Leader. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Running, Role: LEADER
I20240503 08:12:30.158829 30188 consensus_queue.cc:237] T f1f2318ab9a24643a4007dc51769ae9c P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:30.160149 30180 tablet_bootstrap.cc:492] T afc92633cbb644be92ab6480743761c1 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:30.160183 30160 leader_election.cc:288] T f1f2318ab9a24643a4007dc51769ae9c P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Requested vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:12:30.160287 30180 ts_tablet_manager.cc:1367] T afc92633cbb644be92ab6480743761c1 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20240503 08:12:30.160951 30180 raft_consensus.cc:348] T afc92633cbb644be92ab6480743761c1 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:30.161114 30180 raft_consensus.cc:374] T afc92633cbb644be92ab6480743761c1 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:30.161188 30180 raft_consensus.cc:729] T afc92633cbb644be92ab6480743761c1 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:30.138479 30178 raft_consensus.cc:348] T f1f2318ab9a24643a4007dc51769ae9c P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:30.161396 30178 raft_consensus.cc:374] T f1f2318ab9a24643a4007dc51769ae9c P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:30.161382 30180 consensus_queue.cc:260] T afc92633cbb644be92ab6480743761c1 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:30.161478 30178 raft_consensus.cc:729] T f1f2318ab9a24643a4007dc51769ae9c P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:30.161736 30178 consensus_queue.cc:260] T f1f2318ab9a24643a4007dc51769ae9c P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:30.161893 30180 ts_tablet_manager.cc:1398] T afc92633cbb644be92ab6480743761c1 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:12:30.162026 28393 raft_consensus.cc:2454] T f1f2318ab9a24643a4007dc51769ae9c P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:30.162406 30178 ts_tablet_manager.cc:1398] T f1f2318ab9a24643a4007dc51769ae9c P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20240503 08:12:30.162667 30178 tablet_bootstrap.cc:492] T 53af98acddc84bd3b69e00db733379a8 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:30.162534 27859 catalog_manager.cc:5484] T f1f2318ab9a24643a4007dc51769ae9c P 1b411c6bb25549368041773d7b710d48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b411c6bb25549368041773d7b710d48 (127.20.230.66). New cstate: current_term: 1 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:12:30.164366 30178 tablet_bootstrap.cc:654] T 53af98acddc84bd3b69e00db733379a8 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:30.167671 30178 tablet_bootstrap.cc:492] T 53af98acddc84bd3b69e00db733379a8 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:30.167817 30178 ts_tablet_manager.cc:1367] T 53af98acddc84bd3b69e00db733379a8 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240503 08:12:30.168396 30178 raft_consensus.cc:348] T 53af98acddc84bd3b69e00db733379a8 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:30.168620 30178 raft_consensus.cc:374] T 53af98acddc84bd3b69e00db733379a8 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:30.168709 30178 raft_consensus.cc:729] T 53af98acddc84bd3b69e00db733379a8 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:30.168900 30178 consensus_queue.cc:260] T 53af98acddc84bd3b69e00db733379a8 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:30.169288 30178 ts_tablet_manager.cc:1398] T 53af98acddc84bd3b69e00db733379a8 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:12:30.169647 30178 tablet_bootstrap.cc:492] T afc92633cbb644be92ab6480743761c1 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:30.170717 30178 tablet_bootstrap.cc:654] T afc92633cbb644be92ab6480743761c1 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:30.172142 30178 tablet_bootstrap.cc:492] T afc92633cbb644be92ab6480743761c1 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:30.172260 30178 ts_tablet_manager.cc:1367] T afc92633cbb644be92ab6480743761c1 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:30.172750 30178 raft_consensus.cc:348] T afc92633cbb644be92ab6480743761c1 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:30.172881 30178 raft_consensus.cc:374] T afc92633cbb644be92ab6480743761c1 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:30.172962 30178 raft_consensus.cc:729] T afc92633cbb644be92ab6480743761c1 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:30.173184 30178 consensus_queue.cc:260] T afc92633cbb644be92ab6480743761c1 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:30.173621 30178 ts_tablet_manager.cc:1398] T afc92633cbb644be92ab6480743761c1 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 08:12:30.219002 30188 raft_consensus.cc:482] T afc92633cbb644be92ab6480743761c1 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:30.219147 30188 raft_consensus.cc:504] T afc92633cbb644be92ab6480743761c1 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:30.219651 30188 leader_election.cc:288] T afc92633cbb644be92ab6480743761c1 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:30.220039 28001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc92633cbb644be92ab6480743761c1" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:12:30.220163 28393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc92633cbb644be92ab6480743761c1" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:30.220232 28001 raft_consensus.cc:2454] T afc92633cbb644be92ab6480743761c1 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:30.220360 28393 raft_consensus.cc:2454] T afc92633cbb644be92ab6480743761c1 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:30.220561 28069 leader_election.cc:302] T afc92633cbb644be92ab6480743761c1 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 833582ff68c94269be364ebd187a75db; no voters:
I20240503 08:12:30.220813 30188 raft_consensus.cc:2781] T afc92633cbb644be92ab6480743761c1 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:30.220908 30188 raft_consensus.cc:482] T afc92633cbb644be92ab6480743761c1 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:30.221005 30188 raft_consensus.cc:3037] T afc92633cbb644be92ab6480743761c1 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:30.222836 30188 raft_consensus.cc:504] T afc92633cbb644be92ab6480743761c1 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:30.223351 30188 leader_election.cc:288] T afc92633cbb644be92ab6480743761c1 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Requested vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:30.223726 28001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc92633cbb644be92ab6480743761c1" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:12:30.223780 28393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc92633cbb644be92ab6480743761c1" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:12:30.223901 28001 raft_consensus.cc:3037] T afc92633cbb644be92ab6480743761c1 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:30.223951 28393 raft_consensus.cc:3037] T afc92633cbb644be92ab6480743761c1 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:30.225627 28001 raft_consensus.cc:2454] T afc92633cbb644be92ab6480743761c1 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:12:30.225874 28393 raft_consensus.cc:2454] T afc92633cbb644be92ab6480743761c1 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:12:30.225984 28069 leader_election.cc:302] T afc92633cbb644be92ab6480743761c1 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 833582ff68c94269be364ebd187a75db; no voters:
I20240503 08:12:30.226219 30188 raft_consensus.cc:2781] T afc92633cbb644be92ab6480743761c1 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:30.226372 30188 raft_consensus.cc:686] T afc92633cbb644be92ab6480743761c1 P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Becoming Leader. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Running, Role: LEADER
I20240503 08:12:30.226585 30188 consensus_queue.cc:237] T afc92633cbb644be92ab6480743761c1 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:30.229408 27854 catalog_manager.cc:5484] T afc92633cbb644be92ab6480743761c1 P 1b411c6bb25549368041773d7b710d48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b411c6bb25549368041773d7b710d48 (127.20.230.66). New cstate: current_term: 1 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:12:30.397238 30166 raft_consensus.cc:482] T 53af98acddc84bd3b69e00db733379a8 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:30.397392 30166 raft_consensus.cc:504] T 53af98acddc84bd3b69e00db733379a8 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:30.397962 30166 leader_election.cc:288] T 53af98acddc84bd3b69e00db733379a8 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:30.398319 28001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53af98acddc84bd3b69e00db733379a8" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:12:30.398404 28393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53af98acddc84bd3b69e00db733379a8" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:30.398507 28001 raft_consensus.cc:2454] T 53af98acddc84bd3b69e00db733379a8 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:12:30.398635 28393 raft_consensus.cc:2454] T 53af98acddc84bd3b69e00db733379a8 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:12:30.398859 28199 leader_election.cc:302] T 53af98acddc84bd3b69e00db733379a8 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:30.399119 30166 raft_consensus.cc:2781] T 53af98acddc84bd3b69e00db733379a8 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:30.399212 30166 raft_consensus.cc:482] T 53af98acddc84bd3b69e00db733379a8 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:30.399299 30166 raft_consensus.cc:3037] T 53af98acddc84bd3b69e00db733379a8 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:30.401291 30166 raft_consensus.cc:504] T 53af98acddc84bd3b69e00db733379a8 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:30.401809 30166 leader_election.cc:288] T 53af98acddc84bd3b69e00db733379a8 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Requested vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:30.402155 28393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53af98acddc84bd3b69e00db733379a8" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:12:30.402138 28001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53af98acddc84bd3b69e00db733379a8" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:12:30.402305 28001 raft_consensus.cc:3037] T 53af98acddc84bd3b69e00db733379a8 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:30.402308 28393 raft_consensus.cc:3037] T 53af98acddc84bd3b69e00db733379a8 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:30.404008 28393 raft_consensus.cc:2454] T 53af98acddc84bd3b69e00db733379a8 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:30.404042 28001 raft_consensus.cc:2454] T 53af98acddc84bd3b69e00db733379a8 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:30.404318 28199 leader_election.cc:302] T 53af98acddc84bd3b69e00db733379a8 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:30.404560 30166 raft_consensus.cc:2781] T 53af98acddc84bd3b69e00db733379a8 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:30.404671 30166 raft_consensus.cc:686] T 53af98acddc84bd3b69e00db733379a8 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Becoming Leader. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Running, Role: LEADER
I20240503 08:12:30.404901 30166 consensus_queue.cc:237] T 53af98acddc84bd3b69e00db733379a8 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:30.407526 27854 catalog_manager.cc:5484] T 53af98acddc84bd3b69e00db733379a8 P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: term changed from 0 to 1, leader changed from <none> to aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67). New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:12:30.566465 28393 raft_consensus.cc:3037] T f1f2318ab9a24643a4007dc51769ae9c P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:30.569319 30188 consensus_queue.cc:1035] T f1f2318ab9a24643a4007dc51769ae9c P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:30.575225 30160 consensus_queue.cc:1035] T f1f2318ab9a24643a4007dc51769ae9c P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:30.621461 30190 consensus_queue.cc:1035] T afc92633cbb644be92ab6480743761c1 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:30.626089 30190 consensus_queue.cc:1035] T afc92633cbb644be92ab6480743761c1 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:30.764582 30197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:30.764873 30197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:30.772022 30197 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_14'
I20240503 08:12:30.809875 30166 consensus_queue.cc:1035] T 53af98acddc84bd3b69e00db733379a8 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:30.815435 30166 consensus_queue.cc:1035] T 53af98acddc84bd3b69e00db733379a8 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 213.171 ms
time per row: 0.106585 ms
I20240503 08:12:31.127759 27857 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalance_test_table_15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:31.135593 28242 tablet_service.cc:1467] Processing CreateTablet for tablet fe4d26ee022b424195e836834bd2f051 (DEFAULT_TABLE table=rebalance_test_table_15 [id=261c96b04fc141ddb92650c24ce76fa8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:31.136039 28242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe4d26ee022b424195e836834bd2f051. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:31.135994 27983 tablet_service.cc:1467] Processing CreateTablet for tablet fe4d26ee022b424195e836834bd2f051 (DEFAULT_TABLE table=rebalance_test_table_15 [id=261c96b04fc141ddb92650c24ce76fa8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:31.136384 27983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe4d26ee022b424195e836834bd2f051. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:31.136386 28241 tablet_service.cc:1467] Processing CreateTablet for tablet 46d27fa12990405cae5f94976ac7a014 (DEFAULT_TABLE table=rebalance_test_table_15 [id=261c96b04fc141ddb92650c24ce76fa8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:31.136718 28241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d27fa12990405cae5f94976ac7a014. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:31.138325 28112 tablet_service.cc:1467] Processing CreateTablet for tablet fe4d26ee022b424195e836834bd2f051 (DEFAULT_TABLE table=rebalance_test_table_15 [id=261c96b04fc141ddb92650c24ce76fa8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:31.138705 28112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe4d26ee022b424195e836834bd2f051. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:31.140971 28372 tablet_service.cc:1467] Processing CreateTablet for tablet 46d27fa12990405cae5f94976ac7a014 (DEFAULT_TABLE table=rebalance_test_table_15 [id=261c96b04fc141ddb92650c24ce76fa8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:31.141346 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d27fa12990405cae5f94976ac7a014. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:31.141367 28113 tablet_service.cc:1467] Processing CreateTablet for tablet 33f0f4bc5722485a8f978a02367ca80e (DEFAULT_TABLE table=rebalance_test_table_15 [id=261c96b04fc141ddb92650c24ce76fa8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:31.141697 28113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33f0f4bc5722485a8f978a02367ca80e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:31.142891 30260 tablet_bootstrap.cc:492] T fe4d26ee022b424195e836834bd2f051 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:31.143867 27983 tablet_service.cc:1467] Processing CreateTablet for tablet 46d27fa12990405cae5f94976ac7a014 (DEFAULT_TABLE table=rebalance_test_table_15 [id=261c96b04fc141ddb92650c24ce76fa8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:31.144259 27983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d27fa12990405cae5f94976ac7a014. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:31.144177 28243 tablet_service.cc:1467] Processing CreateTablet for tablet 33f0f4bc5722485a8f978a02367ca80e (DEFAULT_TABLE table=rebalance_test_table_15 [id=261c96b04fc141ddb92650c24ce76fa8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:31.144528 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33f0f4bc5722485a8f978a02367ca80e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:31.144572 30260 tablet_bootstrap.cc:654] T fe4d26ee022b424195e836834bd2f051 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:31.149343 28373 tablet_service.cc:1467] Processing CreateTablet for tablet 33f0f4bc5722485a8f978a02367ca80e (DEFAULT_TABLE table=rebalance_test_table_15 [id=261c96b04fc141ddb92650c24ce76fa8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:31.149680 28373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33f0f4bc5722485a8f978a02367ca80e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:31.153898 30262 tablet_bootstrap.cc:492] T fe4d26ee022b424195e836834bd2f051 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:31.154836 30260 tablet_bootstrap.cc:492] T fe4d26ee022b424195e836834bd2f051 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:31.155020 30260 ts_tablet_manager.cc:1367] T fe4d26ee022b424195e836834bd2f051 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.010s
I20240503 08:12:31.155807 30260 raft_consensus.cc:348] T fe4d26ee022b424195e836834bd2f051 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:31.155982 30262 tablet_bootstrap.cc:654] T fe4d26ee022b424195e836834bd2f051 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:31.156078 30260 raft_consensus.cc:374] T fe4d26ee022b424195e836834bd2f051 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:31.156755 30260 raft_consensus.cc:729] T fe4d26ee022b424195e836834bd2f051 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:31.157065 30260 consensus_queue.cc:260] T fe4d26ee022b424195e836834bd2f051 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:31.157681 30260 ts_tablet_manager.cc:1398] T fe4d26ee022b424195e836834bd2f051 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:12:31.158178 30260 tablet_bootstrap.cc:492] T 46d27fa12990405cae5f94976ac7a014 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:31.158969 30262 tablet_bootstrap.cc:492] T fe4d26ee022b424195e836834bd2f051 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:31.159147 30262 ts_tablet_manager.cc:1367] T fe4d26ee022b424195e836834bd2f051 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 08:12:31.159654 30260 tablet_bootstrap.cc:654] T 46d27fa12990405cae5f94976ac7a014 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:31.159938 30262 raft_consensus.cc:348] T fe4d26ee022b424195e836834bd2f051 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:31.160259 30262 raft_consensus.cc:374] T fe4d26ee022b424195e836834bd2f051 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:31.160398 30262 raft_consensus.cc:729] T fe4d26ee022b424195e836834bd2f051 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:31.160655 30262 consensus_queue.cc:260] T fe4d26ee022b424195e836834bd2f051 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:31.162147 30262 ts_tablet_manager.cc:1398] T fe4d26ee022b424195e836834bd2f051 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:12:31.162631 30262 tablet_bootstrap.cc:492] T 33f0f4bc5722485a8f978a02367ca80e P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:31.165169 30262 tablet_bootstrap.cc:654] T 33f0f4bc5722485a8f978a02367ca80e P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:31.170745 30260 tablet_bootstrap.cc:492] T 46d27fa12990405cae5f94976ac7a014 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:31.170773 30267 tablet_bootstrap.cc:492] T 33f0f4bc5722485a8f978a02367ca80e P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:31.170939 30265 tablet_bootstrap.cc:492] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:31.170966 30260 ts_tablet_manager.cc:1367] T 46d27fa12990405cae5f94976ac7a014 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.011s
I20240503 08:12:31.171061 30262 tablet_bootstrap.cc:492] T 33f0f4bc5722485a8f978a02367ca80e P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:31.171264 30262 ts_tablet_manager.cc:1367] T 33f0f4bc5722485a8f978a02367ca80e P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240503 08:12:31.171746 30260 raft_consensus.cc:348] T 46d27fa12990405cae5f94976ac7a014 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:31.172039 30260 raft_consensus.cc:374] T 46d27fa12990405cae5f94976ac7a014 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:31.172041 30262 raft_consensus.cc:348] T 33f0f4bc5722485a8f978a02367ca80e P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:31.172152 30260 raft_consensus.cc:729] T 46d27fa12990405cae5f94976ac7a014 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:31.172247 30262 raft_consensus.cc:374] T 33f0f4bc5722485a8f978a02367ca80e P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:31.172330 30262 raft_consensus.cc:729] T 33f0f4bc5722485a8f978a02367ca80e P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:31.172567 30262 consensus_queue.cc:260] T 33f0f4bc5722485a8f978a02367ca80e P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:31.172560 30260 consensus_queue.cc:260] T 46d27fa12990405cae5f94976ac7a014 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:31.172773 30265 tablet_bootstrap.cc:654] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:31.173275 30260 ts_tablet_manager.cc:1398] T 46d27fa12990405cae5f94976ac7a014 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:12:31.173974 30262 ts_tablet_manager.cc:1398] T 33f0f4bc5722485a8f978a02367ca80e P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:31.174122 30267 tablet_bootstrap.cc:654] T 33f0f4bc5722485a8f978a02367ca80e P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:31.174614 30265 tablet_bootstrap.cc:492] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:31.174751 30265 ts_tablet_manager.cc:1367] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240503 08:12:31.175458 30265 raft_consensus.cc:348] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:31.175709 30265 raft_consensus.cc:374] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:31.175838 30265 raft_consensus.cc:729] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:31.176002 30267 tablet_bootstrap.cc:492] T 33f0f4bc5722485a8f978a02367ca80e P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:31.176182 30267 ts_tablet_manager.cc:1367] T 33f0f4bc5722485a8f978a02367ca80e P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 08:12:31.176100 30265 consensus_queue.cc:260] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:31.176640 30265 ts_tablet_manager.cc:1398] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:31.177086 30267 raft_consensus.cc:348] T 33f0f4bc5722485a8f978a02367ca80e P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:31.177325 30267 raft_consensus.cc:374] T 33f0f4bc5722485a8f978a02367ca80e P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:31.177441 30267 raft_consensus.cc:729] T 33f0f4bc5722485a8f978a02367ca80e P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:31.177716 30267 consensus_queue.cc:260] T 33f0f4bc5722485a8f978a02367ca80e P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:31.178428 30267 ts_tablet_manager.cc:1398] T 33f0f4bc5722485a8f978a02367ca80e P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:31.178747 30267 tablet_bootstrap.cc:492] T 46d27fa12990405cae5f94976ac7a014 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:31.180022 30265 tablet_bootstrap.cc:492] T 46d27fa12990405cae5f94976ac7a014 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:31.182029 30267 tablet_bootstrap.cc:654] T 46d27fa12990405cae5f94976ac7a014 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:31.182307 30265 tablet_bootstrap.cc:654] T 46d27fa12990405cae5f94976ac7a014 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:31.183777 30267 tablet_bootstrap.cc:492] T 46d27fa12990405cae5f94976ac7a014 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:31.183890 30267 ts_tablet_manager.cc:1367] T 46d27fa12990405cae5f94976ac7a014 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 08:12:31.184347 30267 raft_consensus.cc:348] T 46d27fa12990405cae5f94976ac7a014 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:31.184525 30267 raft_consensus.cc:374] T 46d27fa12990405cae5f94976ac7a014 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:31.184605 30267 raft_consensus.cc:729] T 46d27fa12990405cae5f94976ac7a014 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:31.184811 30267 consensus_queue.cc:260] T 46d27fa12990405cae5f94976ac7a014 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:31.185217 30267 ts_tablet_manager.cc:1398] T 46d27fa12990405cae5f94976ac7a014 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 08:12:31.185460 30267 tablet_bootstrap.cc:492] T fe4d26ee022b424195e836834bd2f051 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:31.186683 30267 tablet_bootstrap.cc:654] T fe4d26ee022b424195e836834bd2f051 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:31.186688 30265 tablet_bootstrap.cc:492] T 46d27fa12990405cae5f94976ac7a014 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:31.186866 30265 ts_tablet_manager.cc:1367] T 46d27fa12990405cae5f94976ac7a014 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.001s
I20240503 08:12:31.187433 30265 raft_consensus.cc:348] T 46d27fa12990405cae5f94976ac7a014 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:31.188328 30267 tablet_bootstrap.cc:492] T fe4d26ee022b424195e836834bd2f051 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:31.187674 30265 raft_consensus.cc:374] T 46d27fa12990405cae5f94976ac7a014 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:31.188438 30267 ts_tablet_manager.cc:1367] T fe4d26ee022b424195e836834bd2f051 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:31.188499 30265 raft_consensus.cc:729] T 46d27fa12990405cae5f94976ac7a014 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:31.188761 30265 consensus_queue.cc:260] T 46d27fa12990405cae5f94976ac7a014 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:31.189319 30265 ts_tablet_manager.cc:1398] T 46d27fa12990405cae5f94976ac7a014 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:31.189632 30267 raft_consensus.cc:348] T fe4d26ee022b424195e836834bd2f051 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:31.189849 30267 raft_consensus.cc:374] T fe4d26ee022b424195e836834bd2f051 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:31.189960 30267 raft_consensus.cc:729] T fe4d26ee022b424195e836834bd2f051 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:31.190167 30267 consensus_queue.cc:260] T fe4d26ee022b424195e836834bd2f051 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:31.190639 30267 ts_tablet_manager.cc:1398] T fe4d26ee022b424195e836834bd2f051 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:31.195195 30166 raft_consensus.cc:482] T 33f0f4bc5722485a8f978a02367ca80e P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:31.195329 30166 raft_consensus.cc:504] T 33f0f4bc5722485a8f978a02367ca80e P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:31.195860 30166 leader_election.cc:288] T 33f0f4bc5722485a8f978a02367ca80e P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:31.196300 28392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33f0f4bc5722485a8f978a02367ca80e" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:31.196491 28392 raft_consensus.cc:2454] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:12:31.196825 28199 leader_election.cc:302] T 33f0f4bc5722485a8f978a02367ca80e P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:31.197105 30166 raft_consensus.cc:2781] T 33f0f4bc5722485a8f978a02367ca80e P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:31.197206 30166 raft_consensus.cc:482] T 33f0f4bc5722485a8f978a02367ca80e P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:31.197309 30166 raft_consensus.cc:3037] T 33f0f4bc5722485a8f978a02367ca80e P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:31.199875 30166 raft_consensus.cc:504] T 33f0f4bc5722485a8f978a02367ca80e P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:31.200452 30166 leader_election.cc:288] T 33f0f4bc5722485a8f978a02367ca80e P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Requested vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:31.200742 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33f0f4bc5722485a8f978a02367ca80e" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:12:31.200911 28132 raft_consensus.cc:3037] T 33f0f4bc5722485a8f978a02367ca80e P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:31.200939 28392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33f0f4bc5722485a8f978a02367ca80e" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:12:31.201157 28392 raft_consensus.cc:3037] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:31.201988 28133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33f0f4bc5722485a8f978a02367ca80e" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:12:31.203233 28392 raft_consensus.cc:2454] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:31.203490 28132 raft_consensus.cc:2454] T 33f0f4bc5722485a8f978a02367ca80e P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:31.203711 28199 leader_election.cc:302] T 33f0f4bc5722485a8f978a02367ca80e P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:31.203958 30166 raft_consensus.cc:2781] T 33f0f4bc5722485a8f978a02367ca80e P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:31.204125 30166 raft_consensus.cc:686] T 33f0f4bc5722485a8f978a02367ca80e P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Becoming Leader. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Running, Role: LEADER
I20240503 08:12:31.204489 30166 consensus_queue.cc:237] T 33f0f4bc5722485a8f978a02367ca80e P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:31.207345 27859 catalog_manager.cc:5484] T 33f0f4bc5722485a8f978a02367ca80e P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: term changed from 0 to 1, leader changed from <none> to aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67). New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:12:31.280699 30166 raft_consensus.cc:482] T 46d27fa12990405cae5f94976ac7a014 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:31.280877 30166 raft_consensus.cc:504] T 46d27fa12990405cae5f94976ac7a014 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:31.281548 30166 leader_election.cc:288] T 46d27fa12990405cae5f94976ac7a014 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:31.281960 28001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d27fa12990405cae5f94976ac7a014" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:12:31.282224 28001 raft_consensus.cc:2454] T 46d27fa12990405cae5f94976ac7a014 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:12:31.282227 28392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d27fa12990405cae5f94976ac7a014" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:31.282449 28392 raft_consensus.cc:2454] T 46d27fa12990405cae5f94976ac7a014 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:12:31.282789 28199 leader_election.cc:302] T 46d27fa12990405cae5f94976ac7a014 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:31.283031 30166 raft_consensus.cc:2781] T 46d27fa12990405cae5f94976ac7a014 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:31.283140 30166 raft_consensus.cc:482] T 46d27fa12990405cae5f94976ac7a014 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:31.283226 30166 raft_consensus.cc:3037] T 46d27fa12990405cae5f94976ac7a014 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:31.285146 30166 raft_consensus.cc:504] T 46d27fa12990405cae5f94976ac7a014 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:31.285722 30166 leader_election.cc:288] T 46d27fa12990405cae5f94976ac7a014 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Requested vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:31.286417 28392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d27fa12990405cae5f94976ac7a014" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:12:31.286471 28001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d27fa12990405cae5f94976ac7a014" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:12:31.286621 28392 raft_consensus.cc:3037] T 46d27fa12990405cae5f94976ac7a014 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:31.286697 28001 raft_consensus.cc:3037] T 46d27fa12990405cae5f94976ac7a014 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:31.289258 28001 raft_consensus.cc:2454] T 46d27fa12990405cae5f94976ac7a014 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:31.289458 28392 raft_consensus.cc:2454] T 46d27fa12990405cae5f94976ac7a014 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:31.289734 28199 leader_election.cc:302] T 46d27fa12990405cae5f94976ac7a014 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:31.290060 30166 raft_consensus.cc:2781] T 46d27fa12990405cae5f94976ac7a014 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:31.290225 30166 raft_consensus.cc:686] T 46d27fa12990405cae5f94976ac7a014 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Becoming Leader. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Running, Role: LEADER
I20240503 08:12:31.290500 30166 consensus_queue.cc:237] T 46d27fa12990405cae5f94976ac7a014 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:31.293560 27859 catalog_manager.cc:5484] T 46d27fa12990405cae5f94976ac7a014 P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: term changed from 0 to 1, leader changed from <none> to aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67). New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:12:31.332017 30160 raft_consensus.cc:482] T fe4d26ee022b424195e836834bd2f051 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:31.332180 30160 raft_consensus.cc:504] T fe4d26ee022b424195e836834bd2f051 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:31.332765 30160 leader_election.cc:288] T fe4d26ee022b424195e836834bd2f051 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:12:31.333149 28261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4d26ee022b424195e836834bd2f051" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:12:31.333201 28001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4d26ee022b424195e836834bd2f051" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:12:31.333396 28261 raft_consensus.cc:2454] T fe4d26ee022b424195e836834bd2f051 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:31.333448 28001 raft_consensus.cc:2454] T fe4d26ee022b424195e836834bd2f051 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:31.333804 28067 leader_election.cc:302] T fe4d26ee022b424195e836834bd2f051 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:31.334163 30160 raft_consensus.cc:2781] T fe4d26ee022b424195e836834bd2f051 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:31.334252 30160 raft_consensus.cc:482] T fe4d26ee022b424195e836834bd2f051 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:31.334318 30160 raft_consensus.cc:3037] T fe4d26ee022b424195e836834bd2f051 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:31.336585 30160 raft_consensus.cc:504] T fe4d26ee022b424195e836834bd2f051 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:31.337108 30160 leader_election.cc:288] T fe4d26ee022b424195e836834bd2f051 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:12:31.337502 28261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4d26ee022b424195e836834bd2f051" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:12:31.337567 28001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4d26ee022b424195e836834bd2f051" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:12:31.337651 28261 raft_consensus.cc:3037] T fe4d26ee022b424195e836834bd2f051 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:31.337742 28001 raft_consensus.cc:3037] T fe4d26ee022b424195e836834bd2f051 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:31.339413 28261 raft_consensus.cc:2454] T fe4d26ee022b424195e836834bd2f051 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:12:31.339596 28001 raft_consensus.cc:2454] T fe4d26ee022b424195e836834bd2f051 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:12:31.339716 28067 leader_election.cc:302] T fe4d26ee022b424195e836834bd2f051 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:31.339990 30160 raft_consensus.cc:2781] T fe4d26ee022b424195e836834bd2f051 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:31.340116 30160 raft_consensus.cc:686] T fe4d26ee022b424195e836834bd2f051 P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Becoming Leader. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Running, Role: LEADER
I20240503 08:12:31.340363 30160 consensus_queue.cc:237] T fe4d26ee022b424195e836834bd2f051 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:31.343202 27859 catalog_manager.cc:5484] T fe4d26ee022b424195e836834bd2f051 P 1b411c6bb25549368041773d7b710d48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b411c6bb25549368041773d7b710d48 (127.20.230.66). New cstate: current_term: 1 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:12:31.692791 30166 consensus_queue.cc:1035] T 33f0f4bc5722485a8f978a02367ca80e P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:31.708315 30166 consensus_queue.cc:1035] T 33f0f4bc5722485a8f978a02367ca80e P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:31.731925 30276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:31.732293 30276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:31.739784 30276 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_15'
I20240503 08:12:31.798692 30188 consensus_queue.cc:1035] T fe4d26ee022b424195e836834bd2f051 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:31.809198 30166 consensus_queue.cc:1035] T 46d27fa12990405cae5f94976ac7a014 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:31.816323 30160 consensus_queue.cc:1035] T fe4d26ee022b424195e836834bd2f051 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:31.847509 30166 consensus_queue.cc:1035] T 46d27fa12990405cae5f94976ac7a014 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:31.993508 28327 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.67:39919, user_credentials={real_user=slave}} blocked reactor thread for 58371.3us
INSERT report
rows total: 2000
time total: 214.084 ms
time per row: 0.107042 ms
I20240503 08:12:32.091239 27853 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalance_test_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:32.099578 28112 tablet_service.cc:1467] Processing CreateTablet for tablet d03de2915a144ce1b57f1765bc7ec6cf (DEFAULT_TABLE table=rebalance_test_table_16 [id=8910f6c8c8f340c4a138c3d7438bc73a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:32.100040 28112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d03de2915a144ce1b57f1765bc7ec6cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:32.100373 27981 tablet_service.cc:1467] Processing CreateTablet for tablet 366ac74fe5154b15af08bac60648b7b9 (DEFAULT_TABLE table=rebalance_test_table_16 [id=8910f6c8c8f340c4a138c3d7438bc73a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:32.100780 27981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 366ac74fe5154b15af08bac60648b7b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:32.101958 28242 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd92f707605425c92677dfc5f8358d4 (DEFAULT_TABLE table=rebalance_test_table_16 [id=8910f6c8c8f340c4a138c3d7438bc73a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:32.102165 27983 tablet_service.cc:1467] Processing CreateTablet for tablet d03de2915a144ce1b57f1765bc7ec6cf (DEFAULT_TABLE table=rebalance_test_table_16 [id=8910f6c8c8f340c4a138c3d7438bc73a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:32.102329 28242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fd92f707605425c92677dfc5f8358d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:32.102531 27983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d03de2915a144ce1b57f1765bc7ec6cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:32.106532 28243 tablet_service.cc:1467] Processing CreateTablet for tablet d03de2915a144ce1b57f1765bc7ec6cf (DEFAULT_TABLE table=rebalance_test_table_16 [id=8910f6c8c8f340c4a138c3d7438bc73a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:32.106674 28113 tablet_service.cc:1467] Processing CreateTablet for tablet 366ac74fe5154b15af08bac60648b7b9 (DEFAULT_TABLE table=rebalance_test_table_16 [id=8910f6c8c8f340c4a138c3d7438bc73a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:32.106896 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d03de2915a144ce1b57f1765bc7ec6cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:32.107017 28113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 366ac74fe5154b15af08bac60648b7b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:32.113307 28373 tablet_service.cc:1467] Processing CreateTablet for tablet 366ac74fe5154b15af08bac60648b7b9 (DEFAULT_TABLE table=rebalance_test_table_16 [id=8910f6c8c8f340c4a138c3d7438bc73a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:32.113687 28373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 366ac74fe5154b15af08bac60648b7b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:32.113751 27982 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd92f707605425c92677dfc5f8358d4 (DEFAULT_TABLE table=rebalance_test_table_16 [id=8910f6c8c8f340c4a138c3d7438bc73a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:32.114560 27982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fd92f707605425c92677dfc5f8358d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:32.117929 30334 tablet_bootstrap.cc:492] T 3fd92f707605425c92677dfc5f8358d4 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:32.118091 28372 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd92f707605425c92677dfc5f8358d4 (DEFAULT_TABLE table=rebalance_test_table_16 [id=8910f6c8c8f340c4a138c3d7438bc73a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:32.118239 30333 tablet_bootstrap.cc:492] T 366ac74fe5154b15af08bac60648b7b9 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:32.118413 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fd92f707605425c92677dfc5f8358d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:32.120214 30334 tablet_bootstrap.cc:654] T 3fd92f707605425c92677dfc5f8358d4 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:32.120482 30333 tablet_bootstrap.cc:654] T 366ac74fe5154b15af08bac60648b7b9 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:32.127192 30337 tablet_bootstrap.cc:492] T 3fd92f707605425c92677dfc5f8358d4 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:32.129228 30337 tablet_bootstrap.cc:654] T 3fd92f707605425c92677dfc5f8358d4 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:32.129531 30332 tablet_bootstrap.cc:492] T d03de2915a144ce1b57f1765bc7ec6cf P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:32.130071 30333 tablet_bootstrap.cc:492] T 366ac74fe5154b15af08bac60648b7b9 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:32.130276 30333 ts_tablet_manager.cc:1367] T 366ac74fe5154b15af08bac60648b7b9 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20240503 08:12:32.130300 30334 tablet_bootstrap.cc:492] T 3fd92f707605425c92677dfc5f8358d4 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:32.130407 30334 ts_tablet_manager.cc:1367] T 3fd92f707605425c92677dfc5f8358d4 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s
I20240503 08:12:32.131197 30332 tablet_bootstrap.cc:654] T d03de2915a144ce1b57f1765bc7ec6cf P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:32.131134 30334 raft_consensus.cc:348] T 3fd92f707605425c92677dfc5f8358d4 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:32.131325 30334 raft_consensus.cc:374] T 3fd92f707605425c92677dfc5f8358d4 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:32.131417 30334 raft_consensus.cc:729] T 3fd92f707605425c92677dfc5f8358d4 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:32.131134 30333 raft_consensus.cc:348] T 366ac74fe5154b15af08bac60648b7b9 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:32.131633 30333 raft_consensus.cc:374] T 366ac74fe5154b15af08bac60648b7b9 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:32.131690 30334 consensus_queue.cc:260] T 3fd92f707605425c92677dfc5f8358d4 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:32.131765 30333 raft_consensus.cc:729] T 366ac74fe5154b15af08bac60648b7b9 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:32.132076 30333 consensus_queue.cc:260] T 366ac74fe5154b15af08bac60648b7b9 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:32.132807 30333 ts_tablet_manager.cc:1398] T 366ac74fe5154b15af08bac60648b7b9 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:32.133142 30333 tablet_bootstrap.cc:492] T 3fd92f707605425c92677dfc5f8358d4 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:32.134940 30333 tablet_bootstrap.cc:654] T 3fd92f707605425c92677dfc5f8358d4 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:32.136627 30334 ts_tablet_manager.cc:1398] T 3fd92f707605425c92677dfc5f8358d4 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 08:12:32.139214 30334 tablet_bootstrap.cc:492] T d03de2915a144ce1b57f1765bc7ec6cf P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:32.141649 30334 tablet_bootstrap.cc:654] T d03de2915a144ce1b57f1765bc7ec6cf P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:32.143253 30333 tablet_bootstrap.cc:492] T 3fd92f707605425c92677dfc5f8358d4 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:32.143416 30333 ts_tablet_manager.cc:1367] T 3fd92f707605425c92677dfc5f8358d4 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.004s
I20240503 08:12:32.144191 30333 raft_consensus.cc:348] T 3fd92f707605425c92677dfc5f8358d4 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:32.144455 30333 raft_consensus.cc:374] T 3fd92f707605425c92677dfc5f8358d4 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:32.144547 30332 tablet_bootstrap.cc:492] T d03de2915a144ce1b57f1765bc7ec6cf P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:32.144531 30333 raft_consensus.cc:729] T 3fd92f707605425c92677dfc5f8358d4 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:32.144692 30332 ts_tablet_manager.cc:1367] T d03de2915a144ce1b57f1765bc7ec6cf P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.013s
I20240503 08:12:32.144789 30337 tablet_bootstrap.cc:492] T 3fd92f707605425c92677dfc5f8358d4 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:32.144805 30333 consensus_queue.cc:260] T 3fd92f707605425c92677dfc5f8358d4 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:32.144989 30337 ts_tablet_manager.cc:1367] T 3fd92f707605425c92677dfc5f8358d4 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.003s
I20240503 08:12:32.145486 30333 ts_tablet_manager.cc:1398] T 3fd92f707605425c92677dfc5f8358d4 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:32.145406 30332 raft_consensus.cc:348] T d03de2915a144ce1b57f1765bc7ec6cf P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:32.145586 30332 raft_consensus.cc:374] T d03de2915a144ce1b57f1765bc7ec6cf P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:32.145664 30332 raft_consensus.cc:729] T d03de2915a144ce1b57f1765bc7ec6cf P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:32.145942 30333 tablet_bootstrap.cc:492] T d03de2915a144ce1b57f1765bc7ec6cf P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:32.145870 30332 consensus_queue.cc:260] T d03de2915a144ce1b57f1765bc7ec6cf P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:32.146365 30332 ts_tablet_manager.cc:1398] T d03de2915a144ce1b57f1765bc7ec6cf P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:32.146624 30332 tablet_bootstrap.cc:492] T 366ac74fe5154b15af08bac60648b7b9 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:32.147934 30337 raft_consensus.cc:348] T 3fd92f707605425c92677dfc5f8358d4 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:32.148068 30332 tablet_bootstrap.cc:654] T 366ac74fe5154b15af08bac60648b7b9 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:32.148187 30337 raft_consensus.cc:374] T 3fd92f707605425c92677dfc5f8358d4 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:32.148321 30337 raft_consensus.cc:729] T 3fd92f707605425c92677dfc5f8358d4 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:32.148573 30337 consensus_queue.cc:260] T 3fd92f707605425c92677dfc5f8358d4 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:32.149144 30337 ts_tablet_manager.cc:1398] T 3fd92f707605425c92677dfc5f8358d4 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:12:32.150081 30334 tablet_bootstrap.cc:492] T d03de2915a144ce1b57f1765bc7ec6cf P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:32.150108 30332 tablet_bootstrap.cc:492] T 366ac74fe5154b15af08bac60648b7b9 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:32.150233 30334 ts_tablet_manager.cc:1367] T d03de2915a144ce1b57f1765bc7ec6cf P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.001s
I20240503 08:12:32.150249 30332 ts_tablet_manager.cc:1367] T 366ac74fe5154b15af08bac60648b7b9 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:32.150792 30337 tablet_bootstrap.cc:492] T 366ac74fe5154b15af08bac60648b7b9 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:32.150916 30332 raft_consensus.cc:348] T 366ac74fe5154b15af08bac60648b7b9 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:32.150943 30334 raft_consensus.cc:348] T d03de2915a144ce1b57f1765bc7ec6cf P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:32.151131 30332 raft_consensus.cc:374] T 366ac74fe5154b15af08bac60648b7b9 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:32.151147 30334 raft_consensus.cc:374] T d03de2915a144ce1b57f1765bc7ec6cf P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:32.151204 30332 raft_consensus.cc:729] T 366ac74fe5154b15af08bac60648b7b9 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:32.151213 30334 raft_consensus.cc:729] T d03de2915a144ce1b57f1765bc7ec6cf P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:32.151427 30332 consensus_queue.cc:260] T 366ac74fe5154b15af08bac60648b7b9 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:32.151473 30334 consensus_queue.cc:260] T d03de2915a144ce1b57f1765bc7ec6cf P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:32.151978 30332 ts_tablet_manager.cc:1398] T 366ac74fe5154b15af08bac60648b7b9 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:12:32.152135 30334 ts_tablet_manager.cc:1398] T d03de2915a144ce1b57f1765bc7ec6cf P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:32.152747 30337 tablet_bootstrap.cc:654] T 366ac74fe5154b15af08bac60648b7b9 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:32.150496 30206 raft_consensus.cc:482] T 366ac74fe5154b15af08bac60648b7b9 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:32.155483 30206 raft_consensus.cc:504] T 366ac74fe5154b15af08bac60648b7b9 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:32.156273 30206 leader_election.cc:288] T 366ac74fe5154b15af08bac60648b7b9 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:32.156723 30333 tablet_bootstrap.cc:654] T d03de2915a144ce1b57f1765bc7ec6cf P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:32.156700 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "366ac74fe5154b15af08bac60648b7b9" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:32.157591 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "366ac74fe5154b15af08bac60648b7b9" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
W20240503 08:12:32.157632 27939 leader_election.cc:341] T 366ac74fe5154b15af08bac60648b7b9 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:32.157946 28132 raft_consensus.cc:2454] T 366ac74fe5154b15af08bac60648b7b9 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:32.158565 27938 leader_election.cc:302] T 366ac74fe5154b15af08bac60648b7b9 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 833582ff68c94269be364ebd187a75db; no voters: d09a11a03989421b9f28019dc1c14cc2
I20240503 08:12:32.161267 30343 raft_consensus.cc:2781] T 366ac74fe5154b15af08bac60648b7b9 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:32.161377 30343 raft_consensus.cc:482] T 366ac74fe5154b15af08bac60648b7b9 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:32.161504 30343 raft_consensus.cc:3037] T 366ac74fe5154b15af08bac60648b7b9 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:32.164763 30343 raft_consensus.cc:504] T 366ac74fe5154b15af08bac60648b7b9 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:32.165935 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "366ac74fe5154b15af08bac60648b7b9" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:12:32.166141 28132 raft_consensus.cc:3037] T 366ac74fe5154b15af08bac60648b7b9 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:32.168902 28132 raft_consensus.cc:2454] T 366ac74fe5154b15af08bac60648b7b9 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:32.169229 27938 leader_election.cc:302] T 366ac74fe5154b15af08bac60648b7b9 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 833582ff68c94269be364ebd187a75db; no voters:
I20240503 08:12:32.169489 30206 raft_consensus.cc:2781] T 366ac74fe5154b15af08bac60648b7b9 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:32.169911 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "366ac74fe5154b15af08bac60648b7b9" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
W20240503 08:12:32.170526 27939 leader_election.cc:341] T 366ac74fe5154b15af08bac60648b7b9 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:32.170259 30343 leader_election.cc:288] T 366ac74fe5154b15af08bac60648b7b9 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Requested vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:32.174237 30206 raft_consensus.cc:686] T 366ac74fe5154b15af08bac60648b7b9 P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Becoming Leader. State: Replica: 833582ff68c94269be364ebd187a75db, State: Running, Role: LEADER
I20240503 08:12:32.174510 30206 consensus_queue.cc:237] T 366ac74fe5154b15af08bac60648b7b9 P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:32.174716 30337 tablet_bootstrap.cc:492] T 366ac74fe5154b15af08bac60648b7b9 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:32.174878 30337 ts_tablet_manager.cc:1367] T 366ac74fe5154b15af08bac60648b7b9 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.001s
I20240503 08:12:32.174949 30333 tablet_bootstrap.cc:492] T d03de2915a144ce1b57f1765bc7ec6cf P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:32.175084 30333 ts_tablet_manager.cc:1367] T d03de2915a144ce1b57f1765bc7ec6cf P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.006s
I20240503 08:12:32.175854 30333 raft_consensus.cc:348] T d03de2915a144ce1b57f1765bc7ec6cf P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:32.176052 30333 raft_consensus.cc:374] T d03de2915a144ce1b57f1765bc7ec6cf P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:32.176154 30333 raft_consensus.cc:729] T d03de2915a144ce1b57f1765bc7ec6cf P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:32.176360 30333 consensus_queue.cc:260] T d03de2915a144ce1b57f1765bc7ec6cf P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:32.176896 30333 ts_tablet_manager.cc:1398] T d03de2915a144ce1b57f1765bc7ec6cf P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:12:32.177153 30337 raft_consensus.cc:348] T 366ac74fe5154b15af08bac60648b7b9 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:32.177328 30337 raft_consensus.cc:374] T 366ac74fe5154b15af08bac60648b7b9 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:32.177399 30337 raft_consensus.cc:729] T 366ac74fe5154b15af08bac60648b7b9 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:32.177615 30337 consensus_queue.cc:260] T 366ac74fe5154b15af08bac60648b7b9 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:32.178167 30337 ts_tablet_manager.cc:1398] T 366ac74fe5154b15af08bac60648b7b9 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:32.178352 27857 catalog_manager.cc:5484] T 366ac74fe5154b15af08bac60648b7b9 P 833582ff68c94269be364ebd187a75db reported cstate change: term changed from 0 to 1, leader changed from <none> to 833582ff68c94269be364ebd187a75db (127.20.230.65). New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:12:32.205056 30160 raft_consensus.cc:482] T d03de2915a144ce1b57f1765bc7ec6cf P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:32.205228 30160 raft_consensus.cc:504] T d03de2915a144ce1b57f1765bc7ec6cf P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:32.205832 30160 leader_election.cc:288] T d03de2915a144ce1b57f1765bc7ec6cf P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:12:32.206161 28001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d03de2915a144ce1b57f1765bc7ec6cf" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:12:32.206364 28001 raft_consensus.cc:2454] T d03de2915a144ce1b57f1765bc7ec6cf P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:32.206296 28261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d03de2915a144ce1b57f1765bc7ec6cf" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:12:32.206508 28261 raft_consensus.cc:2454] T d03de2915a144ce1b57f1765bc7ec6cf P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:32.206699 28069 leader_election.cc:302] T d03de2915a144ce1b57f1765bc7ec6cf P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 833582ff68c94269be364ebd187a75db; no voters:
I20240503 08:12:32.206926 30160 raft_consensus.cc:2781] T d03de2915a144ce1b57f1765bc7ec6cf P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:32.207041 30160 raft_consensus.cc:482] T d03de2915a144ce1b57f1765bc7ec6cf P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:32.207127 30160 raft_consensus.cc:3037] T d03de2915a144ce1b57f1765bc7ec6cf P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:32.208956 30160 raft_consensus.cc:504] T d03de2915a144ce1b57f1765bc7ec6cf P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:32.209462 30160 leader_election.cc:288] T d03de2915a144ce1b57f1765bc7ec6cf P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Requested vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:12:32.209807 28001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d03de2915a144ce1b57f1765bc7ec6cf" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:12:32.209913 28261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d03de2915a144ce1b57f1765bc7ec6cf" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:12:32.210003 28001 raft_consensus.cc:3037] T d03de2915a144ce1b57f1765bc7ec6cf P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:32.210094 28261 raft_consensus.cc:3037] T d03de2915a144ce1b57f1765bc7ec6cf P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:32.212260 28001 raft_consensus.cc:2454] T d03de2915a144ce1b57f1765bc7ec6cf P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:12:32.212671 28069 leader_election.cc:302] T d03de2915a144ce1b57f1765bc7ec6cf P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 833582ff68c94269be364ebd187a75db; no voters:
I20240503 08:12:32.212728 28261 raft_consensus.cc:2454] T d03de2915a144ce1b57f1765bc7ec6cf P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:12:32.212924 30160 raft_consensus.cc:2781] T d03de2915a144ce1b57f1765bc7ec6cf P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:32.213145 30160 raft_consensus.cc:686] T d03de2915a144ce1b57f1765bc7ec6cf P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Becoming Leader. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Running, Role: LEADER
I20240503 08:12:32.213403 30160 consensus_queue.cc:237] T d03de2915a144ce1b57f1765bc7ec6cf P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:32.216610 27857 catalog_manager.cc:5484] T d03de2915a144ce1b57f1765bc7ec6cf P 1b411c6bb25549368041773d7b710d48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b411c6bb25549368041773d7b710d48 (127.20.230.66). New cstate: current_term: 1 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:12:32.451602 30252 raft_consensus.cc:482] T 3fd92f707605425c92677dfc5f8358d4 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:32.451774 30252 raft_consensus.cc:504] T 3fd92f707605425c92677dfc5f8358d4 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:32.452384 30252 leader_election.cc:288] T 3fd92f707605425c92677dfc5f8358d4 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:12:32.452754 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd92f707605425c92677dfc5f8358d4" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:32.452874 28001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd92f707605425c92677dfc5f8358d4" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:12:32.453027 28391 raft_consensus.cc:2454] T 3fd92f707605425c92677dfc5f8358d4 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:12:32.453411 28199 leader_election.cc:302] T 3fd92f707605425c92677dfc5f8358d4 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:32.453486 28001 raft_consensus.cc:2454] T 3fd92f707605425c92677dfc5f8358d4 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:12:32.453685 30252 raft_consensus.cc:2781] T 3fd92f707605425c92677dfc5f8358d4 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:32.453995 30252 raft_consensus.cc:482] T 3fd92f707605425c92677dfc5f8358d4 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:32.454087 30252 raft_consensus.cc:3037] T 3fd92f707605425c92677dfc5f8358d4 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:32.455858 30252 raft_consensus.cc:504] T 3fd92f707605425c92677dfc5f8358d4 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:32.456296 30252 leader_election.cc:288] T 3fd92f707605425c92677dfc5f8358d4 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Requested vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:12:32.456631 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd92f707605425c92677dfc5f8358d4" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:12:32.456718 28001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd92f707605425c92677dfc5f8358d4" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:12:32.456918 28001 raft_consensus.cc:3037] T 3fd92f707605425c92677dfc5f8358d4 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:32.457204 28391 raft_consensus.cc:3037] T 3fd92f707605425c92677dfc5f8358d4 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:32.459097 28001 raft_consensus.cc:2454] T 3fd92f707605425c92677dfc5f8358d4 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:32.459466 28199 leader_election.cc:302] T 3fd92f707605425c92677dfc5f8358d4 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:32.459712 30252 raft_consensus.cc:2781] T 3fd92f707605425c92677dfc5f8358d4 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:32.460054 30252 raft_consensus.cc:686] T 3fd92f707605425c92677dfc5f8358d4 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Becoming Leader. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Running, Role: LEADER
I20240503 08:12:32.460289 28391 raft_consensus.cc:2454] T 3fd92f707605425c92677dfc5f8358d4 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:32.461014 30252 consensus_queue.cc:237] T 3fd92f707605425c92677dfc5f8358d4 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:32.467008 27857 catalog_manager.cc:5484] T 3fd92f707605425c92677dfc5f8358d4 P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: term changed from 0 to 1, leader changed from <none> to aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67). New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:12:32.583065 30203 raft_consensus.cc:482] T 366ac74fe5154b15af08bac60648b7b9 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:32.583284 30203 raft_consensus.cc:504] T 366ac74fe5154b15af08bac60648b7b9 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:32.584146 30203 leader_election.cc:288] T 366ac74fe5154b15af08bac60648b7b9 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:32.584692 28001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "366ac74fe5154b15af08bac60648b7b9" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:12:32.584715 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "366ac74fe5154b15af08bac60648b7b9" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:12:32.585033 28132 raft_consensus.cc:2379] T 366ac74fe5154b15af08bac60648b7b9 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d09a11a03989421b9f28019dc1c14cc2 in current term 1: Already voted for candidate 833582ff68c94269be364ebd187a75db in this term.
I20240503 08:12:32.585649 28328 leader_election.cc:302] T 366ac74fe5154b15af08bac60648b7b9 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d09a11a03989421b9f28019dc1c14cc2; no voters: 1b411c6bb25549368041773d7b710d48, 833582ff68c94269be364ebd187a75db
I20240503 08:12:32.585986 30203 raft_consensus.cc:3037] T 366ac74fe5154b15af08bac60648b7b9 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:32.588380 30203 raft_consensus.cc:2726] T 366ac74fe5154b15af08bac60648b7b9 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:12:32.598582 30188 consensus_queue.cc:1035] T d03de2915a144ce1b57f1765bc7ec6cf P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:32.600137 30346 consensus_queue.cc:1035] T 366ac74fe5154b15af08bac60648b7b9 P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:32.638532 30346 consensus_queue.cc:1035] T 366ac74fe5154b15af08bac60648b7b9 P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:32.652236 30160 consensus_queue.cc:1035] T d03de2915a144ce1b57f1765bc7ec6cf P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:32.839848 30350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:32.840131 30350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:32.847790 30350 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_16'
I20240503 08:12:32.926057 30252 consensus_queue.cc:1035] T 3fd92f707605425c92677dfc5f8358d4 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:32.930378 30252 consensus_queue.cc:1035] T 3fd92f707605425c92677dfc5f8358d4 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:33.090673 28199 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.65:39081, user_credentials={real_user=slave}} blocked reactor thread for 46268.6us
INSERT report
rows total: 2000
time total: 235.414 ms
time per row: 0.117707 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 08:12:34.099922 27856 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalance_test_table_17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:34.108578 28243 tablet_service.cc:1467] Processing CreateTablet for tablet cdad6ef53b6f413ebfda67f9b42c9c23 (DEFAULT_TABLE table=rebalance_test_table_17 [id=7e5325b51bd7410e93410447bfd72f9c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:34.108610 28112 tablet_service.cc:1467] Processing CreateTablet for tablet cdad6ef53b6f413ebfda67f9b42c9c23 (DEFAULT_TABLE table=rebalance_test_table_17 [id=7e5325b51bd7410e93410447bfd72f9c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:34.109004 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdad6ef53b6f413ebfda67f9b42c9c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:34.109012 28112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdad6ef53b6f413ebfda67f9b42c9c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:34.110097 28372 tablet_service.cc:1467] Processing CreateTablet for tablet cdad6ef53b6f413ebfda67f9b42c9c23 (DEFAULT_TABLE table=rebalance_test_table_17 [id=7e5325b51bd7410e93410447bfd72f9c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:34.110414 28113 tablet_service.cc:1467] Processing CreateTablet for tablet 41aa0e288ca4469ba34a07ff159ace91 (DEFAULT_TABLE table=rebalance_test_table_17 [id=7e5325b51bd7410e93410447bfd72f9c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:34.110515 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdad6ef53b6f413ebfda67f9b42c9c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:34.110805 28113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41aa0e288ca4469ba34a07ff159ace91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:34.113997 28242 tablet_service.cc:1467] Processing CreateTablet for tablet 41aa0e288ca4469ba34a07ff159ace91 (DEFAULT_TABLE table=rebalance_test_table_17 [id=7e5325b51bd7410e93410447bfd72f9c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:34.114391 28242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41aa0e288ca4469ba34a07ff159ace91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:34.114318 28111 tablet_service.cc:1467] Processing CreateTablet for tablet 0ceedfb43869465bb63931c39c28583a (DEFAULT_TABLE table=rebalance_test_table_17 [id=7e5325b51bd7410e93410447bfd72f9c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:34.114483 27983 tablet_service.cc:1467] Processing CreateTablet for tablet 0ceedfb43869465bb63931c39c28583a (DEFAULT_TABLE table=rebalance_test_table_17 [id=7e5325b51bd7410e93410447bfd72f9c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:34.114676 28111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ceedfb43869465bb63931c39c28583a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:34.114847 27983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ceedfb43869465bb63931c39c28583a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:34.118171 28373 tablet_service.cc:1467] Processing CreateTablet for tablet 41aa0e288ca4469ba34a07ff159ace91 (DEFAULT_TABLE table=rebalance_test_table_17 [id=7e5325b51bd7410e93410447bfd72f9c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:34.113972 28241 tablet_service.cc:1467] Processing CreateTablet for tablet 0ceedfb43869465bb63931c39c28583a (DEFAULT_TABLE table=rebalance_test_table_17 [id=7e5325b51bd7410e93410447bfd72f9c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:34.118501 28373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41aa0e288ca4469ba34a07ff159ace91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:34.118587 28241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ceedfb43869465bb63931c39c28583a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:34.129493 30429 tablet_bootstrap.cc:492] T 0ceedfb43869465bb63931c39c28583a P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:34.132683 30429 tablet_bootstrap.cc:654] T 0ceedfb43869465bb63931c39c28583a P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:34.135064 30431 tablet_bootstrap.cc:492] T cdad6ef53b6f413ebfda67f9b42c9c23 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:34.136328 30430 tablet_bootstrap.cc:492] T cdad6ef53b6f413ebfda67f9b42c9c23 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:34.136694 30431 tablet_bootstrap.cc:654] T cdad6ef53b6f413ebfda67f9b42c9c23 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:34.138063 30430 tablet_bootstrap.cc:654] T cdad6ef53b6f413ebfda67f9b42c9c23 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:34.138828 30428 tablet_bootstrap.cc:492] T cdad6ef53b6f413ebfda67f9b42c9c23 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:34.139096 30429 tablet_bootstrap.cc:492] T 0ceedfb43869465bb63931c39c28583a P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:34.139675 30429 ts_tablet_manager.cc:1367] T 0ceedfb43869465bb63931c39c28583a P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.003s
I20240503 08:12:34.140416 30429 raft_consensus.cc:348] T 0ceedfb43869465bb63931c39c28583a P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:34.140707 30429 raft_consensus.cc:374] T 0ceedfb43869465bb63931c39c28583a P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:34.140862 30429 raft_consensus.cc:729] T 0ceedfb43869465bb63931c39c28583a P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:34.141124 30429 consensus_queue.cc:260] T 0ceedfb43869465bb63931c39c28583a P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:34.141742 30429 ts_tablet_manager.cc:1398] T 0ceedfb43869465bb63931c39c28583a P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:34.142386 30428 tablet_bootstrap.cc:654] T cdad6ef53b6f413ebfda67f9b42c9c23 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:34.145104 30431 tablet_bootstrap.cc:492] T cdad6ef53b6f413ebfda67f9b42c9c23 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:34.145356 30431 ts_tablet_manager.cc:1367] T cdad6ef53b6f413ebfda67f9b42c9c23 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240503 08:12:34.146780 30428 tablet_bootstrap.cc:492] T cdad6ef53b6f413ebfda67f9b42c9c23 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:34.146958 30428 ts_tablet_manager.cc:1367] T cdad6ef53b6f413ebfda67f9b42c9c23 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240503 08:12:34.147544 30428 raft_consensus.cc:348] T cdad6ef53b6f413ebfda67f9b42c9c23 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:34.147720 30428 raft_consensus.cc:374] T cdad6ef53b6f413ebfda67f9b42c9c23 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:34.146968 30431 raft_consensus.cc:348] T cdad6ef53b6f413ebfda67f9b42c9c23 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:34.147805 30431 raft_consensus.cc:374] T cdad6ef53b6f413ebfda67f9b42c9c23 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:34.147830 30428 raft_consensus.cc:729] T cdad6ef53b6f413ebfda67f9b42c9c23 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:34.147945 30431 raft_consensus.cc:729] T cdad6ef53b6f413ebfda67f9b42c9c23 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:34.148046 30428 consensus_queue.cc:260] T cdad6ef53b6f413ebfda67f9b42c9c23 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:34.148171 30431 consensus_queue.cc:260] T cdad6ef53b6f413ebfda67f9b42c9c23 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:34.148846 30428 ts_tablet_manager.cc:1398] T cdad6ef53b6f413ebfda67f9b42c9c23 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:34.149114 30428 tablet_bootstrap.cc:492] T 41aa0e288ca4469ba34a07ff159ace91 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:34.149621 30431 ts_tablet_manager.cc:1398] T cdad6ef53b6f413ebfda67f9b42c9c23 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:34.149883 30431 tablet_bootstrap.cc:492] T 0ceedfb43869465bb63931c39c28583a P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:34.150542 30428 tablet_bootstrap.cc:654] T 41aa0e288ca4469ba34a07ff159ace91 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:34.151321 30431 tablet_bootstrap.cc:654] T 0ceedfb43869465bb63931c39c28583a P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:34.152912 30430 tablet_bootstrap.cc:492] T cdad6ef53b6f413ebfda67f9b42c9c23 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:34.153071 30430 ts_tablet_manager.cc:1367] T cdad6ef53b6f413ebfda67f9b42c9c23 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.005s
I20240503 08:12:34.154428 30430 raft_consensus.cc:348] T cdad6ef53b6f413ebfda67f9b42c9c23 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:34.154829 30430 raft_consensus.cc:374] T cdad6ef53b6f413ebfda67f9b42c9c23 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:34.154964 30430 raft_consensus.cc:729] T cdad6ef53b6f413ebfda67f9b42c9c23 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:34.155185 30430 consensus_queue.cc:260] T cdad6ef53b6f413ebfda67f9b42c9c23 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:34.155745 30430 ts_tablet_manager.cc:1398] T cdad6ef53b6f413ebfda67f9b42c9c23 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 08:12:34.156116 30430 tablet_bootstrap.cc:492] T 41aa0e288ca4469ba34a07ff159ace91 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:34.157562 30430 tablet_bootstrap.cc:654] T 41aa0e288ca4469ba34a07ff159ace91 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:34.159013 30431 tablet_bootstrap.cc:492] T 0ceedfb43869465bb63931c39c28583a P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:34.159149 30431 ts_tablet_manager.cc:1367] T 0ceedfb43869465bb63931c39c28583a P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20240503 08:12:34.159408 30430 tablet_bootstrap.cc:492] T 41aa0e288ca4469ba34a07ff159ace91 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:34.159590 30430 ts_tablet_manager.cc:1367] T 41aa0e288ca4469ba34a07ff159ace91 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 08:12:34.159801 30431 raft_consensus.cc:348] T 0ceedfb43869465bb63931c39c28583a P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:34.160012 30431 raft_consensus.cc:374] T 0ceedfb43869465bb63931c39c28583a P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:34.160094 30428 tablet_bootstrap.cc:492] T 41aa0e288ca4469ba34a07ff159ace91 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:34.160090 30431 raft_consensus.cc:729] T 0ceedfb43869465bb63931c39c28583a P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:34.160252 30428 ts_tablet_manager.cc:1367] T 41aa0e288ca4469ba34a07ff159ace91 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.002s
I20240503 08:12:34.160326 30431 consensus_queue.cc:260] T 0ceedfb43869465bb63931c39c28583a P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:34.160331 30430 raft_consensus.cc:348] T 41aa0e288ca4469ba34a07ff159ace91 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:34.160564 30430 raft_consensus.cc:374] T 41aa0e288ca4469ba34a07ff159ace91 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:34.160652 30430 raft_consensus.cc:729] T 41aa0e288ca4469ba34a07ff159ace91 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:34.160878 30430 consensus_queue.cc:260] T 41aa0e288ca4469ba34a07ff159ace91 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:34.161391 30430 ts_tablet_manager.cc:1398] T 41aa0e288ca4469ba34a07ff159ace91 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:12:34.161674 30431 ts_tablet_manager.cc:1398] T 0ceedfb43869465bb63931c39c28583a P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:34.161957 30431 tablet_bootstrap.cc:492] T 41aa0e288ca4469ba34a07ff159ace91 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:34.161216 30428 raft_consensus.cc:348] T 41aa0e288ca4469ba34a07ff159ace91 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:34.162478 30428 raft_consensus.cc:374] T 41aa0e288ca4469ba34a07ff159ace91 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:34.162561 30428 raft_consensus.cc:729] T 41aa0e288ca4469ba34a07ff159ace91 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:34.162811 30428 consensus_queue.cc:260] T 41aa0e288ca4469ba34a07ff159ace91 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:34.163287 30431 tablet_bootstrap.cc:654] T 41aa0e288ca4469ba34a07ff159ace91 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:34.163313 30428 ts_tablet_manager.cc:1398] T 41aa0e288ca4469ba34a07ff159ace91 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 08:12:34.164273 30428 tablet_bootstrap.cc:492] T 0ceedfb43869465bb63931c39c28583a P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:34.166044 30431 tablet_bootstrap.cc:492] T 41aa0e288ca4469ba34a07ff159ace91 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:34.166167 30431 ts_tablet_manager.cc:1367] T 41aa0e288ca4469ba34a07ff159ace91 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:12:34.167002 30431 raft_consensus.cc:348] T 41aa0e288ca4469ba34a07ff159ace91 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:34.167184 30431 raft_consensus.cc:374] T 41aa0e288ca4469ba34a07ff159ace91 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:34.167272 30431 raft_consensus.cc:729] T 41aa0e288ca4469ba34a07ff159ace91 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:34.167462 30431 consensus_queue.cc:260] T 41aa0e288ca4469ba34a07ff159ace91 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:34.168252 30428 tablet_bootstrap.cc:654] T 0ceedfb43869465bb63931c39c28583a P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:34.168612 30431 ts_tablet_manager.cc:1398] T 41aa0e288ca4469ba34a07ff159ace91 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:34.170033 30428 tablet_bootstrap.cc:492] T 0ceedfb43869465bb63931c39c28583a P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:34.170140 30428 ts_tablet_manager.cc:1367] T 0ceedfb43869465bb63931c39c28583a P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20240503 08:12:34.170614 30428 raft_consensus.cc:348] T 0ceedfb43869465bb63931c39c28583a P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:34.170751 30428 raft_consensus.cc:374] T 0ceedfb43869465bb63931c39c28583a P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:34.170836 30428 raft_consensus.cc:729] T 0ceedfb43869465bb63931c39c28583a P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:34.171027 30428 consensus_queue.cc:260] T 0ceedfb43869465bb63931c39c28583a P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:34.171566 30428 ts_tablet_manager.cc:1398] T 0ceedfb43869465bb63931c39c28583a P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:12:34.410300 30252 raft_consensus.cc:482] T cdad6ef53b6f413ebfda67f9b42c9c23 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:34.410501 30252 raft_consensus.cc:504] T cdad6ef53b6f413ebfda67f9b42c9c23 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:34.411134 30252 leader_election.cc:288] T cdad6ef53b6f413ebfda67f9b42c9c23 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:34.411489 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdad6ef53b6f413ebfda67f9b42c9c23" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:12:34.411482 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdad6ef53b6f413ebfda67f9b42c9c23" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:34.411674 28132 raft_consensus.cc:2454] T cdad6ef53b6f413ebfda67f9b42c9c23 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:12:34.411666 28391 raft_consensus.cc:2454] T cdad6ef53b6f413ebfda67f9b42c9c23 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:12:34.412057 28199 leader_election.cc:302] T cdad6ef53b6f413ebfda67f9b42c9c23 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:34.412312 30252 raft_consensus.cc:2781] T cdad6ef53b6f413ebfda67f9b42c9c23 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:34.412412 30252 raft_consensus.cc:482] T cdad6ef53b6f413ebfda67f9b42c9c23 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:34.412501 30252 raft_consensus.cc:3037] T cdad6ef53b6f413ebfda67f9b42c9c23 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:34.414368 30252 raft_consensus.cc:504] T cdad6ef53b6f413ebfda67f9b42c9c23 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:34.414973 30252 leader_election.cc:288] T cdad6ef53b6f413ebfda67f9b42c9c23 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Requested vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:34.415292 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdad6ef53b6f413ebfda67f9b42c9c23" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:12:34.415304 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdad6ef53b6f413ebfda67f9b42c9c23" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:12:34.415436 28132 raft_consensus.cc:3037] T cdad6ef53b6f413ebfda67f9b42c9c23 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:34.415470 28391 raft_consensus.cc:3037] T cdad6ef53b6f413ebfda67f9b42c9c23 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:34.417212 28132 raft_consensus.cc:2454] T cdad6ef53b6f413ebfda67f9b42c9c23 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:34.417198 28391 raft_consensus.cc:2454] T cdad6ef53b6f413ebfda67f9b42c9c23 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:34.417537 28199 leader_election.cc:302] T cdad6ef53b6f413ebfda67f9b42c9c23 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:34.418043 30252 raft_consensus.cc:2781] T cdad6ef53b6f413ebfda67f9b42c9c23 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:34.418205 30252 raft_consensus.cc:686] T cdad6ef53b6f413ebfda67f9b42c9c23 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Becoming Leader. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Running, Role: LEADER
I20240503 08:12:34.418432 30252 consensus_queue.cc:237] T cdad6ef53b6f413ebfda67f9b42c9c23 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:34.421463 27856 catalog_manager.cc:5484] T cdad6ef53b6f413ebfda67f9b42c9c23 P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: term changed from 0 to 1, leader changed from <none> to aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67). New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:12:34.451021 30252 raft_consensus.cc:482] T 41aa0e288ca4469ba34a07ff159ace91 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:34.451153 30252 raft_consensus.cc:504] T 41aa0e288ca4469ba34a07ff159ace91 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:34.451678 30252 leader_election.cc:288] T 41aa0e288ca4469ba34a07ff159ace91 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:34.452019 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41aa0e288ca4469ba34a07ff159ace91" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:34.452082 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41aa0e288ca4469ba34a07ff159ace91" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:12:34.452216 28391 raft_consensus.cc:2454] T 41aa0e288ca4469ba34a07ff159ace91 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:12:34.452306 28132 raft_consensus.cc:2454] T 41aa0e288ca4469ba34a07ff159ace91 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:12:34.452610 28199 leader_election.cc:302] T 41aa0e288ca4469ba34a07ff159ace91 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:34.452986 30252 raft_consensus.cc:2781] T 41aa0e288ca4469ba34a07ff159ace91 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:34.453079 30252 raft_consensus.cc:482] T 41aa0e288ca4469ba34a07ff159ace91 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:34.453163 30252 raft_consensus.cc:3037] T 41aa0e288ca4469ba34a07ff159ace91 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:34.454972 30252 raft_consensus.cc:504] T 41aa0e288ca4469ba34a07ff159ace91 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:34.455430 30252 leader_election.cc:288] T 41aa0e288ca4469ba34a07ff159ace91 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Requested vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:34.455794 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41aa0e288ca4469ba34a07ff159ace91" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:12:34.455773 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41aa0e288ca4469ba34a07ff159ace91" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:12:34.455977 28132 raft_consensus.cc:3037] T 41aa0e288ca4469ba34a07ff159ace91 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:34.455978 28391 raft_consensus.cc:3037] T 41aa0e288ca4469ba34a07ff159ace91 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:34.458622 28132 raft_consensus.cc:2454] T 41aa0e288ca4469ba34a07ff159ace91 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:34.458622 28391 raft_consensus.cc:2454] T 41aa0e288ca4469ba34a07ff159ace91 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:34.458956 28199 leader_election.cc:302] T 41aa0e288ca4469ba34a07ff159ace91 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:34.459183 30252 raft_consensus.cc:2781] T 41aa0e288ca4469ba34a07ff159ace91 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:34.459329 30252 raft_consensus.cc:686] T 41aa0e288ca4469ba34a07ff159ace91 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Becoming Leader. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Running, Role: LEADER
I20240503 08:12:34.459579 30252 consensus_queue.cc:237] T 41aa0e288ca4469ba34a07ff159ace91 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:34.462694 27856 catalog_manager.cc:5484] T 41aa0e288ca4469ba34a07ff159ace91 P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: term changed from 0 to 1, leader changed from <none> to aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67). New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:12:34.556543 30419 raft_consensus.cc:482] T 0ceedfb43869465bb63931c39c28583a P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:34.556721 30419 raft_consensus.cc:504] T 0ceedfb43869465bb63931c39c28583a P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:34.557322 30419 leader_election.cc:288] T 0ceedfb43869465bb63931c39c28583a P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:34.557652 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ceedfb43869465bb63931c39c28583a" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:12:34.557811 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ceedfb43869465bb63931c39c28583a" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:12:34.557896 28262 raft_consensus.cc:2454] T 0ceedfb43869465bb63931c39c28583a P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:34.558003 28132 raft_consensus.cc:2454] T 0ceedfb43869465bb63931c39c28583a P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:34.558375 27938 leader_election.cc:302] T 0ceedfb43869465bb63931c39c28583a P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 833582ff68c94269be364ebd187a75db; no voters:
I20240503 08:12:34.558696 30419 raft_consensus.cc:2781] T 0ceedfb43869465bb63931c39c28583a P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:34.558789 30419 raft_consensus.cc:482] T 0ceedfb43869465bb63931c39c28583a P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:34.558864 30419 raft_consensus.cc:3037] T 0ceedfb43869465bb63931c39c28583a P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:34.560678 30419 raft_consensus.cc:504] T 0ceedfb43869465bb63931c39c28583a P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:34.561170 30419 leader_election.cc:288] T 0ceedfb43869465bb63931c39c28583a P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:34.561548 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ceedfb43869465bb63931c39c28583a" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:12:34.561652 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ceedfb43869465bb63931c39c28583a" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:12:34.561728 28262 raft_consensus.cc:3037] T 0ceedfb43869465bb63931c39c28583a P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:34.561849 28132 raft_consensus.cc:3037] T 0ceedfb43869465bb63931c39c28583a P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:34.564422 28262 raft_consensus.cc:2454] T 0ceedfb43869465bb63931c39c28583a P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:34.564503 28132 raft_consensus.cc:2454] T 0ceedfb43869465bb63931c39c28583a P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:34.564819 27937 leader_election.cc:302] T 0ceedfb43869465bb63931c39c28583a P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:34.565101 30419 raft_consensus.cc:2781] T 0ceedfb43869465bb63931c39c28583a P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:34.565217 30419 raft_consensus.cc:686] T 0ceedfb43869465bb63931c39c28583a P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Becoming Leader. State: Replica: 833582ff68c94269be364ebd187a75db, State: Running, Role: LEADER
I20240503 08:12:34.565474 30419 consensus_queue.cc:237] T 0ceedfb43869465bb63931c39c28583a P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:34.568593 27856 catalog_manager.cc:5484] T 0ceedfb43869465bb63931c39c28583a P 833582ff68c94269be364ebd187a75db reported cstate change: term changed from 0 to 1, leader changed from <none> to 833582ff68c94269be364ebd187a75db (127.20.230.65). New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:12:34.885746 30422 consensus_queue.cc:1035] T cdad6ef53b6f413ebfda67f9b42c9c23 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:34.891512 30252 consensus_queue.cc:1035] T cdad6ef53b6f413ebfda67f9b42c9c23 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:34.906679 30445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:34.907029 30445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:34.914343 30445 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_17'
I20240503 08:12:34.957437 30419 consensus_queue.cc:1035] T 0ceedfb43869465bb63931c39c28583a P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:34.965565 30346 consensus_queue.cc:1035] T 0ceedfb43869465bb63931c39c28583a P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:34.972492 30252 consensus_queue.cc:1035] T 41aa0e288ca4469ba34a07ff159ace91 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:34.978055 30422 consensus_queue.cc:1035] T 41aa0e288ca4469ba34a07ff159ace91 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 210.332 ms
time per row: 0.105166 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 08:12:36.159967 27860 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalance_test_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:36.169054 27983 tablet_service.cc:1467] Processing CreateTablet for tablet 479beba3dd3e42faa60c598ad74c7041 (DEFAULT_TABLE table=rebalance_test_table_18 [id=1cae04951c764544be14f32e3cbc0ed0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:36.169519 27983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 479beba3dd3e42faa60c598ad74c7041. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:36.170112 27982 tablet_service.cc:1467] Processing CreateTablet for tablet 7c3e836df884424e99a892a2942faebe (DEFAULT_TABLE table=rebalance_test_table_18 [id=1cae04951c764544be14f32e3cbc0ed0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:36.170506 27982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c3e836df884424e99a892a2942faebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:36.170547 28241 tablet_service.cc:1467] Processing CreateTablet for tablet 7c3e836df884424e99a892a2942faebe (DEFAULT_TABLE table=rebalance_test_table_18 [id=1cae04951c764544be14f32e3cbc0ed0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:36.170874 28241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c3e836df884424e99a892a2942faebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:36.174834 28112 tablet_service.cc:1467] Processing CreateTablet for tablet 479beba3dd3e42faa60c598ad74c7041 (DEFAULT_TABLE table=rebalance_test_table_18 [id=1cae04951c764544be14f32e3cbc0ed0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:36.175251 28112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 479beba3dd3e42faa60c598ad74c7041. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:36.174826 28111 tablet_service.cc:1467] Processing CreateTablet for tablet 66c9df9a0fa24db49a6120cc9fe1bc77 (DEFAULT_TABLE table=rebalance_test_table_18 [id=1cae04951c764544be14f32e3cbc0ed0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:36.175858 28111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66c9df9a0fa24db49a6120cc9fe1bc77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:36.176462 28372 tablet_service.cc:1467] Processing CreateTablet for tablet 479beba3dd3e42faa60c598ad74c7041 (DEFAULT_TABLE table=rebalance_test_table_18 [id=1cae04951c764544be14f32e3cbc0ed0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:36.176856 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 479beba3dd3e42faa60c598ad74c7041. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:36.181408 28371 tablet_service.cc:1467] Processing CreateTablet for tablet 66c9df9a0fa24db49a6120cc9fe1bc77 (DEFAULT_TABLE table=rebalance_test_table_18 [id=1cae04951c764544be14f32e3cbc0ed0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:36.181582 28373 tablet_service.cc:1467] Processing CreateTablet for tablet 7c3e836df884424e99a892a2942faebe (DEFAULT_TABLE table=rebalance_test_table_18 [id=1cae04951c764544be14f32e3cbc0ed0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:36.181825 28371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66c9df9a0fa24db49a6120cc9fe1bc77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:36.182085 28373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c3e836df884424e99a892a2942faebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:36.183713 30517 tablet_bootstrap.cc:492] T 7c3e836df884424e99a892a2942faebe P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:36.185636 30517 tablet_bootstrap.cc:654] T 7c3e836df884424e99a892a2942faebe P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:36.189983 30518 tablet_bootstrap.cc:492] T 479beba3dd3e42faa60c598ad74c7041 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:36.191556 30518 tablet_bootstrap.cc:654] T 479beba3dd3e42faa60c598ad74c7041 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:36.190915 27982 tablet_service.cc:1467] Processing CreateTablet for tablet 66c9df9a0fa24db49a6120cc9fe1bc77 (DEFAULT_TABLE table=rebalance_test_table_18 [id=1cae04951c764544be14f32e3cbc0ed0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:36.192638 27982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66c9df9a0fa24db49a6120cc9fe1bc77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:36.195887 30516 tablet_bootstrap.cc:492] T 479beba3dd3e42faa60c598ad74c7041 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:36.197165 30517 tablet_bootstrap.cc:492] T 7c3e836df884424e99a892a2942faebe P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:36.197402 30517 ts_tablet_manager.cc:1367] T 7c3e836df884424e99a892a2942faebe P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20240503 08:12:36.197664 30516 tablet_bootstrap.cc:654] T 479beba3dd3e42faa60c598ad74c7041 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:36.201180 30517 raft_consensus.cc:348] T 7c3e836df884424e99a892a2942faebe P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:36.201553 30517 raft_consensus.cc:374] T 7c3e836df884424e99a892a2942faebe P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:36.202553 30520 tablet_bootstrap.cc:492] T 7c3e836df884424e99a892a2942faebe P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:36.203260 30518 tablet_bootstrap.cc:492] T 479beba3dd3e42faa60c598ad74c7041 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:36.203521 30518 ts_tablet_manager.cc:1367] T 479beba3dd3e42faa60c598ad74c7041 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20240503 08:12:36.204196 30520 tablet_bootstrap.cc:654] T 7c3e836df884424e99a892a2942faebe P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:36.204933 30518 raft_consensus.cc:348] T 479beba3dd3e42faa60c598ad74c7041 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:36.205190 30518 raft_consensus.cc:374] T 479beba3dd3e42faa60c598ad74c7041 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:36.205325 30518 raft_consensus.cc:729] T 479beba3dd3e42faa60c598ad74c7041 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:36.205610 30518 consensus_queue.cc:260] T 479beba3dd3e42faa60c598ad74c7041 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:36.206259 30518 ts_tablet_manager.cc:1398] T 479beba3dd3e42faa60c598ad74c7041 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:12:36.207433 30516 tablet_bootstrap.cc:492] T 479beba3dd3e42faa60c598ad74c7041 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:36.207693 30516 ts_tablet_manager.cc:1367] T 479beba3dd3e42faa60c598ad74c7041 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.002s
I20240503 08:12:36.201683 30517 raft_consensus.cc:729] T 7c3e836df884424e99a892a2942faebe P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:36.208549 30516 raft_consensus.cc:348] T 479beba3dd3e42faa60c598ad74c7041 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:36.209906 30516 raft_consensus.cc:374] T 479beba3dd3e42faa60c598ad74c7041 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:36.209858 30517 consensus_queue.cc:260] T 7c3e836df884424e99a892a2942faebe P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:36.210027 30516 raft_consensus.cc:729] T 479beba3dd3e42faa60c598ad74c7041 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:36.210450 30517 ts_tablet_manager.cc:1398] T 7c3e836df884424e99a892a2942faebe P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20240503 08:12:36.210376 30516 consensus_queue.cc:260] T 479beba3dd3e42faa60c598ad74c7041 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:36.212011 30516 ts_tablet_manager.cc:1398] T 479beba3dd3e42faa60c598ad74c7041 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 08:12:36.213253 30516 tablet_bootstrap.cc:492] T 7c3e836df884424e99a892a2942faebe P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:36.213850 30518 tablet_bootstrap.cc:492] T 66c9df9a0fa24db49a6120cc9fe1bc77 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:36.218745 30520 tablet_bootstrap.cc:492] T 7c3e836df884424e99a892a2942faebe P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:36.218930 30520 ts_tablet_manager.cc:1367] T 7c3e836df884424e99a892a2942faebe P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.007s
I20240503 08:12:36.219630 30520 raft_consensus.cc:348] T 7c3e836df884424e99a892a2942faebe P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:36.219900 30520 raft_consensus.cc:374] T 7c3e836df884424e99a892a2942faebe P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:36.220005 30520 raft_consensus.cc:729] T 7c3e836df884424e99a892a2942faebe P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:36.220254 30520 consensus_queue.cc:260] T 7c3e836df884424e99a892a2942faebe P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:36.220829 30520 ts_tablet_manager.cc:1398] T 7c3e836df884424e99a892a2942faebe P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:12:36.221093 30520 tablet_bootstrap.cc:492] T 66c9df9a0fa24db49a6120cc9fe1bc77 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:36.221298 30518 tablet_bootstrap.cc:654] T 66c9df9a0fa24db49a6120cc9fe1bc77 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:36.222615 30520 tablet_bootstrap.cc:654] T 66c9df9a0fa24db49a6120cc9fe1bc77 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:36.223683 30518 tablet_bootstrap.cc:492] T 66c9df9a0fa24db49a6120cc9fe1bc77 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:36.223909 30516 tablet_bootstrap.cc:654] T 7c3e836df884424e99a892a2942faebe P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:36.224450 30520 tablet_bootstrap.cc:492] T 66c9df9a0fa24db49a6120cc9fe1bc77 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:36.224587 30520 ts_tablet_manager.cc:1367] T 66c9df9a0fa24db49a6120cc9fe1bc77 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:36.225170 30520 raft_consensus.cc:348] T 66c9df9a0fa24db49a6120cc9fe1bc77 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:36.225345 30520 raft_consensus.cc:374] T 66c9df9a0fa24db49a6120cc9fe1bc77 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:36.225417 30520 raft_consensus.cc:729] T 66c9df9a0fa24db49a6120cc9fe1bc77 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:36.225625 30520 consensus_queue.cc:260] T 66c9df9a0fa24db49a6120cc9fe1bc77 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:36.225682 30518 ts_tablet_manager.cc:1367] T 66c9df9a0fa24db49a6120cc9fe1bc77 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.002s
I20240503 08:12:36.226235 30520 ts_tablet_manager.cc:1398] T 66c9df9a0fa24db49a6120cc9fe1bc77 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:36.226534 30518 raft_consensus.cc:348] T 66c9df9a0fa24db49a6120cc9fe1bc77 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:36.226760 30518 raft_consensus.cc:374] T 66c9df9a0fa24db49a6120cc9fe1bc77 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:36.226846 30518 raft_consensus.cc:729] T 66c9df9a0fa24db49a6120cc9fe1bc77 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:36.227312 30520 tablet_bootstrap.cc:492] T 479beba3dd3e42faa60c598ad74c7041 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:36.228737 30520 tablet_bootstrap.cc:654] T 479beba3dd3e42faa60c598ad74c7041 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:36.229830 30454 raft_consensus.cc:482] T 7c3e836df884424e99a892a2942faebe P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:36.229954 30454 raft_consensus.cc:504] T 7c3e836df884424e99a892a2942faebe P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:36.230585 30454 leader_election.cc:288] T 7c3e836df884424e99a892a2942faebe P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:12:36.230734 30520 tablet_bootstrap.cc:492] T 479beba3dd3e42faa60c598ad74c7041 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:36.230857 30520 ts_tablet_manager.cc:1367] T 479beba3dd3e42faa60c598ad74c7041 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:36.231683 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c3e836df884424e99a892a2942faebe" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:12:36.231904 30520 raft_consensus.cc:348] T 479beba3dd3e42faa60c598ad74c7041 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:36.232270 30520 raft_consensus.cc:374] T 479beba3dd3e42faa60c598ad74c7041 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240503 08:12:36.232265 28329 leader_election.cc:341] T 7c3e836df884424e99a892a2942faebe P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:36.232481 30520 raft_consensus.cc:729] T 479beba3dd3e42faa60c598ad74c7041 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:36.233038 28261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c3e836df884424e99a892a2942faebe" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:12:36.233172 30516 tablet_bootstrap.cc:492] T 7c3e836df884424e99a892a2942faebe P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:36.233256 28261 raft_consensus.cc:2454] T 7c3e836df884424e99a892a2942faebe P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:12:36.233311 30516 ts_tablet_manager.cc:1367] T 7c3e836df884424e99a892a2942faebe P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.000s
I20240503 08:12:36.233592 30518 consensus_queue.cc:260] T 66c9df9a0fa24db49a6120cc9fe1bc77 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:36.233690 30520 consensus_queue.cc:260] T 479beba3dd3e42faa60c598ad74c7041 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:36.234067 30516 raft_consensus.cc:348] T 7c3e836df884424e99a892a2942faebe P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:36.234216 30520 ts_tablet_manager.cc:1398] T 479beba3dd3e42faa60c598ad74c7041 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:36.234288 30518 ts_tablet_manager.cc:1398] T 66c9df9a0fa24db49a6120cc9fe1bc77 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20240503 08:12:36.234293 30516 raft_consensus.cc:374] T 7c3e836df884424e99a892a2942faebe P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:36.234362 30516 raft_consensus.cc:729] T 7c3e836df884424e99a892a2942faebe P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:36.234599 30516 consensus_queue.cc:260] T 7c3e836df884424e99a892a2942faebe P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:36.235749 28327 leader_election.cc:302] T 7c3e836df884424e99a892a2942faebe P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, d09a11a03989421b9f28019dc1c14cc2; no voters: 833582ff68c94269be364ebd187a75db
I20240503 08:12:36.235905 30516 ts_tablet_manager.cc:1398] T 7c3e836df884424e99a892a2942faebe P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:12:36.236162 30516 tablet_bootstrap.cc:492] T 66c9df9a0fa24db49a6120cc9fe1bc77 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:36.236196 30454 raft_consensus.cc:2781] T 7c3e836df884424e99a892a2942faebe P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:36.236316 30454 raft_consensus.cc:482] T 7c3e836df884424e99a892a2942faebe P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:36.236392 30454 raft_consensus.cc:3037] T 7c3e836df884424e99a892a2942faebe P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:36.237675 30516 tablet_bootstrap.cc:654] T 66c9df9a0fa24db49a6120cc9fe1bc77 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:36.238703 30454 raft_consensus.cc:504] T 7c3e836df884424e99a892a2942faebe P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:36.239257 30454 leader_election.cc:288] T 7c3e836df884424e99a892a2942faebe P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:12:36.239392 30516 tablet_bootstrap.cc:492] T 66c9df9a0fa24db49a6120cc9fe1bc77 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:36.239620 30516 ts_tablet_manager.cc:1367] T 66c9df9a0fa24db49a6120cc9fe1bc77 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:12:36.240432 28261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c3e836df884424e99a892a2942faebe" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:12:36.240655 28261 raft_consensus.cc:3037] T 7c3e836df884424e99a892a2942faebe P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:36.244531 28261 raft_consensus.cc:2454] T 7c3e836df884424e99a892a2942faebe P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:12:36.244800 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c3e836df884424e99a892a2942faebe" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:12:36.245183 28002 raft_consensus.cc:3037] T 7c3e836df884424e99a892a2942faebe P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:36.245152 30516 raft_consensus.cc:348] T 66c9df9a0fa24db49a6120cc9fe1bc77 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:36.245350 30516 raft_consensus.cc:374] T 66c9df9a0fa24db49a6120cc9fe1bc77 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:36.245434 30516 raft_consensus.cc:729] T 66c9df9a0fa24db49a6120cc9fe1bc77 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:36.245640 30516 consensus_queue.cc:260] T 66c9df9a0fa24db49a6120cc9fe1bc77 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:36.245677 28327 leader_election.cc:302] T 7c3e836df884424e99a892a2942faebe P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:36.246322 30516 ts_tablet_manager.cc:1398] T 66c9df9a0fa24db49a6120cc9fe1bc77 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:12:36.247467 28002 raft_consensus.cc:2454] T 7c3e836df884424e99a892a2942faebe P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:12:36.247884 30523 raft_consensus.cc:2781] T 7c3e836df884424e99a892a2942faebe P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:36.248813 30523 raft_consensus.cc:686] T 7c3e836df884424e99a892a2942faebe P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Becoming Leader. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Running, Role: LEADER
I20240503 08:12:36.249101 30523 consensus_queue.cc:237] T 7c3e836df884424e99a892a2942faebe P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:36.252663 27860 catalog_manager.cc:5484] T 7c3e836df884424e99a892a2942faebe P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68). New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:12:36.269850 30511 raft_consensus.cc:482] T 479beba3dd3e42faa60c598ad74c7041 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:36.269991 30511 raft_consensus.cc:504] T 479beba3dd3e42faa60c598ad74c7041 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:36.270522 30511 leader_election.cc:288] T 479beba3dd3e42faa60c598ad74c7041 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:36.270964 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "479beba3dd3e42faa60c598ad74c7041" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:12:36.271034 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "479beba3dd3e42faa60c598ad74c7041" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:36.271162 28002 raft_consensus.cc:2454] T 479beba3dd3e42faa60c598ad74c7041 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:36.271212 28391 raft_consensus.cc:2454] T 479beba3dd3e42faa60c598ad74c7041 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:36.271646 28069 leader_election.cc:302] T 479beba3dd3e42faa60c598ad74c7041 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 833582ff68c94269be364ebd187a75db; no voters:
I20240503 08:12:36.271915 30511 raft_consensus.cc:2781] T 479beba3dd3e42faa60c598ad74c7041 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:36.272046 30511 raft_consensus.cc:482] T 479beba3dd3e42faa60c598ad74c7041 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:36.272138 30511 raft_consensus.cc:3037] T 479beba3dd3e42faa60c598ad74c7041 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:36.274037 30511 raft_consensus.cc:504] T 479beba3dd3e42faa60c598ad74c7041 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:36.274567 30511 leader_election.cc:288] T 479beba3dd3e42faa60c598ad74c7041 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Requested vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:36.274924 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "479beba3dd3e42faa60c598ad74c7041" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:12:36.274945 28392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "479beba3dd3e42faa60c598ad74c7041" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:12:36.275075 28002 raft_consensus.cc:3037] T 479beba3dd3e42faa60c598ad74c7041 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:36.275101 28392 raft_consensus.cc:3037] T 479beba3dd3e42faa60c598ad74c7041 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:36.276808 28392 raft_consensus.cc:2454] T 479beba3dd3e42faa60c598ad74c7041 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:12:36.276816 28002 raft_consensus.cc:2454] T 479beba3dd3e42faa60c598ad74c7041 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:12:36.277129 28069 leader_election.cc:302] T 479beba3dd3e42faa60c598ad74c7041 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:36.277380 30511 raft_consensus.cc:2781] T 479beba3dd3e42faa60c598ad74c7041 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:36.277500 30511 raft_consensus.cc:686] T 479beba3dd3e42faa60c598ad74c7041 P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Becoming Leader. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Running, Role: LEADER
I20240503 08:12:36.277729 30511 consensus_queue.cc:237] T 479beba3dd3e42faa60c598ad74c7041 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:36.280644 27860 catalog_manager.cc:5484] T 479beba3dd3e42faa60c598ad74c7041 P 1b411c6bb25549368041773d7b710d48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b411c6bb25549368041773d7b710d48 (127.20.230.66). New cstate: current_term: 1 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:12:36.345451 30454 raft_consensus.cc:482] T 66c9df9a0fa24db49a6120cc9fe1bc77 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:36.345618 30454 raft_consensus.cc:504] T 66c9df9a0fa24db49a6120cc9fe1bc77 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:36.346379 30454 leader_election.cc:288] T 66c9df9a0fa24db49a6120cc9fe1bc77 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:36.346621 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66c9df9a0fa24db49a6120cc9fe1bc77" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:12:36.346743 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66c9df9a0fa24db49a6120cc9fe1bc77" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:12:36.346858 28002 raft_consensus.cc:2454] T 66c9df9a0fa24db49a6120cc9fe1bc77 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:12:36.346976 28132 raft_consensus.cc:2454] T 66c9df9a0fa24db49a6120cc9fe1bc77 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:12:36.347205 28329 leader_election.cc:302] T 66c9df9a0fa24db49a6120cc9fe1bc77 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:36.347476 30454 raft_consensus.cc:2781] T 66c9df9a0fa24db49a6120cc9fe1bc77 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:36.347558 30454 raft_consensus.cc:482] T 66c9df9a0fa24db49a6120cc9fe1bc77 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:36.347618 30454 raft_consensus.cc:3037] T 66c9df9a0fa24db49a6120cc9fe1bc77 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:36.349380 30454 raft_consensus.cc:504] T 66c9df9a0fa24db49a6120cc9fe1bc77 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:36.349926 30454 leader_election.cc:288] T 66c9df9a0fa24db49a6120cc9fe1bc77 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Requested vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:36.350335 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66c9df9a0fa24db49a6120cc9fe1bc77" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:12:36.350524 28132 raft_consensus.cc:3037] T 66c9df9a0fa24db49a6120cc9fe1bc77 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:36.352526 28132 raft_consensus.cc:2454] T 66c9df9a0fa24db49a6120cc9fe1bc77 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:12:36.352814 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66c9df9a0fa24db49a6120cc9fe1bc77" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:12:36.352981 28002 raft_consensus.cc:3037] T 66c9df9a0fa24db49a6120cc9fe1bc77 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:36.354183 28328 leader_election.cc:302] T 66c9df9a0fa24db49a6120cc9fe1bc77 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:36.354457 30530 raft_consensus.cc:2781] T 66c9df9a0fa24db49a6120cc9fe1bc77 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:36.354586 30530 raft_consensus.cc:686] T 66c9df9a0fa24db49a6120cc9fe1bc77 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Becoming Leader. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Running, Role: LEADER
I20240503 08:12:36.355175 28002 raft_consensus.cc:2454] T 66c9df9a0fa24db49a6120cc9fe1bc77 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:12:36.355228 30530 consensus_queue.cc:237] T 66c9df9a0fa24db49a6120cc9fe1bc77 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:36.359870 27860 catalog_manager.cc:5484] T 66c9df9a0fa24db49a6120cc9fe1bc77 P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68). New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:12:36.696573 30530 consensus_queue.cc:1035] T 7c3e836df884424e99a892a2942faebe P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:36.708117 30454 consensus_queue.cc:1035] T 7c3e836df884424e99a892a2942faebe P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:36.730281 30535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:36.730645 30535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:36.738556 30535 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:12:36.742347 30454 consensus_queue.cc:1035] T 66c9df9a0fa24db49a6120cc9fe1bc77 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:36.747670 30530 consensus_queue.cc:1035] T 66c9df9a0fa24db49a6120cc9fe1bc77 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_18'
I20240503 08:12:36.792645 30511 consensus_queue.cc:1035] T 479beba3dd3e42faa60c598ad74c7041 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:36.813496 30511 consensus_queue.cc:1035] T 479beba3dd3e42faa60c598ad74c7041 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 242.118 ms
time per row: 0.121059 ms
I20240503 08:12:37.140705 27862 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalance_test_table_19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:37.149071 28241 tablet_service.cc:1467] Processing CreateTablet for tablet 08fb64977309479b831635344969c662 (DEFAULT_TABLE table=rebalance_test_table_19 [id=18d795e9c19046fdba033edcea369490]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:37.149380 27982 tablet_service.cc:1467] Processing CreateTablet for tablet 08fb64977309479b831635344969c662 (DEFAULT_TABLE table=rebalance_test_table_19 [id=18d795e9c19046fdba033edcea369490]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:37.149730 27982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08fb64977309479b831635344969c662. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:37.151181 27983 tablet_service.cc:1467] Processing CreateTablet for tablet bdfe68a127f043f08f04749843a38375 (DEFAULT_TABLE table=rebalance_test_table_19 [id=18d795e9c19046fdba033edcea369490]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:37.151582 27983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdfe68a127f043f08f04749843a38375. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:37.151787 28242 tablet_service.cc:1467] Processing CreateTablet for tablet bdfe68a127f043f08f04749843a38375 (DEFAULT_TABLE table=rebalance_test_table_19 [id=18d795e9c19046fdba033edcea369490]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:37.152143 28241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08fb64977309479b831635344969c662. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:37.152618 28243 tablet_service.cc:1467] Processing CreateTablet for tablet 978bf5783b0e47e3bfb30441adc600ae (DEFAULT_TABLE table=rebalance_test_table_19 [id=18d795e9c19046fdba033edcea369490]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:37.152978 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 978bf5783b0e47e3bfb30441adc600ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:37.153492 28372 tablet_service.cc:1467] Processing CreateTablet for tablet 978bf5783b0e47e3bfb30441adc600ae (DEFAULT_TABLE table=rebalance_test_table_19 [id=18d795e9c19046fdba033edcea369490]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:37.153896 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 978bf5783b0e47e3bfb30441adc600ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:37.156777 28242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdfe68a127f043f08f04749843a38375. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:37.158072 28373 tablet_service.cc:1467] Processing CreateTablet for tablet bdfe68a127f043f08f04749843a38375 (DEFAULT_TABLE table=rebalance_test_table_19 [id=18d795e9c19046fdba033edcea369490]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:37.158424 28373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdfe68a127f043f08f04749843a38375. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:37.150681 28112 tablet_service.cc:1467] Processing CreateTablet for tablet 08fb64977309479b831635344969c662 (DEFAULT_TABLE table=rebalance_test_table_19 [id=18d795e9c19046fdba033edcea369490]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:37.158741 28112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08fb64977309479b831635344969c662. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:37.161244 28111 tablet_service.cc:1467] Processing CreateTablet for tablet 978bf5783b0e47e3bfb30441adc600ae (DEFAULT_TABLE table=rebalance_test_table_19 [id=18d795e9c19046fdba033edcea369490]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:37.161403 30596 tablet_bootstrap.cc:492] T bdfe68a127f043f08f04749843a38375 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:37.161589 28111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 978bf5783b0e47e3bfb30441adc600ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:37.163266 30596 tablet_bootstrap.cc:654] T bdfe68a127f043f08f04749843a38375 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:37.169135 30599 tablet_bootstrap.cc:492] T 08fb64977309479b831635344969c662 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:37.171500 30599 tablet_bootstrap.cc:654] T 08fb64977309479b831635344969c662 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:37.172060 30600 tablet_bootstrap.cc:492] T 08fb64977309479b831635344969c662 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:37.173820 30600 tablet_bootstrap.cc:654] T 08fb64977309479b831635344969c662 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:37.175848 30596 tablet_bootstrap.cc:492] T bdfe68a127f043f08f04749843a38375 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:37.176048 30596 ts_tablet_manager.cc:1367] T bdfe68a127f043f08f04749843a38375 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.000s
I20240503 08:12:37.176841 30596 raft_consensus.cc:348] T bdfe68a127f043f08f04749843a38375 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:37.177325 30598 tablet_bootstrap.cc:492] T bdfe68a127f043f08f04749843a38375 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:37.177106 30596 raft_consensus.cc:374] T bdfe68a127f043f08f04749843a38375 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:37.177824 30596 raft_consensus.cc:729] T bdfe68a127f043f08f04749843a38375 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:37.178197 30596 consensus_queue.cc:260] T bdfe68a127f043f08f04749843a38375 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:37.179148 30598 tablet_bootstrap.cc:654] T bdfe68a127f043f08f04749843a38375 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:37.179191 30600 tablet_bootstrap.cc:492] T 08fb64977309479b831635344969c662 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:37.179311 30600 ts_tablet_manager.cc:1367] T 08fb64977309479b831635344969c662 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240503 08:12:37.179999 30600 raft_consensus.cc:348] T 08fb64977309479b831635344969c662 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:37.180256 30600 raft_consensus.cc:374] T 08fb64977309479b831635344969c662 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:37.180356 30600 raft_consensus.cc:729] T 08fb64977309479b831635344969c662 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:37.180593 30600 consensus_queue.cc:260] T 08fb64977309479b831635344969c662 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:37.181164 30600 ts_tablet_manager.cc:1398] T 08fb64977309479b831635344969c662 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:37.181413 30600 tablet_bootstrap.cc:492] T bdfe68a127f043f08f04749843a38375 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:37.182929 30600 tablet_bootstrap.cc:654] T bdfe68a127f043f08f04749843a38375 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:37.183261 30596 ts_tablet_manager.cc:1398] T bdfe68a127f043f08f04749843a38375 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240503 08:12:37.184700 30598 tablet_bootstrap.cc:492] T bdfe68a127f043f08f04749843a38375 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:37.184871 30598 ts_tablet_manager.cc:1367] T bdfe68a127f043f08f04749843a38375 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240503 08:12:37.185755 30598 raft_consensus.cc:348] T bdfe68a127f043f08f04749843a38375 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:37.186059 30598 raft_consensus.cc:374] T bdfe68a127f043f08f04749843a38375 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:37.186183 30598 raft_consensus.cc:729] T bdfe68a127f043f08f04749843a38375 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:37.186467 30598 consensus_queue.cc:260] T bdfe68a127f043f08f04749843a38375 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:37.184877 30599 tablet_bootstrap.cc:492] T 08fb64977309479b831635344969c662 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:37.187093 30599 ts_tablet_manager.cc:1367] T 08fb64977309479b831635344969c662 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.005s
I20240503 08:12:37.187103 30598 ts_tablet_manager.cc:1398] T bdfe68a127f043f08f04749843a38375 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:37.187857 30599 raft_consensus.cc:348] T 08fb64977309479b831635344969c662 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:37.188103 30599 raft_consensus.cc:374] T 08fb64977309479b831635344969c662 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:37.188232 30599 raft_consensus.cc:729] T 08fb64977309479b831635344969c662 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:37.188437 30598 tablet_bootstrap.cc:492] T 978bf5783b0e47e3bfb30441adc600ae P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:37.188517 30599 consensus_queue.cc:260] T 08fb64977309479b831635344969c662 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:37.189153 30599 ts_tablet_manager.cc:1398] T 08fb64977309479b831635344969c662 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:37.189954 30598 tablet_bootstrap.cc:654] T 978bf5783b0e47e3bfb30441adc600ae P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:37.190488 30599 tablet_bootstrap.cc:492] T 978bf5783b0e47e3bfb30441adc600ae P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:37.194000 30599 tablet_bootstrap.cc:654] T 978bf5783b0e47e3bfb30441adc600ae P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:37.200173 30596 tablet_bootstrap.cc:492] T 08fb64977309479b831635344969c662 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:37.202492 30596 tablet_bootstrap.cc:654] T 08fb64977309479b831635344969c662 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:37.208782 30600 tablet_bootstrap.cc:492] T bdfe68a127f043f08f04749843a38375 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:37.209015 30600 ts_tablet_manager.cc:1367] T bdfe68a127f043f08f04749843a38375 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.028s user 0.013s sys 0.009s
I20240503 08:12:37.209414 30598 tablet_bootstrap.cc:492] T 978bf5783b0e47e3bfb30441adc600ae P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:37.209607 30598 ts_tablet_manager.cc:1367] T 978bf5783b0e47e3bfb30441adc600ae P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.008s
I20240503 08:12:37.210458 30598 raft_consensus.cc:348] T 978bf5783b0e47e3bfb30441adc600ae P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:37.210716 30598 raft_consensus.cc:374] T 978bf5783b0e47e3bfb30441adc600ae P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:37.210834 30598 raft_consensus.cc:729] T 978bf5783b0e47e3bfb30441adc600ae P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:37.211112 30598 consensus_queue.cc:260] T 978bf5783b0e47e3bfb30441adc600ae P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:37.209883 30600 raft_consensus.cc:348] T bdfe68a127f043f08f04749843a38375 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:37.211426 30600 raft_consensus.cc:374] T bdfe68a127f043f08f04749843a38375 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:37.211572 30600 raft_consensus.cc:729] T bdfe68a127f043f08f04749843a38375 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:37.211894 30600 consensus_queue.cc:260] T bdfe68a127f043f08f04749843a38375 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:37.212473 30600 ts_tablet_manager.cc:1398] T bdfe68a127f043f08f04749843a38375 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:37.212635 30598 ts_tablet_manager.cc:1398] T 978bf5783b0e47e3bfb30441adc600ae P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:12:37.218000 30596 tablet_bootstrap.cc:492] T 08fb64977309479b831635344969c662 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:37.218158 30596 ts_tablet_manager.cc:1367] T 08fb64977309479b831635344969c662 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.007s
I20240503 08:12:37.218644 30599 tablet_bootstrap.cc:492] T 978bf5783b0e47e3bfb30441adc600ae P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:37.218842 30599 ts_tablet_manager.cc:1367] T 978bf5783b0e47e3bfb30441adc600ae P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.001s
I20240503 08:12:37.218888 30596 raft_consensus.cc:348] T 08fb64977309479b831635344969c662 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:37.219090 30596 raft_consensus.cc:374] T 08fb64977309479b831635344969c662 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:37.219170 30596 raft_consensus.cc:729] T 08fb64977309479b831635344969c662 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:37.219424 30596 consensus_queue.cc:260] T 08fb64977309479b831635344969c662 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:37.219949 30596 ts_tablet_manager.cc:1398] T 08fb64977309479b831635344969c662 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:37.219635 30599 raft_consensus.cc:348] T 978bf5783b0e47e3bfb30441adc600ae P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:37.220145 30599 raft_consensus.cc:374] T 978bf5783b0e47e3bfb30441adc600ae P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:37.220197 30531 raft_consensus.cc:482] T bdfe68a127f043f08f04749843a38375 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:37.220239 30599 raft_consensus.cc:729] T 978bf5783b0e47e3bfb30441adc600ae P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:37.220299 30596 tablet_bootstrap.cc:492] T 978bf5783b0e47e3bfb30441adc600ae P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:37.220335 30531 raft_consensus.cc:504] T bdfe68a127f043f08f04749843a38375 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:37.220455 30599 consensus_queue.cc:260] T 978bf5783b0e47e3bfb30441adc600ae P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:37.220995 30599 ts_tablet_manager.cc:1398] T 978bf5783b0e47e3bfb30441adc600ae P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:37.221118 30531 leader_election.cc:288] T bdfe68a127f043f08f04749843a38375 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:12:37.221711 30596 tablet_bootstrap.cc:654] T 978bf5783b0e47e3bfb30441adc600ae P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:37.221814 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdfe68a127f043f08f04749843a38375" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:12:37.222005 28003 raft_consensus.cc:2454] T bdfe68a127f043f08f04749843a38375 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:12:37.223438 28329 leader_election.cc:302] T bdfe68a127f043f08f04749843a38375 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:37.227603 30531 raft_consensus.cc:2781] T bdfe68a127f043f08f04749843a38375 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:37.227705 30531 raft_consensus.cc:482] T bdfe68a127f043f08f04749843a38375 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:37.227770 30531 raft_consensus.cc:3037] T bdfe68a127f043f08f04749843a38375 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:37.230875 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdfe68a127f043f08f04749843a38375" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:12:37.231104 28262 raft_consensus.cc:2454] T bdfe68a127f043f08f04749843a38375 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:12:37.232323 30596 tablet_bootstrap.cc:492] T 978bf5783b0e47e3bfb30441adc600ae P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:37.232465 30596 ts_tablet_manager.cc:1367] T 978bf5783b0e47e3bfb30441adc600ae P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.003s
I20240503 08:12:37.233179 30596 raft_consensus.cc:348] T 978bf5783b0e47e3bfb30441adc600ae P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:37.233430 30596 raft_consensus.cc:374] T 978bf5783b0e47e3bfb30441adc600ae P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:37.233574 30596 raft_consensus.cc:729] T 978bf5783b0e47e3bfb30441adc600ae P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:37.233819 30596 consensus_queue.cc:260] T 978bf5783b0e47e3bfb30441adc600ae P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:37.234256 30596 ts_tablet_manager.cc:1398] T 978bf5783b0e47e3bfb30441adc600ae P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:37.230662 30531 raft_consensus.cc:504] T bdfe68a127f043f08f04749843a38375 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:37.235795 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdfe68a127f043f08f04749843a38375" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:12:37.235987 28262 raft_consensus.cc:3037] T bdfe68a127f043f08f04749843a38375 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:37.235951 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdfe68a127f043f08f04749843a38375" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:12:37.236126 28003 raft_consensus.cc:3037] T bdfe68a127f043f08f04749843a38375 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:37.238665 28262 raft_consensus.cc:2454] T bdfe68a127f043f08f04749843a38375 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:12:37.239068 28327 leader_election.cc:302] T bdfe68a127f043f08f04749843a38375 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:37.239382 30454 raft_consensus.cc:2781] T bdfe68a127f043f08f04749843a38375 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:37.239523 28003 raft_consensus.cc:2454] T bdfe68a127f043f08f04749843a38375 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:12:37.239584 30454 raft_consensus.cc:686] T bdfe68a127f043f08f04749843a38375 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Becoming Leader. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Running, Role: LEADER
I20240503 08:12:37.239848 30454 consensus_queue.cc:237] T bdfe68a127f043f08f04749843a38375 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:37.242039 30531 leader_election.cc:288] T bdfe68a127f043f08f04749843a38375 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:12:37.244181 27858 catalog_manager.cc:5484] T bdfe68a127f043f08f04749843a38375 P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68). New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:12:37.246207 30422 raft_consensus.cc:482] T 978bf5783b0e47e3bfb30441adc600ae P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:37.246371 30422 raft_consensus.cc:504] T 978bf5783b0e47e3bfb30441adc600ae P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:37.247047 30422 leader_election.cc:288] T 978bf5783b0e47e3bfb30441adc600ae P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:37.247500 28392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "978bf5783b0e47e3bfb30441adc600ae" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:37.247653 28392 raft_consensus.cc:2454] T 978bf5783b0e47e3bfb30441adc600ae P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:12:37.247869 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "978bf5783b0e47e3bfb30441adc600ae" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:12:37.248076 28132 raft_consensus.cc:2454] T 978bf5783b0e47e3bfb30441adc600ae P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:12:37.248462 28199 leader_election.cc:302] T 978bf5783b0e47e3bfb30441adc600ae P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:37.248713 30422 raft_consensus.cc:2781] T 978bf5783b0e47e3bfb30441adc600ae P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:37.248818 30422 raft_consensus.cc:482] T 978bf5783b0e47e3bfb30441adc600ae P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:37.248896 30422 raft_consensus.cc:3037] T 978bf5783b0e47e3bfb30441adc600ae P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:37.250826 30422 raft_consensus.cc:504] T 978bf5783b0e47e3bfb30441adc600ae P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:37.251302 30422 leader_election.cc:288] T 978bf5783b0e47e3bfb30441adc600ae P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Requested vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:37.252589 28392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "978bf5783b0e47e3bfb30441adc600ae" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:12:37.252751 28392 raft_consensus.cc:3037] T 978bf5783b0e47e3bfb30441adc600ae P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:37.253204 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "978bf5783b0e47e3bfb30441adc600ae" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:12:37.253371 28132 raft_consensus.cc:3037] T 978bf5783b0e47e3bfb30441adc600ae P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:37.254618 28392 raft_consensus.cc:2454] T 978bf5783b0e47e3bfb30441adc600ae P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:37.254971 28199 leader_election.cc:302] T 978bf5783b0e47e3bfb30441adc600ae P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:37.255195 30422 raft_consensus.cc:2781] T 978bf5783b0e47e3bfb30441adc600ae P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:37.255324 30422 raft_consensus.cc:686] T 978bf5783b0e47e3bfb30441adc600ae P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Becoming Leader. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Running, Role: LEADER
I20240503 08:12:37.255589 28132 raft_consensus.cc:2454] T 978bf5783b0e47e3bfb30441adc600ae P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:37.255578 30422 consensus_queue.cc:237] T 978bf5783b0e47e3bfb30441adc600ae P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:37.258944 27858 catalog_manager.cc:5484] T 978bf5783b0e47e3bfb30441adc600ae P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: term changed from 0 to 1, leader changed from <none> to aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67). New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:12:37.328305 30422 raft_consensus.cc:482] T 08fb64977309479b831635344969c662 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:37.328495 30422 raft_consensus.cc:504] T 08fb64977309479b831635344969c662 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:37.329087 30422 leader_election.cc:288] T 08fb64977309479b831635344969c662 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:37.329495 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08fb64977309479b831635344969c662" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:12:37.329524 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08fb64977309479b831635344969c662" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:12:37.329744 28132 raft_consensus.cc:2454] T 08fb64977309479b831635344969c662 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:12:37.329743 28003 raft_consensus.cc:2454] T 08fb64977309479b831635344969c662 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:12:37.330137 28198 leader_election.cc:302] T 08fb64977309479b831635344969c662 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:37.330420 30422 raft_consensus.cc:2781] T 08fb64977309479b831635344969c662 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:37.330500 30422 raft_consensus.cc:482] T 08fb64977309479b831635344969c662 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:37.330570 30422 raft_consensus.cc:3037] T 08fb64977309479b831635344969c662 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:37.332407 30422 raft_consensus.cc:504] T 08fb64977309479b831635344969c662 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:37.332962 30422 leader_election.cc:288] T 08fb64977309479b831635344969c662 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Requested vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:37.333216 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08fb64977309479b831635344969c662" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:12:37.333329 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08fb64977309479b831635344969c662" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:12:37.333415 28003 raft_consensus.cc:3037] T 08fb64977309479b831635344969c662 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:37.333527 28132 raft_consensus.cc:3037] T 08fb64977309479b831635344969c662 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:37.335309 28003 raft_consensus.cc:2454] T 08fb64977309479b831635344969c662 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:37.335474 28132 raft_consensus.cc:2454] T 08fb64977309479b831635344969c662 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:37.335656 28199 leader_election.cc:302] T 08fb64977309479b831635344969c662 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:37.335922 30422 raft_consensus.cc:2781] T 08fb64977309479b831635344969c662 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:37.336061 30422 raft_consensus.cc:686] T 08fb64977309479b831635344969c662 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Becoming Leader. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Running, Role: LEADER
I20240503 08:12:37.336282 30422 consensus_queue.cc:237] T 08fb64977309479b831635344969c662 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:37.339394 27858 catalog_manager.cc:5484] T 08fb64977309479b831635344969c662 P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: term changed from 0 to 1, leader changed from <none> to aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67). New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:12:37.648758 30422 consensus_queue.cc:1035] T 978bf5783b0e47e3bfb30441adc600ae P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:37.663390 30493 consensus_queue.cc:1035] T 978bf5783b0e47e3bfb30441adc600ae P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:37.718015 30612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:37.718348 30612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:37.726691 30612 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_19'
I20240503 08:12:37.799113 30493 consensus_queue.cc:1035] T 08fb64977309479b831635344969c662 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:37.804175 30493 consensus_queue.cc:1035] T 08fb64977309479b831635344969c662 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:37.813462 30454 consensus_queue.cc:1035] T bdfe68a127f043f08f04749843a38375 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:37.819444 30531 consensus_queue.cc:1035] T bdfe68a127f043f08f04749843a38375 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 239.472 ms
time per row: 0.119736 ms
I20240503 08:12:38.127478 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.69:34725
--local_ip_for_outbound_sockets=127.20.230.69
--tserver_master_addrs=127.20.230.126:39173
--webserver_port=40539
--webserver_interface=127.20.230.69
--builtin_ntp_servers=127.20.230.84:44873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 08:12:38.470784 30677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:38.471107 30677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:38.471238 30677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:38.484156 30677 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:12:38.484299 30677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:38.484478 30677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.69
I20240503 08:12:38.498728 30677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:44873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.69:34725
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.230.69
--webserver_port=40539
--tserver_master_addrs=127.20.230.126:39173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.69
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:38.500348 30677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:38.502388 30677 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:38.524991 30683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:38.525355 30682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:38.525903 30677 server_base.cc:1034] running on GCE node
W20240503 08:12:38.526185 30687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:38.526844 30677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:38.527710 30677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:38.528908 30677 hybrid_clock.cc:648] HybridClock initialized: now 1714723958528859 us; error 44 us; skew 500 ppm
I20240503 08:12:38.529203 30677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:38.532064 30677 webserver.cc:460] Webserver started at http://127.20.230.69:40539/ using document root <none> and password file <none>
I20240503 08:12:38.532862 30677 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:38.532982 30677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:38.538192 30677 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20240503 08:12:38.540967 30693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:38.542224 30677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:12:38.542364 30677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "6d1d7de19459435fbde1999e3dfd226b"
format_stamp: "Formatted at 2024-05-03 08:12:02 on dist-test-slave-6vp8"
I20240503 08:12:38.543115 30677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:38.572154 30677 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:38.572906 30677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:38.573179 30677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:38.574067 30677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:38.575500 30677 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:38.575577 30677 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:38.575688 30677 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:38.575740 30677 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:38.610355 30677 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.69:34725
I20240503 08:12:38.610431 30807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.69:34725 every 8 connection(s)
I20240503 08:12:38.612136 30677 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:12:38.617308 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 30677
I20240503 08:12:38.618079 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.70:33747
--local_ip_for_outbound_sockets=127.20.230.70
--tserver_master_addrs=127.20.230.126:39173
--webserver_port=41939
--webserver_interface=127.20.230.70
--builtin_ntp_servers=127.20.230.84:44873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:12:38.627785 30808 heartbeater.cc:347] Connected to a master server at 127.20.230.126:39173
I20240503 08:12:38.628075 30808 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:38.628837 30808 heartbeater.cc:511] Master 127.20.230.126:39173 requested a full tablet report, sending...
I20240503 08:12:38.630301 27862 ts_manager.cc:194] Re-registered known tserver with Master: 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:12:38.631073 27862 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.69:39203
W20240503 08:12:38.949601 30812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:38.949910 30812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:38.950038 30812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:38.957702 30812 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:12:38.957856 30812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:38.958014 30812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.70
I20240503 08:12:38.966990 30812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:44873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.70:33747
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.20.230.70
--webserver_port=41939
--tserver_master_addrs=127.20.230.126:39173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.70
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:38.968133 30812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:38.969789 30812 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:38.981702 30817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:38.982086 30818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:38.982442 30820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:38.982982 30812 server_base.cc:1034] running on GCE node
I20240503 08:12:38.983491 30812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:38.984467 30812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:38.985685 30812 hybrid_clock.cc:648] HybridClock initialized: now 1714723958985621 us; error 54 us; skew 500 ppm
I20240503 08:12:38.985991 30812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:38.988291 30812 webserver.cc:460] Webserver started at http://127.20.230.70:41939/ using document root <none> and password file <none>
I20240503 08:12:38.989166 30812 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:38.989284 30812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:38.994508 30812 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20240503 08:12:38.997025 30826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:38.998185 30812 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 08:12:38.998309 30812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "ce1496a47d084365ad786cd09e472a2c"
format_stamp: "Formatted at 2024-05-03 08:12:03 on dist-test-slave-6vp8"
I20240503 08:12:38.999011 30812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:39.018111 30812 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:39.018776 30812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:39.019050 30812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:39.019889 30812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:39.021229 30812 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:39.021297 30812 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:39.021384 30812 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:39.021425 30812 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:39.068142 30812 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.70:33747
I20240503 08:12:39.068172 30939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.70:33747 every 8 connection(s)
I20240503 08:12:39.069718 30812 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 08:12:39.070946 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 30812
I20240503 08:12:39.071651 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.71:41915
--local_ip_for_outbound_sockets=127.20.230.71
--tserver_master_addrs=127.20.230.126:39173
--webserver_port=35683
--webserver_interface=127.20.230.71
--builtin_ntp_servers=127.20.230.84:44873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:12:39.084692 30940 heartbeater.cc:347] Connected to a master server at 127.20.230.126:39173
I20240503 08:12:39.085064 30940 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:39.085856 30940 heartbeater.cc:511] Master 127.20.230.126:39173 requested a full tablet report, sending...
I20240503 08:12:39.087182 27862 ts_manager.cc:194] Re-registered known tserver with Master: ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:12:39.087976 27862 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.70:52597
W20240503 08:12:39.403003 30943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:39.403301 30943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:39.403432 30943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:39.411147 30943 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:12:39.411267 30943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:39.411417 30943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.71
I20240503 08:12:39.420266 30943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:44873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.71:41915
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.20.230.71
--webserver_port=35683
--tserver_master_addrs=127.20.230.126:39173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.71
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:39.421589 30943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:39.423415 30943 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:39.435231 30952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:39.435626 30949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:39.436014 30950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:39.436446 30943 server_base.cc:1034] running on GCE node
I20240503 08:12:39.436928 30943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:39.437719 30943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:39.438925 30943 hybrid_clock.cc:648] HybridClock initialized: now 1714723959438884 us; error 35 us; skew 500 ppm
I20240503 08:12:39.439211 30943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:39.441239 30943 webserver.cc:460] Webserver started at http://127.20.230.71:35683/ using document root <none> and password file <none>
I20240503 08:12:39.441924 30943 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:39.442046 30943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:39.447170 30943 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20240503 08:12:39.449703 30958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:39.450978 30943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240503 08:12:39.451121 30943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
format_stamp: "Formatted at 2024-05-03 08:12:03 on dist-test-slave-6vp8"
I20240503 08:12:39.451850 30943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:39.474452 30943 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:39.475131 30943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:39.475386 30943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:39.476182 30943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:39.477506 30943 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:39.477583 30943 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:39.477701 30943 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:39.477818 30943 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20240503 08:12:39.514514 30943 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.71:41915
I20240503 08:12:39.514590 31072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.71:41915 every 8 connection(s)
I20240503 08:12:39.515964 30943 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 08:12:39.522655 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 30943
I20240503 08:12:39.523190 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.72:40893
--local_ip_for_outbound_sockets=127.20.230.72
--tserver_master_addrs=127.20.230.126:39173
--webserver_port=44491
--webserver_interface=127.20.230.72
--builtin_ntp_servers=127.20.230.84:44873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:12:39.530272 31073 heartbeater.cc:347] Connected to a master server at 127.20.230.126:39173
I20240503 08:12:39.530583 31073 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:39.531231 31073 heartbeater.cc:511] Master 127.20.230.126:39173 requested a full tablet report, sending...
I20240503 08:12:39.532527 27862 ts_manager.cc:194] Re-registered known tserver with Master: ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:39.533406 27862 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.71:50969
I20240503 08:12:39.633965 30808 heartbeater.cc:503] Master 127.20.230.126:39173 was elected leader, sending a full tablet report...
W20240503 08:12:39.858814 31076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:39.859050 31076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:39.859165 31076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:39.866353 31076 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:12:39.866480 31076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:39.866633 31076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.72
I20240503 08:12:39.875588 31076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:44873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.72:40893
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.20.230.72
--webserver_port=44491
--tserver_master_addrs=127.20.230.126:39173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.72
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:39.876832 31076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:39.878855 31076 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:39.890887 31083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:39.891404 31084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:39.891701 31087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:39.892280 31076 server_base.cc:1034] running on GCE node
I20240503 08:12:39.892809 31076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:39.893644 31076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:39.894832 31076 hybrid_clock.cc:648] HybridClock initialized: now 1714723959894799 us; error 45 us; skew 500 ppm
I20240503 08:12:39.895114 31076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:39.897334 31076 webserver.cc:460] Webserver started at http://127.20.230.72:44491/ using document root <none> and password file <none>
I20240503 08:12:39.898054 31076 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:39.898173 31076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:39.903017 31076 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 08:12:39.905666 31093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:39.906939 31076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240503 08:12:39.907066 31076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "b00f244c69df401997ea99d492218a98"
format_stamp: "Formatted at 2024-05-03 08:12:04 on dist-test-slave-6vp8"
I20240503 08:12:39.907860 31076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:39.928607 31076 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:39.929283 31076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:39.929525 31076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:39.930392 31076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:39.931757 31076 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:39.931825 31076 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:39.931949 31076 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:39.931995 31076 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:39.969938 31076 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.72:40893
I20240503 08:12:39.970013 31205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.72:40893 every 8 connection(s)
I20240503 08:12:39.971321 31076 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 08:12:39.973691 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 31076
I20240503 08:12:39.989984 31206 heartbeater.cc:347] Connected to a master server at 127.20.230.126:39173
I20240503 08:12:39.990278 31206 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:39.990758 27862 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:39.990952 31206 heartbeater.cc:511] Master 127.20.230.126:39173 requested a full tablet report, sending...
I20240503 08:12:39.992162 27858 ts_manager.cc:194] Re-registered known tserver with Master: b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:39.993191 27858 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.72:36443
I20240503 08:12:40.000890 27982 tablet_service.cc:1467] Processing CreateTablet for tablet f2fafbbe2b7c464aa1fb76c25a4427d6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6e65df8c3c12429595da5f097a8de5f3]), partition=
I20240503 08:12:40.001196 28112 tablet_service.cc:1467] Processing CreateTablet for tablet f2fafbbe2b7c464aa1fb76c25a4427d6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6e65df8c3c12429595da5f097a8de5f3]), partition=
I20240503 08:12:40.001281 27982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2fafbbe2b7c464aa1fb76c25a4427d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:40.001567 28112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2fafbbe2b7c464aa1fb76c25a4427d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:40.018265 31221 tablet_bootstrap.cc:492] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:40.019593 31221 tablet_bootstrap.cc:654] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:40.025051 31222 tablet_bootstrap.cc:492] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:40.026770 31222 tablet_bootstrap.cc:654] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:40.028584 31222 tablet_bootstrap.cc:492] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:40.030210 31222 ts_tablet_manager.cc:1367] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 08:12:40.031004 31222 raft_consensus.cc:348] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:40.031349 31222 raft_consensus.cc:374] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:40.031507 31222 raft_consensus.cc:729] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:40.031821 31222 consensus_queue.cc:260] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:40.032598 31222 ts_tablet_manager.cc:1398] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:40.033243 30874 tablet_service.cc:1467] Processing CreateTablet for tablet f2fafbbe2b7c464aa1fb76c25a4427d6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6e65df8c3c12429595da5f097a8de5f3]), partition=
I20240503 08:12:40.035128 31221 tablet_bootstrap.cc:492] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:40.035267 31221 ts_tablet_manager.cc:1367] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.003s
I20240503 08:12:40.037456 31221 raft_consensus.cc:348] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:40.037662 31221 raft_consensus.cc:374] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:40.037739 31221 raft_consensus.cc:729] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:40.038451 31221 consensus_queue.cc:260] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:40.039011 31221 ts_tablet_manager.cc:1398] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:40.039583 30874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2fafbbe2b7c464aa1fb76c25a4427d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:40.055980 31225 tablet_bootstrap.cc:492] T f2fafbbe2b7c464aa1fb76c25a4427d6 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:40.060482 31225 tablet_bootstrap.cc:654] T f2fafbbe2b7c464aa1fb76c25a4427d6 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:40.061790 31225 log.cc:824] T f2fafbbe2b7c464aa1fb76c25a4427d6 P ce1496a47d084365ad786cd09e472a2c: Log is configured to *not* fsync() on all Append() calls
I20240503 08:12:40.076647 31225 tablet_bootstrap.cc:492] T f2fafbbe2b7c464aa1fb76c25a4427d6 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:40.076918 31225 ts_tablet_manager.cc:1367] T f2fafbbe2b7c464aa1fb76c25a4427d6 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.008s
I20240503 08:12:40.083619 31225 raft_consensus.cc:348] T f2fafbbe2b7c464aa1fb76c25a4427d6 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:40.083907 31225 raft_consensus.cc:374] T f2fafbbe2b7c464aa1fb76c25a4427d6 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:40.083999 31225 raft_consensus.cc:729] T f2fafbbe2b7c464aa1fb76c25a4427d6 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:40.084965 31225 consensus_queue.cc:260] T f2fafbbe2b7c464aa1fb76c25a4427d6 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:40.086062 30940 heartbeater.cc:503] Master 127.20.230.126:39173 was elected leader, sending a full tablet report...
I20240503 08:12:40.087220 31225 ts_tablet_manager.cc:1398] T f2fafbbe2b7c464aa1fb76c25a4427d6 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.010s user 0.010s sys 0.001s
W20240503 08:12:40.129726 28049 tablet.cc:2369] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 833582ff68c94269be364ebd187a75db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:12:40.210028 30944 raft_consensus.cc:482] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:40.210211 30944 raft_consensus.cc:504] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:40.211068 30944 leader_election.cc:288] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:40.212481 28131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fafbbe2b7c464aa1fb76c25a4427d6" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:12:40.212729 28131 raft_consensus.cc:2454] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:40.214890 27938 leader_election.cc:302] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 833582ff68c94269be364ebd187a75db; no voters:
I20240503 08:12:40.215248 30686 raft_consensus.cc:2781] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:40.215353 30686 raft_consensus.cc:482] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:40.215451 30686 raft_consensus.cc:3037] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:40.218725 30686 raft_consensus.cc:504] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:40.220527 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fafbbe2b7c464aa1fb76c25a4427d6" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:12:40.220937 30686 leader_election.cc:288] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Requested vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:40.221083 30895 raft_consensus.cc:2454] T f2fafbbe2b7c464aa1fb76c25a4427d6 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:40.221577 28131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fafbbe2b7c464aa1fb76c25a4427d6" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:12:40.221576 30894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fafbbe2b7c464aa1fb76c25a4427d6" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:12:40.221800 28131 raft_consensus.cc:3037] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:40.221846 30894 raft_consensus.cc:3037] T f2fafbbe2b7c464aa1fb76c25a4427d6 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:40.224730 30894 raft_consensus.cc:2454] T f2fafbbe2b7c464aa1fb76c25a4427d6 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:40.224758 28131 raft_consensus.cc:2454] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:40.225142 27936 leader_election.cc:302] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:40.225556 30686 raft_consensus.cc:2781] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:40.225711 30686 raft_consensus.cc:686] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Becoming Leader. State: Replica: 833582ff68c94269be364ebd187a75db, State: Running, Role: LEADER
I20240503 08:12:40.225991 30686 consensus_queue.cc:237] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:40.231580 27862 catalog_manager.cc:5484] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 833582ff68c94269be364ebd187a75db reported cstate change: term changed from 0 to 1, leader changed from <none> to 833582ff68c94269be364ebd187a75db (127.20.230.65). New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
W20240503 08:12:40.275336 28179 tablet.cc:2369] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 1b411c6bb25549368041773d7b710d48: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:12:40.329048 30941 tablet.cc:2369] T f2fafbbe2b7c464aa1fb76c25a4427d6 P ce1496a47d084365ad786cd09e472a2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:12:40.344563 27862 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:40.351625 28131 raft_consensus.cc:1261] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:40.352123 30944 consensus_queue.cc:1035] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:40.352442 30894 raft_consensus.cc:1261] T f2fafbbe2b7c464aa1fb76c25a4427d6 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:40.353551 30944 consensus_queue.cc:1035] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:40.357991 27862 catalog_manager.cc:5576] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143) does not have the latest schema for tablet f2fafbbe2b7c464aa1fb76c25a4427d6 (table rebalancer_dynamic_table [id=6e65df8c3c12429595da5f097a8de5f3]). Expected version 1 got 0
I20240503 08:12:40.361711 31238 tablet.cc:1722] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 1b411c6bb25549368041773d7b710d48: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:40.362915 27862 catalog_manager.cc:5576] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143) does not have the latest schema for tablet f2fafbbe2b7c464aa1fb76c25a4427d6 (table rebalancer_dynamic_table [id=6e65df8c3c12429595da5f097a8de5f3]). Expected version 1 got 0
I20240503 08:12:40.363721 31237 tablet.cc:1722] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 833582ff68c94269be364ebd187a75db: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:40.364620 31238 tablet.cc:1620] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 1b411c6bb25549368041773d7b710d48: MemRowSet was empty: no flush needed.
I20240503 08:12:40.365523 31237 tablet.cc:1620] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 833582ff68c94269be364ebd187a75db: MemRowSet was empty: no flush needed.
I20240503 08:12:40.375494 27862 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=6e65df8c3c12429595da5f097a8de5f3] alter complete (version 1)
I20240503 08:12:40.381474 27858 catalog_manager.cc:5576] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747) does not have the latest schema for tablet f2fafbbe2b7c464aa1fb76c25a4427d6 (table rebalancer_dynamic_table [id=6e65df8c3c12429595da5f097a8de5f3]). Expected version 1 got 0
I20240503 08:12:40.382053 31243 tablet.cc:1722] T f2fafbbe2b7c464aa1fb76c25a4427d6 P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:40.382463 31236 mvcc.cc:205] Tried to move back new op lower bound from 7023509341594673152 to 7023509341087731712. Current Snapshot: MvccSnapshot[applied={T|T < 7023509341594673152}]
I20240503 08:12:40.383656 31243 tablet.cc:1620] T f2fafbbe2b7c464aa1fb76c25a4427d6 P ce1496a47d084365ad786cd09e472a2c: MemRowSet was empty: no flush needed.
I20240503 08:12:40.483980 27859 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240503 08:12:40.484212 27859 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240503 08:12:40.486588 27856 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:40.490605 27854 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_extra_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:40.492178 27859 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet d845da554a404c20be95a0bc11f49826
I20240503 08:12:40.492933 27859 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet bd1407f5301a41438e79d872961d74d6
I20240503 08:12:40.493690 28112 tablet_service.cc:1514] Processing DeleteTablet for tablet d845da554a404c20be95a0bc11f49826 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:40 UTC) from {username='slave'} at 127.0.0.1:57912
I20240503 08:12:40.493834 28373 tablet_service.cc:1514] Processing DeleteTablet for tablet d845da554a404c20be95a0bc11f49826 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:40 UTC) from {username='slave'} at 127.0.0.1:34786
I20240503 08:12:40.493822 27982 tablet_service.cc:1514] Processing DeleteTablet for tablet d845da554a404c20be95a0bc11f49826 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:40 UTC) from {username='slave'} at 127.0.0.1:55610
I20240503 08:12:40.498807 27859 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet 02da1912b357405fa7e48e8d6ae8329d
I20240503 08:12:40.499656 28111 tablet_service.cc:1514] Processing DeleteTablet for tablet bd1407f5301a41438e79d872961d74d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:40 UTC) from {username='slave'} at 127.0.0.1:57912
I20240503 08:12:40.500602 28372 tablet_service.cc:1514] Processing DeleteTablet for tablet 02da1912b357405fa7e48e8d6ae8329d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:40 UTC) from {username='slave'} at 127.0.0.1:34786
I20240503 08:12:40.501835 31267 tablet_replica.cc:330] T d845da554a404c20be95a0bc11f49826 P 1b411c6bb25549368041773d7b710d48: stopping tablet replica
I20240503 08:12:40.503250 27983 tablet_service.cc:1514] Processing DeleteTablet for tablet bd1407f5301a41438e79d872961d74d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:40 UTC) from {username='slave'} at 127.0.0.1:55610
I20240503 08:12:40.503367 27981 tablet_service.cc:1514] Processing DeleteTablet for tablet 02da1912b357405fa7e48e8d6ae8329d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:40 UTC) from {username='slave'} at 127.0.0.1:55610
I20240503 08:12:40.502295 31267 raft_consensus.cc:2229] T d845da554a404c20be95a0bc11f49826 P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:40.503957 31267 raft_consensus.cc:2258] T d845da554a404c20be95a0bc11f49826 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:40.504127 31237 tablet.cc:1722] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 833582ff68c94269be364ebd187a75db: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:40.504647 31237 tablet.cc:1620] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 833582ff68c94269be364ebd187a75db: MemRowSet was empty: no flush needed.
I20240503 08:12:40.505851 31238 tablet.cc:1722] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 1b411c6bb25549368041773d7b710d48: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:40.506312 31238 tablet.cc:1620] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 1b411c6bb25549368041773d7b710d48: MemRowSet was empty: no flush needed.
I20240503 08:12:40.506770 31243 tablet.cc:1722] T f2fafbbe2b7c464aa1fb76c25a4427d6 P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:40.506992 28242 tablet_service.cc:1514] Processing DeleteTablet for tablet 02da1912b357405fa7e48e8d6ae8329d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:40 UTC) from {username='slave'} at 127.0.0.1:41960
I20240503 08:12:40.507210 31243 tablet.cc:1620] T f2fafbbe2b7c464aa1fb76c25a4427d6 P ce1496a47d084365ad786cd09e472a2c: MemRowSet was empty: no flush needed.
I20240503 08:12:40.507620 28243 tablet_service.cc:1514] Processing DeleteTablet for tablet bd1407f5301a41438e79d872961d74d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:40 UTC) from {username='slave'} at 127.0.0.1:41960
I20240503 08:12:40.515128 27981 tablet_service.cc:1467] Processing CreateTablet for tablet 9fbe2eaba5344dfaa29e78a0d73f224a (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=10122b5266cf4446bb541a06acf3a16f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:40.515566 27981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fbe2eaba5344dfaa29e78a0d73f224a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:40.525413 30874 tablet_service.cc:1467] Processing CreateTablet for tablet 7815cb2db9514c02b0ee3e3cd12b2212 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=10122b5266cf4446bb541a06acf3a16f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:40.525571 28112 tablet_service.cc:1467] Processing CreateTablet for tablet 7815cb2db9514c02b0ee3e3cd12b2212 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=10122b5266cf4446bb541a06acf3a16f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:40.525933 30874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7815cb2db9514c02b0ee3e3cd12b2212. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:40.525961 28112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7815cb2db9514c02b0ee3e3cd12b2212. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:40.535435 31073 heartbeater.cc:503] Master 127.20.230.126:39173 was elected leader, sending a full tablet report...
I20240503 08:12:40.536309 31225 tablet_bootstrap.cc:492] T 7815cb2db9514c02b0ee3e3cd12b2212 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:40.537783 31225 tablet_bootstrap.cc:654] T 7815cb2db9514c02b0ee3e3cd12b2212 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:40.539376 27859 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=6e65df8c3c12429595da5f097a8de5f3] alter complete (version 2)
I20240503 08:12:40.548753 31270 tablet_replica.cc:330] T 02da1912b357405fa7e48e8d6ae8329d P aae860d7ae2b4002bf7794c278f9cc56: stopping tablet replica
I20240503 08:12:40.550297 31270 raft_consensus.cc:2229] T 02da1912b357405fa7e48e8d6ae8329d P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:40.551949 31270 raft_consensus.cc:2258] T 02da1912b357405fa7e48e8d6ae8329d P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:40.555812 31270 ts_tablet_manager.cc:1875] T 02da1912b357405fa7e48e8d6ae8329d P aae860d7ae2b4002bf7794c278f9cc56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:40.506003 31267 ts_tablet_manager.cc:1875] T d845da554a404c20be95a0bc11f49826 P 1b411c6bb25549368041773d7b710d48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:40.579092 31277 tablet_bootstrap.cc:492] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:40.581221 31270 ts_tablet_manager.cc:1888] T 02da1912b357405fa7e48e8d6ae8329d P aae860d7ae2b4002bf7794c278f9cc56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:40.581966 31270 log.cc:1196] T 02da1912b357405fa7e48e8d6ae8329d P aae860d7ae2b4002bf7794c278f9cc56: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal/wals/02da1912b357405fa7e48e8d6ae8329d
I20240503 08:12:40.583499 31270 ts_tablet_manager.cc:1909] T 02da1912b357405fa7e48e8d6ae8329d P aae860d7ae2b4002bf7794c278f9cc56: Deleting consensus metadata
I20240503 08:12:40.584203 31277 tablet_bootstrap.cc:654] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:40.584390 31278 tablet_bootstrap.cc:492] T 7815cb2db9514c02b0ee3e3cd12b2212 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:40.585997 31278 tablet_bootstrap.cc:654] T 7815cb2db9514c02b0ee3e3cd12b2212 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:40.588989 27847 catalog_manager.cc:4830] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): tablet 02da1912b357405fa7e48e8d6ae8329d (table rebalance_test_table_0 [id=06bbbe2220124afd9e58aad99f4c6e3d]) successfully deleted
I20240503 08:12:40.589381 31225 tablet_bootstrap.cc:492] T 7815cb2db9514c02b0ee3e3cd12b2212 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:40.590142 31270 tablet_replica.cc:330] T bd1407f5301a41438e79d872961d74d6 P aae860d7ae2b4002bf7794c278f9cc56: stopping tablet replica
I20240503 08:12:40.590512 31270 raft_consensus.cc:2229] T bd1407f5301a41438e79d872961d74d6 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:40.591217 31270 raft_consensus.cc:2258] T bd1407f5301a41438e79d872961d74d6 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:40.591548 31225 ts_tablet_manager.cc:1367] T 7815cb2db9514c02b0ee3e3cd12b2212 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.055s user 0.000s sys 0.011s
I20240503 08:12:40.590409 30742 tablet_service.cc:1467] Processing CreateTablet for tablet 9fbe2eaba5344dfaa29e78a0d73f224a (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=10122b5266cf4446bb541a06acf3a16f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:40.592489 31270 ts_tablet_manager.cc:1875] T bd1407f5301a41438e79d872961d74d6 P aae860d7ae2b4002bf7794c278f9cc56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:40.592483 31225 raft_consensus.cc:348] T 7815cb2db9514c02b0ee3e3cd12b2212 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:40.592702 31225 raft_consensus.cc:374] T 7815cb2db9514c02b0ee3e3cd12b2212 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:40.592788 31225 raft_consensus.cc:729] T 7815cb2db9514c02b0ee3e3cd12b2212 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:40.589639 31268 tablet_replica.cc:330] T d845da554a404c20be95a0bc11f49826 P d09a11a03989421b9f28019dc1c14cc2: stopping tablet replica
I20240503 08:12:40.593099 30742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fbe2eaba5344dfaa29e78a0d73f224a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:40.593446 31268 raft_consensus.cc:2229] T d845da554a404c20be95a0bc11f49826 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:40.593828 31268 raft_consensus.cc:2258] T d845da554a404c20be95a0bc11f49826 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:40.594307 31267 ts_tablet_manager.cc:1888] T d845da554a404c20be95a0bc11f49826 P 1b411c6bb25549368041773d7b710d48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:40.595134 31225 consensus_queue.cc:260] T 7815cb2db9514c02b0ee3e3cd12b2212 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:40.595925 30741 tablet_service.cc:1467] Processing CreateTablet for tablet 7815cb2db9514c02b0ee3e3cd12b2212 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=10122b5266cf4446bb541a06acf3a16f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:40.596037 30740 tablet_service.cc:1467] Processing CreateTablet for tablet 013e00c67a1c4ae9b3c1a050b966e9a5 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=10122b5266cf4446bb541a06acf3a16f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:40.596402 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7815cb2db9514c02b0ee3e3cd12b2212. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:40.596624 30740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 013e00c67a1c4ae9b3c1a050b966e9a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:40.599447 31267 log.cc:1196] T d845da554a404c20be95a0bc11f49826 P 1b411c6bb25549368041773d7b710d48: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/d845da554a404c20be95a0bc11f49826
I20240503 08:12:40.599684 31277 tablet_bootstrap.cc:492] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:40.600057 31267 ts_tablet_manager.cc:1909] T d845da554a404c20be95a0bc11f49826 P 1b411c6bb25549368041773d7b710d48: Deleting consensus metadata
I20240503 08:12:40.600049 31277 ts_tablet_manager.cc:1367] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20240503 08:12:40.600790 31277 raft_consensus.cc:348] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:40.601047 31277 raft_consensus.cc:374] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:40.601157 31277 raft_consensus.cc:729] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:40.601395 31277 consensus_queue.cc:260] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:40.601552 31267 tablet_replica.cc:330] T bd1407f5301a41438e79d872961d74d6 P 1b411c6bb25549368041773d7b710d48: stopping tablet replica
I20240503 08:12:40.601635 27848 catalog_manager.cc:4830] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): tablet d845da554a404c20be95a0bc11f49826 (table rebalance_test_table_0 [id=06bbbe2220124afd9e58aad99f4c6e3d]) successfully deleted
I20240503 08:12:40.601825 31267 raft_consensus.cc:2229] T bd1407f5301a41438e79d872961d74d6 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:40.602072 31267 raft_consensus.cc:2258] T bd1407f5301a41438e79d872961d74d6 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:40.602784 31267 ts_tablet_manager.cc:1875] T bd1407f5301a41438e79d872961d74d6 P 1b411c6bb25549368041773d7b710d48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:40.603147 31277 ts_tablet_manager.cc:1398] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:40.608688 31267 ts_tablet_manager.cc:1888] T bd1407f5301a41438e79d872961d74d6 P 1b411c6bb25549368041773d7b710d48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:40.608855 31267 log.cc:1196] T bd1407f5301a41438e79d872961d74d6 P 1b411c6bb25549368041773d7b710d48: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/bd1407f5301a41438e79d872961d74d6
I20240503 08:12:40.609251 31267 ts_tablet_manager.cc:1909] T bd1407f5301a41438e79d872961d74d6 P 1b411c6bb25549368041773d7b710d48: Deleting consensus metadata
I20240503 08:12:40.610381 27848 catalog_manager.cc:4830] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): tablet bd1407f5301a41438e79d872961d74d6 (table rebalance_test_table_0 [id=06bbbe2220124afd9e58aad99f4c6e3d]) successfully deleted
I20240503 08:12:40.611766 31270 ts_tablet_manager.cc:1888] T bd1407f5301a41438e79d872961d74d6 P aae860d7ae2b4002bf7794c278f9cc56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:40.612936 31270 log.cc:1196] T bd1407f5301a41438e79d872961d74d6 P aae860d7ae2b4002bf7794c278f9cc56: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal/wals/bd1407f5301a41438e79d872961d74d6
I20240503 08:12:40.614033 31270 ts_tablet_manager.cc:1909] T bd1407f5301a41438e79d872961d74d6 P aae860d7ae2b4002bf7794c278f9cc56: Deleting consensus metadata
I20240503 08:12:40.617327 27847 catalog_manager.cc:4830] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): tablet bd1407f5301a41438e79d872961d74d6 (table rebalance_test_table_0 [id=06bbbe2220124afd9e58aad99f4c6e3d]) successfully deleted
I20240503 08:12:40.617972 31225 ts_tablet_manager.cc:1398] T 7815cb2db9514c02b0ee3e3cd12b2212 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.026s user 0.000s sys 0.002s
I20240503 08:12:40.618592 31268 ts_tablet_manager.cc:1875] T d845da554a404c20be95a0bc11f49826 P d09a11a03989421b9f28019dc1c14cc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:40.620278 31278 tablet_bootstrap.cc:492] T 7815cb2db9514c02b0ee3e3cd12b2212 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:40.620400 31278 ts_tablet_manager.cc:1367] T 7815cb2db9514c02b0ee3e3cd12b2212 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.000s
I20240503 08:12:40.621115 31278 raft_consensus.cc:348] T 7815cb2db9514c02b0ee3e3cd12b2212 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:40.621280 31278 raft_consensus.cc:374] T 7815cb2db9514c02b0ee3e3cd12b2212 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:40.621347 31278 raft_consensus.cc:729] T 7815cb2db9514c02b0ee3e3cd12b2212 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:40.621577 31278 consensus_queue.cc:260] T 7815cb2db9514c02b0ee3e3cd12b2212 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:40.623842 31269 tablet_replica.cc:330] T d845da554a404c20be95a0bc11f49826 P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:12:40.624315 31269 raft_consensus.cc:2229] T d845da554a404c20be95a0bc11f49826 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:40.624351 31278 ts_tablet_manager.cc:1398] T 7815cb2db9514c02b0ee3e3cd12b2212 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:40.624794 31269 raft_consensus.cc:2258] T d845da554a404c20be95a0bc11f49826 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:40.629310 31285 tablet_bootstrap.cc:492] T 013e00c67a1c4ae9b3c1a050b966e9a5 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:40.630422 31269 ts_tablet_manager.cc:1875] T d845da554a404c20be95a0bc11f49826 P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:40.632349 31285 tablet_bootstrap.cc:654] T 013e00c67a1c4ae9b3c1a050b966e9a5 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:40.633298 31285 log.cc:824] T 013e00c67a1c4ae9b3c1a050b966e9a5 P 6d1d7de19459435fbde1999e3dfd226b: Log is configured to *not* fsync() on all Append() calls
I20240503 08:12:40.636363 31285 tablet_bootstrap.cc:492] T 013e00c67a1c4ae9b3c1a050b966e9a5 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:40.636788 31285 ts_tablet_manager.cc:1367] T 013e00c67a1c4ae9b3c1a050b966e9a5 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240503 08:12:40.642771 31268 ts_tablet_manager.cc:1888] T d845da554a404c20be95a0bc11f49826 P d09a11a03989421b9f28019dc1c14cc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:40.642918 31268 log.cc:1196] T d845da554a404c20be95a0bc11f49826 P d09a11a03989421b9f28019dc1c14cc2: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/wals/d845da554a404c20be95a0bc11f49826
I20240503 08:12:40.643421 31268 ts_tablet_manager.cc:1909] T d845da554a404c20be95a0bc11f49826 P d09a11a03989421b9f28019dc1c14cc2: Deleting consensus metadata
I20240503 08:12:40.645291 27854 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:40.645541 31141 tablet_service.cc:1467] Processing CreateTablet for tablet 013e00c67a1c4ae9b3c1a050b966e9a5 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=10122b5266cf4446bb541a06acf3a16f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:40.647246 31141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 013e00c67a1c4ae9b3c1a050b966e9a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:40.647609 31285 raft_consensus.cc:348] T 013e00c67a1c4ae9b3c1a050b966e9a5 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:40.648058 31285 raft_consensus.cc:374] T 013e00c67a1c4ae9b3c1a050b966e9a5 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:40.648232 31285 raft_consensus.cc:729] T 013e00c67a1c4ae9b3c1a050b966e9a5 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:40.648255 31008 tablet_service.cc:1467] Processing CreateTablet for tablet 9fbe2eaba5344dfaa29e78a0d73f224a (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=10122b5266cf4446bb541a06acf3a16f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:40.648898 31285 consensus_queue.cc:260] T 013e00c67a1c4ae9b3c1a050b966e9a5 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:40.649783 31008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fbe2eaba5344dfaa29e78a0d73f224a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:40.654515 31007 tablet_service.cc:1467] Processing CreateTablet for tablet 013e00c67a1c4ae9b3c1a050b966e9a5 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=10122b5266cf4446bb541a06acf3a16f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:40.655074 31007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 013e00c67a1c4ae9b3c1a050b966e9a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:40.655855 27849 catalog_manager.cc:4830] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): tablet d845da554a404c20be95a0bc11f49826 (table rebalance_test_table_0 [id=06bbbe2220124afd9e58aad99f4c6e3d]) successfully deleted
I20240503 08:12:40.656654 31268 tablet_replica.cc:330] T 02da1912b357405fa7e48e8d6ae8329d P d09a11a03989421b9f28019dc1c14cc2: stopping tablet replica
I20240503 08:12:40.656878 31268 raft_consensus.cc:2229] T 02da1912b357405fa7e48e8d6ae8329d P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:40.657104 31268 raft_consensus.cc:2258] T 02da1912b357405fa7e48e8d6ae8329d P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:40.660881 31268 ts_tablet_manager.cc:1875] T 02da1912b357405fa7e48e8d6ae8329d P d09a11a03989421b9f28019dc1c14cc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:40.660965 31237 tablet.cc:1722] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 833582ff68c94269be364ebd187a75db: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:40.663544 31269 ts_tablet_manager.cc:1888] T d845da554a404c20be95a0bc11f49826 P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:40.663880 31269 log.cc:1196] T d845da554a404c20be95a0bc11f49826 P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/d845da554a404c20be95a0bc11f49826
I20240503 08:12:40.664932 31269 ts_tablet_manager.cc:1909] T d845da554a404c20be95a0bc11f49826 P 833582ff68c94269be364ebd187a75db: Deleting consensus metadata
I20240503 08:12:40.667380 31289 tablet_bootstrap.cc:492] T 013e00c67a1c4ae9b3c1a050b966e9a5 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:40.668401 31243 tablet.cc:1722] T f2fafbbe2b7c464aa1fb76c25a4427d6 P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:40.673341 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet d845da554a404c20be95a0bc11f49826 (table rebalance_test_table_0 [id=06bbbe2220124afd9e58aad99f4c6e3d]) successfully deleted
I20240503 08:12:40.673833 27854 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table_new_name [id=6e65df8c3c12429595da5f097a8de5f3] alter complete (version 3)
I20240503 08:12:40.675740 31238 tablet.cc:1722] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 1b411c6bb25549368041773d7b710d48: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:40.676211 31269 tablet_replica.cc:330] T bd1407f5301a41438e79d872961d74d6 P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:12:40.676412 31269 raft_consensus.cc:2229] T bd1407f5301a41438e79d872961d74d6 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:40.679771 31269 raft_consensus.cc:2258] T bd1407f5301a41438e79d872961d74d6 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:40.682749 31268 ts_tablet_manager.cc:1888] T 02da1912b357405fa7e48e8d6ae8329d P d09a11a03989421b9f28019dc1c14cc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:40.683651 31269 ts_tablet_manager.cc:1875] T bd1407f5301a41438e79d872961d74d6 P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:40.685360 31289 tablet_bootstrap.cc:654] T 013e00c67a1c4ae9b3c1a050b966e9a5 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:40.704447 31289 log.cc:824] T 013e00c67a1c4ae9b3c1a050b966e9a5 P ced21ca983b749d0b4d4f6a0e2426bf8: Log is configured to *not* fsync() on all Append() calls
I20240503 08:12:40.711978 31269 ts_tablet_manager.cc:1888] T bd1407f5301a41438e79d872961d74d6 P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:40.712105 31269 log.cc:1196] T bd1407f5301a41438e79d872961d74d6 P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/bd1407f5301a41438e79d872961d74d6
I20240503 08:12:40.712436 31269 ts_tablet_manager.cc:1909] T bd1407f5301a41438e79d872961d74d6 P 833582ff68c94269be364ebd187a75db: Deleting consensus metadata
I20240503 08:12:40.713285 31269 tablet_replica.cc:330] T 02da1912b357405fa7e48e8d6ae8329d P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:12:40.713438 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet bd1407f5301a41438e79d872961d74d6 (table rebalance_test_table_0 [id=06bbbe2220124afd9e58aad99f4c6e3d]) successfully deleted
I20240503 08:12:40.713480 31268 log.cc:1196] T 02da1912b357405fa7e48e8d6ae8329d P d09a11a03989421b9f28019dc1c14cc2: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/wals/02da1912b357405fa7e48e8d6ae8329d
I20240503 08:12:40.714016 31268 ts_tablet_manager.cc:1909] T 02da1912b357405fa7e48e8d6ae8329d P d09a11a03989421b9f28019dc1c14cc2: Deleting consensus metadata
I20240503 08:12:40.715504 27849 catalog_manager.cc:4830] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): tablet 02da1912b357405fa7e48e8d6ae8329d (table rebalance_test_table_0 [id=06bbbe2220124afd9e58aad99f4c6e3d]) successfully deleted
I20240503 08:12:40.722173 31289 tablet_bootstrap.cc:492] T 013e00c67a1c4ae9b3c1a050b966e9a5 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:40.722582 31289 ts_tablet_manager.cc:1367] T 013e00c67a1c4ae9b3c1a050b966e9a5 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.055s user 0.008s sys 0.000s
I20240503 08:12:40.725013 31285 ts_tablet_manager.cc:1398] T 013e00c67a1c4ae9b3c1a050b966e9a5 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.088s user 0.015s sys 0.001s
I20240503 08:12:40.716120 31269 raft_consensus.cc:2229] T 02da1912b357405fa7e48e8d6ae8329d P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:40.726848 31227 raft_consensus.cc:482] T 7815cb2db9514c02b0ee3e3cd12b2212 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:40.727010 31227 raft_consensus.cc:504] T 7815cb2db9514c02b0ee3e3cd12b2212 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:40.728420 31285 tablet_bootstrap.cc:492] T 9fbe2eaba5344dfaa29e78a0d73f224a P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:40.728463 31227 leader_election.cc:288] T 7815cb2db9514c02b0ee3e3cd12b2212 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:40.730671 31285 tablet_bootstrap.cc:654] T 9fbe2eaba5344dfaa29e78a0d73f224a P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:40.731861 31288 tablet_bootstrap.cc:492] T 013e00c67a1c4ae9b3c1a050b966e9a5 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:40.737308 31288 tablet_bootstrap.cc:654] T 013e00c67a1c4ae9b3c1a050b966e9a5 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:40.742550 31285 tablet_bootstrap.cc:492] T 9fbe2eaba5344dfaa29e78a0d73f224a P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:40.742743 31285 ts_tablet_manager.cc:1367] T 9fbe2eaba5344dfaa29e78a0d73f224a P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.003s
I20240503 08:12:40.743762 31285 raft_consensus.cc:348] T 9fbe2eaba5344dfaa29e78a0d73f224a P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:40.744153 31285 raft_consensus.cc:374] T 9fbe2eaba5344dfaa29e78a0d73f224a P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:40.744326 31285 raft_consensus.cc:729] T 9fbe2eaba5344dfaa29e78a0d73f224a P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:40.744655 31285 consensus_queue.cc:260] T 9fbe2eaba5344dfaa29e78a0d73f224a P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:40.744869 31288 log.cc:824] T 013e00c67a1c4ae9b3c1a050b966e9a5 P b00f244c69df401997ea99d492218a98: Log is configured to *not* fsync() on all Append() calls
I20240503 08:12:40.744904 31269 raft_consensus.cc:2258] T 02da1912b357405fa7e48e8d6ae8329d P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:40.745710 31269 ts_tablet_manager.cc:1875] T 02da1912b357405fa7e48e8d6ae8329d P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:40.749059 31285 ts_tablet_manager.cc:1398] T 9fbe2eaba5344dfaa29e78a0d73f224a P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 08:12:40.752513 31285 tablet_bootstrap.cc:492] T 7815cb2db9514c02b0ee3e3cd12b2212 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:40.754266 31285 tablet_bootstrap.cc:654] T 7815cb2db9514c02b0ee3e3cd12b2212 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:40.755887 31289 raft_consensus.cc:348] T 013e00c67a1c4ae9b3c1a050b966e9a5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:40.757026 31289 raft_consensus.cc:374] T 013e00c67a1c4ae9b3c1a050b966e9a5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:40.757334 31289 raft_consensus.cc:729] T 013e00c67a1c4ae9b3c1a050b966e9a5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:40.758659 31289 consensus_queue.cc:260] T 013e00c67a1c4ae9b3c1a050b966e9a5 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:40.760932 31269 ts_tablet_manager.cc:1888] T 02da1912b357405fa7e48e8d6ae8329d P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:40.761907 31269 log.cc:1196] T 02da1912b357405fa7e48e8d6ae8329d P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/02da1912b357405fa7e48e8d6ae8329d
I20240503 08:12:40.762969 31269 ts_tablet_manager.cc:1909] T 02da1912b357405fa7e48e8d6ae8329d P 833582ff68c94269be364ebd187a75db: Deleting consensus metadata
I20240503 08:12:40.764748 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet 02da1912b357405fa7e48e8d6ae8329d (table rebalance_test_table_0 [id=06bbbe2220124afd9e58aad99f4c6e3d]) successfully deleted
I20240503 08:12:40.766542 31289 ts_tablet_manager.cc:1398] T 013e00c67a1c4ae9b3c1a050b966e9a5 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.043s user 0.008s sys 0.004s
W20240503 08:12:40.768383 31074 tablet.cc:2369] T 013e00c67a1c4ae9b3c1a050b966e9a5 P ced21ca983b749d0b4d4f6a0e2426bf8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:12:40.774194 31289 tablet_bootstrap.cc:492] T 9fbe2eaba5344dfaa29e78a0d73f224a P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:40.776012 31289 tablet_bootstrap.cc:654] T 9fbe2eaba5344dfaa29e78a0d73f224a P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:40.779711 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815cb2db9514c02b0ee3e3cd12b2212" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:12:40.782490 31289 tablet_bootstrap.cc:492] T 9fbe2eaba5344dfaa29e78a0d73f224a P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
W20240503 08:12:40.782444 30829 leader_election.cc:341] T 7815cb2db9514c02b0ee3e3cd12b2212 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:40.782858 31289 ts_tablet_manager.cc:1367] T 9fbe2eaba5344dfaa29e78a0d73f224a P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240503 08:12:40.784008 31289 raft_consensus.cc:348] T 9fbe2eaba5344dfaa29e78a0d73f224a P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:40.784502 31288 tablet_bootstrap.cc:492] T 013e00c67a1c4ae9b3c1a050b966e9a5 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:40.784788 31288 ts_tablet_manager.cc:1367] T 013e00c67a1c4ae9b3c1a050b966e9a5 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.053s user 0.005s sys 0.014s
I20240503 08:12:40.785104 27854 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:40.785295 27854 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:40.785414 31289 raft_consensus.cc:374] T 9fbe2eaba5344dfaa29e78a0d73f224a P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:40.785511 31289 raft_consensus.cc:729] T 9fbe2eaba5344dfaa29e78a0d73f224a P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:40.786087 31289 consensus_queue.cc:260] T 9fbe2eaba5344dfaa29e78a0d73f224a P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:40.786589 31289 ts_tablet_manager.cc:1398] T 9fbe2eaba5344dfaa29e78a0d73f224a P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:12:40.790161 27854 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet f2fafbbe2b7c464aa1fb76c25a4427d6
I20240503 08:12:40.791003 30874 tablet_service.cc:1514] Processing DeleteTablet for tablet f2fafbbe2b7c464aa1fb76c25a4427d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:40 UTC) from {username='slave'} at 127.0.0.1:34426
I20240503 08:12:40.791703 27982 tablet_service.cc:1514] Processing DeleteTablet for tablet f2fafbbe2b7c464aa1fb76c25a4427d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:40 UTC) from {username='slave'} at 127.0.0.1:55610
I20240503 08:12:40.793752 31288 raft_consensus.cc:348] T 013e00c67a1c4ae9b3c1a050b966e9a5 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:40.794229 31288 raft_consensus.cc:374] T 013e00c67a1c4ae9b3c1a050b966e9a5 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:40.794333 31288 raft_consensus.cc:729] T 013e00c67a1c4ae9b3c1a050b966e9a5 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:40.794708 31269 tablet_replica.cc:330] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:12:40.795117 31269 raft_consensus.cc:2229] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:40.795753 31269 raft_consensus.cc:2258] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:40.796125 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815cb2db9514c02b0ee3e3cd12b2212" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:12:40.796352 28132 raft_consensus.cc:2454] T 7815cb2db9514c02b0ee3e3cd12b2212 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:12:40.796463 31288 consensus_queue.cc:260] T 013e00c67a1c4ae9b3c1a050b966e9a5 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:40.797327 31269 ts_tablet_manager.cc:1875] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:40.798017 30829 leader_election.cc:302] T 7815cb2db9514c02b0ee3e3cd12b2212 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, ce1496a47d084365ad786cd09e472a2c; no voters: 6d1d7de19459435fbde1999e3dfd226b
I20240503 08:12:40.798666 31227 raft_consensus.cc:2781] T 7815cb2db9514c02b0ee3e3cd12b2212 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:40.798786 31227 raft_consensus.cc:482] T 7815cb2db9514c02b0ee3e3cd12b2212 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:40.798880 31227 raft_consensus.cc:3037] T 7815cb2db9514c02b0ee3e3cd12b2212 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:40.799515 31285 tablet_bootstrap.cc:492] T 7815cb2db9514c02b0ee3e3cd12b2212 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:40.799718 31285 ts_tablet_manager.cc:1367] T 7815cb2db9514c02b0ee3e3cd12b2212 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.049s user 0.018s sys 0.013s
I20240503 08:12:40.800465 31285 raft_consensus.cc:348] T 7815cb2db9514c02b0ee3e3cd12b2212 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:40.800740 31285 raft_consensus.cc:374] T 7815cb2db9514c02b0ee3e3cd12b2212 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:40.800814 31285 raft_consensus.cc:729] T 7815cb2db9514c02b0ee3e3cd12b2212 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:40.801908 28112 tablet_service.cc:1514] Processing DeleteTablet for tablet f2fafbbe2b7c464aa1fb76c25a4427d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:40 UTC) from {username='slave'} at 127.0.0.1:57912
I20240503 08:12:40.801991 31285 consensus_queue.cc:260] T 7815cb2db9514c02b0ee3e3cd12b2212 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:40.802181 31267 tablet_replica.cc:330] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 1b411c6bb25549368041773d7b710d48: stopping tablet replica
I20240503 08:12:40.802398 31267 raft_consensus.cc:2229] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:40.802608 31267 raft_consensus.cc:2258] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:40.803678 31267 ts_tablet_manager.cc:1875] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 1b411c6bb25549368041773d7b710d48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:40.805022 31285 ts_tablet_manager.cc:1398] T 7815cb2db9514c02b0ee3e3cd12b2212 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240503 08:12:40.801739 31227 raft_consensus.cc:504] T 7815cb2db9514c02b0ee3e3cd12b2212 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:40.806602 31227 leader_election.cc:288] T 7815cb2db9514c02b0ee3e3cd12b2212 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Requested vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:40.806950 31307 tablet_replica.cc:330] T f2fafbbe2b7c464aa1fb76c25a4427d6 P ce1496a47d084365ad786cd09e472a2c: stopping tablet replica
I20240503 08:12:40.807340 31307 raft_consensus.cc:2229] T f2fafbbe2b7c464aa1fb76c25a4427d6 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:40.807626 31307 raft_consensus.cc:2258] T f2fafbbe2b7c464aa1fb76c25a4427d6 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:40.807955 31206 heartbeater.cc:503] Master 127.20.230.126:39173 was elected leader, sending a full tablet report...
I20240503 08:12:40.808717 31288 ts_tablet_manager.cc:1398] T 013e00c67a1c4ae9b3c1a050b966e9a5 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.024s user 0.011s sys 0.011s
I20240503 08:12:40.809506 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815cb2db9514c02b0ee3e3cd12b2212" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:12:40.809512 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7815cb2db9514c02b0ee3e3cd12b2212" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:12:40.809705 28132 raft_consensus.cc:3037] T 7815cb2db9514c02b0ee3e3cd12b2212 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:40.809737 30762 raft_consensus.cc:3037] T 7815cb2db9514c02b0ee3e3cd12b2212 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:40.812284 28132 raft_consensus.cc:2454] T 7815cb2db9514c02b0ee3e3cd12b2212 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:12:40.812696 30762 raft_consensus.cc:2454] T 7815cb2db9514c02b0ee3e3cd12b2212 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:12:40.813899 30829 leader_election.cc:302] T 7815cb2db9514c02b0ee3e3cd12b2212 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:40.814535 31227 raft_consensus.cc:2781] T 7815cb2db9514c02b0ee3e3cd12b2212 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:40.815155 31307 ts_tablet_manager.cc:1875] T f2fafbbe2b7c464aa1fb76c25a4427d6 P ce1496a47d084365ad786cd09e472a2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:40.815801 31267 ts_tablet_manager.cc:1888] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 1b411c6bb25549368041773d7b710d48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:40.816021 31267 log.cc:1196] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 1b411c6bb25549368041773d7b710d48: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/f2fafbbe2b7c464aa1fb76c25a4427d6
I20240503 08:12:40.816329 31269 ts_tablet_manager.cc:1888] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:40.816471 31269 log.cc:1196] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/f2fafbbe2b7c464aa1fb76c25a4427d6
I20240503 08:12:40.816920 31269 ts_tablet_manager.cc:1909] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 833582ff68c94269be364ebd187a75db: Deleting consensus metadata
I20240503 08:12:40.817359 31267 ts_tablet_manager.cc:1909] T f2fafbbe2b7c464aa1fb76c25a4427d6 P 1b411c6bb25549368041773d7b710d48: Deleting consensus metadata
I20240503 08:12:40.817893 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet f2fafbbe2b7c464aa1fb76c25a4427d6 (table rebalancer_dynamic_table_new_name [id=6e65df8c3c12429595da5f097a8de5f3]) successfully deleted
I20240503 08:12:40.819046 27848 catalog_manager.cc:4830] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): tablet f2fafbbe2b7c464aa1fb76c25a4427d6 (table rebalancer_dynamic_table_new_name [id=6e65df8c3c12429595da5f097a8de5f3]) successfully deleted
I20240503 08:12:40.823734 31307 ts_tablet_manager.cc:1888] T f2fafbbe2b7c464aa1fb76c25a4427d6 P ce1496a47d084365ad786cd09e472a2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:40.823890 31307 log.cc:1196] T f2fafbbe2b7c464aa1fb76c25a4427d6 P ce1496a47d084365ad786cd09e472a2c: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal/wals/f2fafbbe2b7c464aa1fb76c25a4427d6
I20240503 08:12:40.824461 31307 ts_tablet_manager.cc:1909] T f2fafbbe2b7c464aa1fb76c25a4427d6 P ce1496a47d084365ad786cd09e472a2c: Deleting consensus metadata
I20240503 08:12:40.826171 27846 catalog_manager.cc:4830] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): tablet f2fafbbe2b7c464aa1fb76c25a4427d6 (table rebalancer_dynamic_table_new_name [id=6e65df8c3c12429595da5f097a8de5f3]) successfully deleted
I20240503 08:12:40.826229 31227 raft_consensus.cc:686] T 7815cb2db9514c02b0ee3e3cd12b2212 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Becoming Leader. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Running, Role: LEADER
I20240503 08:12:40.826581 31227 consensus_queue.cc:237] T 7815cb2db9514c02b0ee3e3cd12b2212 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:40.829169 27854 catalog_manager.cc:5484] T 7815cb2db9514c02b0ee3e3cd12b2212 P ce1496a47d084365ad786cd09e472a2c reported cstate change: term changed from 0 to 1, leader changed from <none> to ce1496a47d084365ad786cd09e472a2c (127.20.230.70). New cstate: current_term: 1 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
W20240503 08:12:40.865684 30809 tablet.cc:2369] T 013e00c67a1c4ae9b3c1a050b966e9a5 P 6d1d7de19459435fbde1999e3dfd226b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:12:40.883200 30944 raft_consensus.cc:482] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:40.883422 30944 raft_consensus.cc:504] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:40.884289 30944 leader_election.cc:288] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:40.892335 27854 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:40.903483 31141 tablet_service.cc:1467] Processing CreateTablet for tablet d472290d0d11409ba94b055065126aa8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=959e4a8f2be647fa9701dd5c969b3c7d]), partition=
I20240503 08:12:40.903652 31008 tablet_service.cc:1467] Processing CreateTablet for tablet d472290d0d11409ba94b055065126aa8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=959e4a8f2be647fa9701dd5c969b3c7d]), partition=
I20240503 08:12:40.903681 30741 tablet_service.cc:1467] Processing CreateTablet for tablet d472290d0d11409ba94b055065126aa8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=959e4a8f2be647fa9701dd5c969b3c7d]), partition=
I20240503 08:12:40.903852 31141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d472290d0d11409ba94b055065126aa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:40.903949 31008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d472290d0d11409ba94b055065126aa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:40.904026 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d472290d0d11409ba94b055065126aa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:40.909742 31289 tablet_bootstrap.cc:492] T d472290d0d11409ba94b055065126aa8 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:40.910745 31288 tablet_bootstrap.cc:492] T d472290d0d11409ba94b055065126aa8 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:40.911206 31289 tablet_bootstrap.cc:654] T d472290d0d11409ba94b055065126aa8 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:40.911914 31288 tablet_bootstrap.cc:654] T d472290d0d11409ba94b055065126aa8 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:40.913936 31289 tablet_bootstrap.cc:492] T d472290d0d11409ba94b055065126aa8 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:40.914081 31289 ts_tablet_manager.cc:1367] T d472290d0d11409ba94b055065126aa8 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 08:12:40.914718 31289 raft_consensus.cc:348] T d472290d0d11409ba94b055065126aa8 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:40.914924 31289 raft_consensus.cc:374] T d472290d0d11409ba94b055065126aa8 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:40.914986 31289 raft_consensus.cc:729] T d472290d0d11409ba94b055065126aa8 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:40.916515 31285 tablet_bootstrap.cc:492] T d472290d0d11409ba94b055065126aa8 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:40.915354 31289 consensus_queue.cc:260] T d472290d0d11409ba94b055065126aa8 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:40.917909 31285 tablet_bootstrap.cc:654] T d472290d0d11409ba94b055065126aa8 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:40.917538 31289 ts_tablet_manager.cc:1398] T d472290d0d11409ba94b055065126aa8 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:12:40.923918 31285 tablet_bootstrap.cc:492] T d472290d0d11409ba94b055065126aa8 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:40.924062 31285 ts_tablet_manager.cc:1367] T d472290d0d11409ba94b055065126aa8 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.002s
I20240503 08:12:40.924865 31285 raft_consensus.cc:348] T d472290d0d11409ba94b055065126aa8 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:40.925061 31285 raft_consensus.cc:374] T d472290d0d11409ba94b055065126aa8 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:40.924961 31028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fbe2eaba5344dfaa29e78a0d73f224a" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:40.925149 31285 raft_consensus.cc:729] T d472290d0d11409ba94b055065126aa8 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:40.925371 31285 consensus_queue.cc:260] T d472290d0d11409ba94b055065126aa8 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:40.925474 31028 raft_consensus.cc:2454] T 9fbe2eaba5344dfaa29e78a0d73f224a P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:40.926251 27937 leader_election.cc:302] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:40.926596 31285 ts_tablet_manager.cc:1398] T d472290d0d11409ba94b055065126aa8 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:40.926597 30944 raft_consensus.cc:2781] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:40.926687 30944 raft_consensus.cc:482] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:40.926658 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fbe2eaba5344dfaa29e78a0d73f224a" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:12:40.926781 30944 raft_consensus.cc:3037] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:40.926823 30762 raft_consensus.cc:2454] T 9fbe2eaba5344dfaa29e78a0d73f224a P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:40.929306 30944 raft_consensus.cc:504] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:40.930222 30944 leader_election.cc:288] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Requested vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:40.930886 31288 tablet_bootstrap.cc:492] T d472290d0d11409ba94b055065126aa8 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:40.930944 31028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fbe2eaba5344dfaa29e78a0d73f224a" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:12:40.931031 31288 ts_tablet_manager.cc:1367] T d472290d0d11409ba94b055065126aa8 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.007s
I20240503 08:12:40.931108 31028 raft_consensus.cc:3037] T 9fbe2eaba5344dfaa29e78a0d73f224a P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:40.932080 31288 raft_consensus.cc:348] T d472290d0d11409ba94b055065126aa8 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:40.932272 31288 raft_consensus.cc:374] T d472290d0d11409ba94b055065126aa8 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:40.932371 31288 raft_consensus.cc:729] T d472290d0d11409ba94b055065126aa8 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:40.932590 31288 consensus_queue.cc:260] T d472290d0d11409ba94b055065126aa8 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:40.933127 31288 ts_tablet_manager.cc:1398] T d472290d0d11409ba94b055065126aa8 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240503 08:12:40.933316 31028 raft_consensus.cc:2454] T 9fbe2eaba5344dfaa29e78a0d73f224a P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:40.933354 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fbe2eaba5344dfaa29e78a0d73f224a" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:12:40.933524 30762 raft_consensus.cc:3037] T 9fbe2eaba5344dfaa29e78a0d73f224a P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:40.934227 27937 leader_election.cc:302] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:40.934607 30944 raft_consensus.cc:2781] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:40.935547 30944 raft_consensus.cc:686] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Becoming Leader. State: Replica: 833582ff68c94269be364ebd187a75db, State: Running, Role: LEADER
I20240503 08:12:40.935863 30944 consensus_queue.cc:237] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:40.936070 30762 raft_consensus.cc:2454] T 9fbe2eaba5344dfaa29e78a0d73f224a P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:40.940976 27854 catalog_manager.cc:5484] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db reported cstate change: term changed from 0 to 1, leader changed from <none> to 833582ff68c94269be364ebd187a75db (127.20.230.65). New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
W20240503 08:12:40.973184 31207 tablet.cc:2369] T 013e00c67a1c4ae9b3c1a050b966e9a5 P b00f244c69df401997ea99d492218a98: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:12:41.005888 27854 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240503 08:12:41.006098 27854 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240503 08:12:41.011978 27854 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet c16e3017f7ca40e5b65b0f8e92f2eaa9
I20240503 08:12:41.012874 28112 tablet_service.cc:1514] Processing DeleteTablet for tablet c16e3017f7ca40e5b65b0f8e92f2eaa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:41 UTC) from {username='slave'} at 127.0.0.1:57912
I20240503 08:12:41.013134 31267 tablet_replica.cc:330] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 1b411c6bb25549368041773d7b710d48: stopping tablet replica
I20240503 08:12:41.013345 31267 raft_consensus.cc:2229] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:41.013592 31267 raft_consensus.cc:2258] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:41.014217 31267 ts_tablet_manager.cc:1875] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 1b411c6bb25549368041773d7b710d48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:41.017887 27854 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet ebb368e3f37945b0aa7526873bf0e5c3
I20240503 08:12:41.018311 27854 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet 8d70b2d43b2945938aba969c7b6d931d
I20240503 08:12:41.018497 28242 tablet_service.cc:1514] Processing DeleteTablet for tablet ebb368e3f37945b0aa7526873bf0e5c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:41 UTC) from {username='slave'} at 127.0.0.1:41960
I20240503 08:12:41.018756 31270 tablet_replica.cc:330] T ebb368e3f37945b0aa7526873bf0e5c3 P aae860d7ae2b4002bf7794c278f9cc56: stopping tablet replica
I20240503 08:12:41.018946 31270 raft_consensus.cc:2229] T ebb368e3f37945b0aa7526873bf0e5c3 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:41.018935 31287 raft_consensus.cc:482] T 013e00c67a1c4ae9b3c1a050b966e9a5 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:41.019174 31270 raft_consensus.cc:2258] T ebb368e3f37945b0aa7526873bf0e5c3 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:41.019169 27982 tablet_service.cc:1514] Processing DeleteTablet for tablet c16e3017f7ca40e5b65b0f8e92f2eaa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:41 UTC) from {username='slave'} at 127.0.0.1:55610
I20240503 08:12:41.019086 31287 raft_consensus.cc:504] T 013e00c67a1c4ae9b3c1a050b966e9a5 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:41.019443 31269 tablet_replica.cc:330] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:12:41.019634 31269 raft_consensus.cc:2229] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:41.019810 31270 ts_tablet_manager.cc:1875] T ebb368e3f37945b0aa7526873bf0e5c3 P aae860d7ae2b4002bf7794c278f9cc56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:41.019866 31269 raft_consensus.cc:2258] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:41.020334 31269 ts_tablet_manager.cc:1875] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:41.021023 31287 leader_election.cc:288] T 013e00c67a1c4ae9b3c1a050b966e9a5 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:41.022073 28112 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d70b2d43b2945938aba969c7b6d931d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:41 UTC) from {username='slave'} at 127.0.0.1:57912
I20240503 08:12:41.024377 31308 raft_consensus.cc:482] T 013e00c67a1c4ae9b3c1a050b966e9a5 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:41.024603 31308 raft_consensus.cc:504] T 013e00c67a1c4ae9b3c1a050b966e9a5 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:41.026223 31267 ts_tablet_manager.cc:1888] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 1b411c6bb25549368041773d7b710d48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:41.026603 31308 leader_election.cc:288] T 013e00c67a1c4ae9b3c1a050b966e9a5 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:41.027465 31270 ts_tablet_manager.cc:1888] T ebb368e3f37945b0aa7526873bf0e5c3 P aae860d7ae2b4002bf7794c278f9cc56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:41.027667 31267 log.cc:1196] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 1b411c6bb25549368041773d7b710d48: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/c16e3017f7ca40e5b65b0f8e92f2eaa9
I20240503 08:12:41.027840 31270 log.cc:1196] T ebb368e3f37945b0aa7526873bf0e5c3 P aae860d7ae2b4002bf7794c278f9cc56: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal/wals/ebb368e3f37945b0aa7526873bf0e5c3
I20240503 08:12:41.028254 31269 ts_tablet_manager.cc:1888] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:41.028945 31267 ts_tablet_manager.cc:1909] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 1b411c6bb25549368041773d7b710d48: Deleting consensus metadata
I20240503 08:12:41.029047 31270 ts_tablet_manager.cc:1909] T ebb368e3f37945b0aa7526873bf0e5c3 P aae860d7ae2b4002bf7794c278f9cc56: Deleting consensus metadata
I20240503 08:12:41.030098 27847 catalog_manager.cc:4830] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): tablet ebb368e3f37945b0aa7526873bf0e5c3 (table rebalance_test_table_2 [id=bc6e5facee9743cbb5a079a9f15c3c2f]) successfully deleted
I20240503 08:12:41.030743 31269 log.cc:1196] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/c16e3017f7ca40e5b65b0f8e92f2eaa9
I20240503 08:12:41.030771 27848 catalog_manager.cc:4830] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): tablet c16e3017f7ca40e5b65b0f8e92f2eaa9 (table rebalance_test_table_2 [id=bc6e5facee9743cbb5a079a9f15c3c2f]) successfully deleted
I20240503 08:12:41.031091 31267 tablet_replica.cc:330] T 8d70b2d43b2945938aba969c7b6d931d P 1b411c6bb25549368041773d7b710d48: stopping tablet replica
I20240503 08:12:41.031152 31269 ts_tablet_manager.cc:1909] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P 833582ff68c94269be364ebd187a75db: Deleting consensus metadata
I20240503 08:12:41.031373 31267 raft_consensus.cc:2229] T 8d70b2d43b2945938aba969c7b6d931d P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:41.031651 31267 raft_consensus.cc:2258] T 8d70b2d43b2945938aba969c7b6d931d P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:41.032358 31267 ts_tablet_manager.cc:1875] T 8d70b2d43b2945938aba969c7b6d931d P 1b411c6bb25549368041773d7b710d48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:41.033430 28373 tablet_service.cc:1514] Processing DeleteTablet for tablet c16e3017f7ca40e5b65b0f8e92f2eaa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:41 UTC) from {username='slave'} at 127.0.0.1:34786
I20240503 08:12:41.033696 31268 tablet_replica.cc:330] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P d09a11a03989421b9f28019dc1c14cc2: stopping tablet replica
I20240503 08:12:41.033916 31268 raft_consensus.cc:2229] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:41.034153 31268 raft_consensus.cc:2258] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:41.034811 31268 ts_tablet_manager.cc:1875] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P d09a11a03989421b9f28019dc1c14cc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:41.035949 27982 tablet_service.cc:1514] Processing DeleteTablet for tablet ebb368e3f37945b0aa7526873bf0e5c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:41 UTC) from {username='slave'} at 127.0.0.1:55610
I20240503 08:12:41.036749 27982 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d70b2d43b2945938aba969c7b6d931d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:41 UTC) from {username='slave'} at 127.0.0.1:55610
I20240503 08:12:41.036761 28373 tablet_service.cc:1514] Processing DeleteTablet for tablet ebb368e3f37945b0aa7526873bf0e5c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:41 UTC) from {username='slave'} at 127.0.0.1:34786
I20240503 08:12:41.037899 28373 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d70b2d43b2945938aba969c7b6d931d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:41 UTC) from {username='slave'} at 127.0.0.1:34786
I20240503 08:12:41.038209 31269 tablet_replica.cc:330] T ebb368e3f37945b0aa7526873bf0e5c3 P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:12:41.038426 31269 raft_consensus.cc:2229] T ebb368e3f37945b0aa7526873bf0e5c3 P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:41.038734 31269 raft_consensus.cc:2258] T ebb368e3f37945b0aa7526873bf0e5c3 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:41.042651 31269 ts_tablet_manager.cc:1875] T ebb368e3f37945b0aa7526873bf0e5c3 P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:41.047878 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet c16e3017f7ca40e5b65b0f8e92f2eaa9 (table rebalance_test_table_2 [id=bc6e5facee9743cbb5a079a9f15c3c2f]) successfully deleted
I20240503 08:12:41.050379 31268 ts_tablet_manager.cc:1888] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P d09a11a03989421b9f28019dc1c14cc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:41.051868 31268 log.cc:1196] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P d09a11a03989421b9f28019dc1c14cc2: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/wals/c16e3017f7ca40e5b65b0f8e92f2eaa9
I20240503 08:12:41.052644 31268 ts_tablet_manager.cc:1909] T c16e3017f7ca40e5b65b0f8e92f2eaa9 P d09a11a03989421b9f28019dc1c14cc2: Deleting consensus metadata
I20240503 08:12:41.054117 31268 tablet_replica.cc:330] T ebb368e3f37945b0aa7526873bf0e5c3 P d09a11a03989421b9f28019dc1c14cc2: stopping tablet replica
I20240503 08:12:41.054297 27849 catalog_manager.cc:4830] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): tablet c16e3017f7ca40e5b65b0f8e92f2eaa9 (table rebalance_test_table_2 [id=bc6e5facee9743cbb5a079a9f15c3c2f]) successfully deleted
I20240503 08:12:41.054373 31268 raft_consensus.cc:2229] T ebb368e3f37945b0aa7526873bf0e5c3 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:41.054754 31268 raft_consensus.cc:2258] T ebb368e3f37945b0aa7526873bf0e5c3 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:41.055575 31268 ts_tablet_manager.cc:1875] T ebb368e3f37945b0aa7526873bf0e5c3 P d09a11a03989421b9f28019dc1c14cc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:41.060875 31269 ts_tablet_manager.cc:1888] T ebb368e3f37945b0aa7526873bf0e5c3 P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:41.061055 31269 log.cc:1196] T ebb368e3f37945b0aa7526873bf0e5c3 P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/ebb368e3f37945b0aa7526873bf0e5c3
I20240503 08:12:41.061480 31269 ts_tablet_manager.cc:1909] T ebb368e3f37945b0aa7526873bf0e5c3 P 833582ff68c94269be364ebd187a75db: Deleting consensus metadata
I20240503 08:12:41.067359 31268 ts_tablet_manager.cc:1888] T ebb368e3f37945b0aa7526873bf0e5c3 P d09a11a03989421b9f28019dc1c14cc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:41.067799 31269 tablet_replica.cc:330] T 8d70b2d43b2945938aba969c7b6d931d P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:12:41.068073 31269 raft_consensus.cc:2229] T 8d70b2d43b2945938aba969c7b6d931d P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:41.068470 31269 raft_consensus.cc:2258] T 8d70b2d43b2945938aba969c7b6d931d P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:41.069322 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet ebb368e3f37945b0aa7526873bf0e5c3 (table rebalance_test_table_2 [id=bc6e5facee9743cbb5a079a9f15c3c2f]) successfully deleted
I20240503 08:12:41.073019 31161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "013e00c67a1c4ae9b3c1a050b966e9a5" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:12:41.073566 31161 raft_consensus.cc:2454] T 013e00c67a1c4ae9b3c1a050b966e9a5 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:12:41.074219 31268 log.cc:1196] T ebb368e3f37945b0aa7526873bf0e5c3 P d09a11a03989421b9f28019dc1c14cc2: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/wals/ebb368e3f37945b0aa7526873bf0e5c3
I20240503 08:12:41.074746 31268 ts_tablet_manager.cc:1909] T ebb368e3f37945b0aa7526873bf0e5c3 P d09a11a03989421b9f28019dc1c14cc2: Deleting consensus metadata
I20240503 08:12:41.074898 31269 ts_tablet_manager.cc:1875] T 8d70b2d43b2945938aba969c7b6d931d P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:41.075722 31268 tablet_replica.cc:330] T 8d70b2d43b2945938aba969c7b6d931d P d09a11a03989421b9f28019dc1c14cc2: stopping tablet replica
I20240503 08:12:41.075937 31268 raft_consensus.cc:2229] T 8d70b2d43b2945938aba969c7b6d931d P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:41.077558 30694 leader_election.cc:302] T 013e00c67a1c4ae9b3c1a050b966e9a5 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, b00f244c69df401997ea99d492218a98; no voters:
I20240503 08:12:41.077961 31028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "013e00c67a1c4ae9b3c1a050b966e9a5" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:41.078230 31028 raft_consensus.cc:2454] T 013e00c67a1c4ae9b3c1a050b966e9a5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 0.
I20240503 08:12:41.078719 31095 leader_election.cc:302] T 013e00c67a1c4ae9b3c1a050b966e9a5 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:41.078541 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "013e00c67a1c4ae9b3c1a050b966e9a5" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:12:41.079200 31308 raft_consensus.cc:2781] T 013e00c67a1c4ae9b3c1a050b966e9a5 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:41.079355 31308 raft_consensus.cc:482] T 013e00c67a1c4ae9b3c1a050b966e9a5 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:41.079401 30762 raft_consensus.cc:2454] T 013e00c67a1c4ae9b3c1a050b966e9a5 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 0.
I20240503 08:12:41.079471 31308 raft_consensus.cc:3037] T 013e00c67a1c4ae9b3c1a050b966e9a5 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:41.080217 31267 ts_tablet_manager.cc:1888] T 8d70b2d43b2945938aba969c7b6d931d P 1b411c6bb25549368041773d7b710d48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:41.080338 31267 log.cc:1196] T 8d70b2d43b2945938aba969c7b6d931d P 1b411c6bb25549368041773d7b710d48: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/8d70b2d43b2945938aba969c7b6d931d
I20240503 08:12:41.080761 31267 ts_tablet_manager.cc:1909] T 8d70b2d43b2945938aba969c7b6d931d P 1b411c6bb25549368041773d7b710d48: Deleting consensus metadata
I20240503 08:12:41.081818 27848 catalog_manager.cc:4830] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): tablet 8d70b2d43b2945938aba969c7b6d931d (table rebalance_test_table_2 [id=bc6e5facee9743cbb5a079a9f15c3c2f]) successfully deleted
I20240503 08:12:41.082176 27849 catalog_manager.cc:4830] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): tablet ebb368e3f37945b0aa7526873bf0e5c3 (table rebalance_test_table_2 [id=bc6e5facee9743cbb5a079a9f15c3c2f]) successfully deleted
I20240503 08:12:41.082239 31308 raft_consensus.cc:504] T 013e00c67a1c4ae9b3c1a050b966e9a5 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:41.082736 31287 raft_consensus.cc:2781] T 013e00c67a1c4ae9b3c1a050b966e9a5 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:41.082919 31287 raft_consensus.cc:482] T 013e00c67a1c4ae9b3c1a050b966e9a5 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:41.083055 31287 raft_consensus.cc:3037] T 013e00c67a1c4ae9b3c1a050b966e9a5 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:41.084333 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "013e00c67a1c4ae9b3c1a050b966e9a5" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:12:41.085106 31308 leader_election.cc:288] T 013e00c67a1c4ae9b3c1a050b966e9a5 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 election: Requested vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:41.085393 31028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "013e00c67a1c4ae9b3c1a050b966e9a5" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:12:41.085623 31028 raft_consensus.cc:3037] T 013e00c67a1c4ae9b3c1a050b966e9a5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:41.086086 31287 raft_consensus.cc:504] T 013e00c67a1c4ae9b3c1a050b966e9a5 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:41.086272 31269 ts_tablet_manager.cc:1888] T 8d70b2d43b2945938aba969c7b6d931d P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:41.086403 31269 log.cc:1196] T 8d70b2d43b2945938aba969c7b6d931d P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/8d70b2d43b2945938aba969c7b6d931d
I20240503 08:12:41.086894 31269 ts_tablet_manager.cc:1909] T 8d70b2d43b2945938aba969c7b6d931d P 833582ff68c94269be364ebd187a75db: Deleting consensus metadata
I20240503 08:12:41.087407 31287 leader_election.cc:288] T 013e00c67a1c4ae9b3c1a050b966e9a5 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Requested vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:41.088521 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet 8d70b2d43b2945938aba969c7b6d931d (table rebalance_test_table_2 [id=bc6e5facee9743cbb5a079a9f15c3c2f]) successfully deleted
I20240503 08:12:41.088754 30762 raft_consensus.cc:2379] T 013e00c67a1c4ae9b3c1a050b966e9a5 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b00f244c69df401997ea99d492218a98 in current term 1: Already voted for candidate 6d1d7de19459435fbde1999e3dfd226b in this term.
I20240503 08:12:41.090607 31028 raft_consensus.cc:2454] T 013e00c67a1c4ae9b3c1a050b966e9a5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 1.
I20240503 08:12:41.088143 31161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "013e00c67a1c4ae9b3c1a050b966e9a5" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:12:41.091492 31161 raft_consensus.cc:2379] T 013e00c67a1c4ae9b3c1a050b966e9a5 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d1d7de19459435fbde1999e3dfd226b in current term 1: Already voted for candidate b00f244c69df401997ea99d492218a98 in this term.
I20240503 08:12:41.091066 31095 leader_election.cc:302] T 013e00c67a1c4ae9b3c1a050b966e9a5 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b00f244c69df401997ea99d492218a98, ced21ca983b749d0b4d4f6a0e2426bf8; no voters: 6d1d7de19459435fbde1999e3dfd226b
I20240503 08:12:41.092491 31308 raft_consensus.cc:2781] T 013e00c67a1c4ae9b3c1a050b966e9a5 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:41.092948 31308 raft_consensus.cc:686] T 013e00c67a1c4ae9b3c1a050b966e9a5 P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Becoming Leader. State: Replica: b00f244c69df401997ea99d492218a98, State: Running, Role: LEADER
I20240503 08:12:41.093442 31308 consensus_queue.cc:237] T 013e00c67a1c4ae9b3c1a050b966e9a5 P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:41.094028 31268 raft_consensus.cc:2258] T 8d70b2d43b2945938aba969c7b6d931d P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:41.094806 31268 ts_tablet_manager.cc:1875] T 8d70b2d43b2945938aba969c7b6d931d P d09a11a03989421b9f28019dc1c14cc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:41.098177 27854 catalog_manager.cc:5484] T 013e00c67a1c4ae9b3c1a050b966e9a5 P b00f244c69df401997ea99d492218a98 reported cstate change: term changed from 0 to 1, leader changed from <none> to b00f244c69df401997ea99d492218a98 (127.20.230.72). New cstate: current_term: 1 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:12:41.099933 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "013e00c67a1c4ae9b3c1a050b966e9a5" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:12:41.100329 31027 raft_consensus.cc:2379] T 013e00c67a1c4ae9b3c1a050b966e9a5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d1d7de19459435fbde1999e3dfd226b in current term 1: Already voted for candidate b00f244c69df401997ea99d492218a98 in this term.
I20240503 08:12:41.100937 31028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "013e00c67a1c4ae9b3c1a050b966e9a5" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:41.101132 31028 raft_consensus.cc:2379] T 013e00c67a1c4ae9b3c1a050b966e9a5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d1d7de19459435fbde1999e3dfd226b in current term 1: Already voted for candidate b00f244c69df401997ea99d492218a98 in this term.
I20240503 08:12:41.104048 30695 leader_election.cc:302] T 013e00c67a1c4ae9b3c1a050b966e9a5 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b; no voters: b00f244c69df401997ea99d492218a98, ced21ca983b749d0b4d4f6a0e2426bf8
I20240503 08:12:41.104386 31287 raft_consensus.cc:2726] T 013e00c67a1c4ae9b3c1a050b966e9a5 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 08:12:41.104513 31268 ts_tablet_manager.cc:1888] T 8d70b2d43b2945938aba969c7b6d931d P d09a11a03989421b9f28019dc1c14cc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:41.104648 31268 log.cc:1196] T 8d70b2d43b2945938aba969c7b6d931d P d09a11a03989421b9f28019dc1c14cc2: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/wals/8d70b2d43b2945938aba969c7b6d931d
I20240503 08:12:41.105065 31268 ts_tablet_manager.cc:1909] T 8d70b2d43b2945938aba969c7b6d931d P d09a11a03989421b9f28019dc1c14cc2: Deleting consensus metadata
I20240503 08:12:41.106140 27849 catalog_manager.cc:4830] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): tablet 8d70b2d43b2945938aba969c7b6d931d (table rebalance_test_table_2 [id=bc6e5facee9743cbb5a079a9f15c3c2f]) successfully deleted
I20240503 08:12:41.117127 31287 raft_consensus.cc:482] T d472290d0d11409ba94b055065126aa8 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:41.117295 31287 raft_consensus.cc:504] T d472290d0d11409ba94b055065126aa8 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:41.120762 31287 leader_election.cc:288] T d472290d0d11409ba94b055065126aa8 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:41.130300 31161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d472290d0d11409ba94b055065126aa8" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:12:41.130542 31161 raft_consensus.cc:2454] T d472290d0d11409ba94b055065126aa8 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:12:41.130976 30694 leader_election.cc:302] T d472290d0d11409ba94b055065126aa8 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, b00f244c69df401997ea99d492218a98; no voters:
I20240503 08:12:41.131285 31287 raft_consensus.cc:2781] T d472290d0d11409ba94b055065126aa8 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:41.131387 31287 raft_consensus.cc:482] T d472290d0d11409ba94b055065126aa8 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:41.131495 31287 raft_consensus.cc:3037] T d472290d0d11409ba94b055065126aa8 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:41.133428 31287 raft_consensus.cc:504] T d472290d0d11409ba94b055065126aa8 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:41.135385 30741 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:41.136384 28112 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 08:12:41.141595 31141 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:12:41.143996 31287 leader_election.cc:288] T d472290d0d11409ba94b055065126aa8 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Requested vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:41.144454 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d472290d0d11409ba94b055065126aa8" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:12:41.144651 31027 raft_consensus.cc:3037] T d472290d0d11409ba94b055065126aa8 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:41.145987 31028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d472290d0d11409ba94b055065126aa8" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:41.147534 31027 raft_consensus.cc:2454] T d472290d0d11409ba94b055065126aa8 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:12:41.148483 31160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d472290d0d11409ba94b055065126aa8" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:12:41.148736 31160 raft_consensus.cc:3037] T d472290d0d11409ba94b055065126aa8 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:41.151525 31160 raft_consensus.cc:2454] T d472290d0d11409ba94b055065126aa8 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:12:41.153040 30695 leader_election.cc:302] T d472290d0d11409ba94b055065126aa8 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:41.156152 31287 raft_consensus.cc:2781] T d472290d0d11409ba94b055065126aa8 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:41.170231 27982 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:12:41.205834 28242 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:12:41.208084 30874 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:12:41.222651 31008 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:41.240079 28373 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:12:41.244499 31287 raft_consensus.cc:686] T d472290d0d11409ba94b055065126aa8 P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Becoming Leader. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Running, Role: LEADER
I20240503 08:12:41.245049 31287 consensus_queue.cc:237] T d472290d0d11409ba94b055065126aa8 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:41.245755 31227 consensus_queue.cc:1035] T 7815cb2db9514c02b0ee3e3cd12b2212 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:41.258565 27854 catalog_manager.cc:5484] T d472290d0d11409ba94b055065126aa8 P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69). New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:12:41.323086 31309 consensus_queue.cc:1035] T 7815cb2db9514c02b0ee3e3cd12b2212 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:41.364921 27854 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:41.367770 30944 consensus_queue.cc:1035] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:41.374441 31027 raft_consensus.cc:1261] T d472290d0d11409ba94b055065126aa8 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:41.374431 31161 raft_consensus.cc:1261] T d472290d0d11409ba94b055065126aa8 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:41.375257 31371 consensus_queue.cc:1035] T d472290d0d11409ba94b055065126aa8 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:41.375844 31371 consensus_queue.cc:1035] T d472290d0d11409ba94b055065126aa8 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:41.398020 27854 catalog_manager.cc:5576] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915) does not have the latest schema for tablet d472290d0d11409ba94b055065126aa8 (table rebalancer_dynamic_table [id=959e4a8f2be647fa9701dd5c969b3c7d]). Expected version 1 got 0
I20240503 08:12:41.400372 31378 mvcc.cc:205] Tried to move back new op lower bound from 7023509345780420608 to 7023509345267929088. Current Snapshot: MvccSnapshot[applied={T|T < 7023509345780420608}]
I20240503 08:12:41.402451 30944 consensus_queue.cc:1035] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:41.403543 31383 mvcc.cc:205] Tried to move back new op lower bound from 7023509345780420608 to 7023509345267929088. Current Snapshot: MvccSnapshot[applied={T|T < 7023509345780420608}]
I20240503 08:12:41.412436 27854 catalog_manager.cc:5576] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915) does not have the latest schema for tablet d472290d0d11409ba94b055065126aa8 (table rebalancer_dynamic_table [id=959e4a8f2be647fa9701dd5c969b3c7d]). Expected version 1 got 0
I20240503 08:12:41.413728 31389 tablet.cc:1722] T d472290d0d11409ba94b055065126aa8 P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:41.414024 27854 catalog_manager.cc:5576] TS b00f244c69df401997ea99d492218a98 (127.20.230.72:40893) does not have the latest schema for tablet d472290d0d11409ba94b055065126aa8 (table rebalancer_dynamic_table [id=959e4a8f2be647fa9701dd5c969b3c7d]). Expected version 1 got 0
I20240503 08:12:41.415156 31389 tablet.cc:1620] T d472290d0d11409ba94b055065126aa8 P ced21ca983b749d0b4d4f6a0e2426bf8: MemRowSet was empty: no flush needed.
I20240503 08:12:41.421689 31392 tablet.cc:1722] T d472290d0d11409ba94b055065126aa8 P b00f244c69df401997ea99d492218a98: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:41.422852 27854 catalog_manager.cc:5576] TS b00f244c69df401997ea99d492218a98 (127.20.230.72:40893) does not have the latest schema for tablet d472290d0d11409ba94b055065126aa8 (table rebalancer_dynamic_table [id=959e4a8f2be647fa9701dd5c969b3c7d]). Expected version 1 got 0
I20240503 08:12:41.423902 31392 tablet.cc:1620] T d472290d0d11409ba94b055065126aa8 P b00f244c69df401997ea99d492218a98: MemRowSet was empty: no flush needed.
I20240503 08:12:41.424851 31384 mvcc.cc:205] Tried to move back new op lower bound from 7023509345780420608 to 7023509345267929088. Current Snapshot: MvccSnapshot[applied={T|T < 7023509345780420608}]
I20240503 08:12:41.427436 31388 tablet.cc:1722] T d472290d0d11409ba94b055065126aa8 P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:41.428471 31388 tablet.cc:1620] T d472290d0d11409ba94b055065126aa8 P 6d1d7de19459435fbde1999e3dfd226b: MemRowSet was empty: no flush needed.
I20240503 08:12:41.437659 27862 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=959e4a8f2be647fa9701dd5c969b3c7d] alter complete (version 1)
I20240503 08:12:41.538815 27859 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240503 08:12:41.539045 27859 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240503 08:12:41.543076 27859 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet c6498d83236c48f88bd21c303c9c9ab3
I20240503 08:12:41.543941 27982 tablet_service.cc:1514] Processing DeleteTablet for tablet c6498d83236c48f88bd21c303c9c9ab3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:41 UTC) from {username='slave'} at 127.0.0.1:55610
I20240503 08:12:41.544229 28112 tablet_service.cc:1514] Processing DeleteTablet for tablet c6498d83236c48f88bd21c303c9c9ab3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:41 UTC) from {username='slave'} at 127.0.0.1:57912
I20240503 08:12:41.544397 31269 tablet_replica.cc:330] T c6498d83236c48f88bd21c303c9c9ab3 P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:12:41.544517 31267 tablet_replica.cc:330] T c6498d83236c48f88bd21c303c9c9ab3 P 1b411c6bb25549368041773d7b710d48: stopping tablet replica
I20240503 08:12:41.544628 31269 raft_consensus.cc:2229] T c6498d83236c48f88bd21c303c9c9ab3 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:41.544720 31267 raft_consensus.cc:2229] T c6498d83236c48f88bd21c303c9c9ab3 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:41.544862 31269 raft_consensus.cc:2258] T c6498d83236c48f88bd21c303c9c9ab3 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:41.544907 31267 raft_consensus.cc:2258] T c6498d83236c48f88bd21c303c9c9ab3 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:41.545464 31267 ts_tablet_manager.cc:1875] T c6498d83236c48f88bd21c303c9c9ab3 P 1b411c6bb25549368041773d7b710d48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:41.545589 31269 ts_tablet_manager.cc:1875] T c6498d83236c48f88bd21c303c9c9ab3 P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:41.548995 27859 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet 6bc2a8940e5b42eb8152be26a9952662
I20240503 08:12:41.549162 28242 tablet_service.cc:1514] Processing DeleteTablet for tablet c6498d83236c48f88bd21c303c9c9ab3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:41 UTC) from {username='slave'} at 127.0.0.1:41960
I20240503 08:12:41.550154 27854 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:41.551549 27982 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bc2a8940e5b42eb8152be26a9952662 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:41 UTC) from {username='slave'} at 127.0.0.1:55610
I20240503 08:12:41.549726 31308 consensus_queue.cc:1035] T 013e00c67a1c4ae9b3c1a050b966e9a5 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240503 08:12:41.552536 31267 ts_tablet_manager.cc:1888] T c6498d83236c48f88bd21c303c9c9ab3 P 1b411c6bb25549368041773d7b710d48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:41.552659 31267 log.cc:1196] T c6498d83236c48f88bd21c303c9c9ab3 P 1b411c6bb25549368041773d7b710d48: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/c6498d83236c48f88bd21c303c9c9ab3
I20240503 08:12:41.552992 31267 ts_tablet_manager.cc:1909] T c6498d83236c48f88bd21c303c9c9ab3 P 1b411c6bb25549368041773d7b710d48: Deleting consensus metadata
I20240503 08:12:41.560778 31388 tablet.cc:1722] T d472290d0d11409ba94b055065126aa8 P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:41.561285 31388 tablet.cc:1620] T d472290d0d11409ba94b055065126aa8 P 6d1d7de19459435fbde1999e3dfd226b: MemRowSet was empty: no flush needed.
I20240503 08:12:41.562254 31389 tablet.cc:1722] T d472290d0d11409ba94b055065126aa8 P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:41.562672 31389 tablet.cc:1620] T d472290d0d11409ba94b055065126aa8 P ced21ca983b749d0b4d4f6a0e2426bf8: MemRowSet was empty: no flush needed.
I20240503 08:12:41.562631 31392 tablet.cc:1722] T d472290d0d11409ba94b055065126aa8 P b00f244c69df401997ea99d492218a98: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:41.563041 31392 tablet.cc:1620] T d472290d0d11409ba94b055065126aa8 P b00f244c69df401997ea99d492218a98: MemRowSet was empty: no flush needed.
I20240503 08:12:41.566131 27848 catalog_manager.cc:4830] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): tablet c6498d83236c48f88bd21c303c9c9ab3 (table rebalance_test_table_4 [id=acb2583756a845cab3b0cb05c246fe65]) successfully deleted
I20240503 08:12:41.568531 27859 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet 48746a3d20534795b50aae78bfd660fe
I20240503 08:12:41.569252 28243 tablet_service.cc:1514] Processing DeleteTablet for tablet 48746a3d20534795b50aae78bfd660fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:41 UTC) from {username='slave'} at 127.0.0.1:41960
I20240503 08:12:41.569844 28373 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bc2a8940e5b42eb8152be26a9952662 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:41 UTC) from {username='slave'} at 127.0.0.1:34786
I20240503 08:12:41.572019 31268 tablet_replica.cc:330] T 6bc2a8940e5b42eb8152be26a9952662 P d09a11a03989421b9f28019dc1c14cc2: stopping tablet replica
I20240503 08:12:41.572252 31268 raft_consensus.cc:2229] T 6bc2a8940e5b42eb8152be26a9952662 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:41.572500 31268 raft_consensus.cc:2258] T 6bc2a8940e5b42eb8152be26a9952662 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:41.573170 31268 ts_tablet_manager.cc:1875] T 6bc2a8940e5b42eb8152be26a9952662 P d09a11a03989421b9f28019dc1c14cc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:41.573724 27982 tablet_service.cc:1514] Processing DeleteTablet for tablet 48746a3d20534795b50aae78bfd660fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:41 UTC) from {username='slave'} at 127.0.0.1:55610
I20240503 08:12:41.574283 28373 tablet_service.cc:1514] Processing DeleteTablet for tablet 48746a3d20534795b50aae78bfd660fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:41 UTC) from {username='slave'} at 127.0.0.1:34786
I20240503 08:12:41.578063 28112 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bc2a8940e5b42eb8152be26a9952662 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:41 UTC) from {username='slave'} at 127.0.0.1:57912
I20240503 08:12:41.578100 31353 consensus_queue.cc:1035] T 013e00c67a1c4ae9b3c1a050b966e9a5 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:41.579366 31268 ts_tablet_manager.cc:1888] T 6bc2a8940e5b42eb8152be26a9952662 P d09a11a03989421b9f28019dc1c14cc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:41.579828 31268 log.cc:1196] T 6bc2a8940e5b42eb8152be26a9952662 P d09a11a03989421b9f28019dc1c14cc2: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/wals/6bc2a8940e5b42eb8152be26a9952662
I20240503 08:12:41.580278 31268 ts_tablet_manager.cc:1909] T 6bc2a8940e5b42eb8152be26a9952662 P d09a11a03989421b9f28019dc1c14cc2: Deleting consensus metadata
I20240503 08:12:41.580497 31267 tablet_replica.cc:330] T 6bc2a8940e5b42eb8152be26a9952662 P 1b411c6bb25549368041773d7b710d48: stopping tablet replica
I20240503 08:12:41.580588 27862 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=959e4a8f2be647fa9701dd5c969b3c7d] alter complete (version 2)
I20240503 08:12:41.580793 31267 raft_consensus.cc:2229] T 6bc2a8940e5b42eb8152be26a9952662 P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:41.581696 31268 tablet_replica.cc:330] T 48746a3d20534795b50aae78bfd660fe P d09a11a03989421b9f28019dc1c14cc2: stopping tablet replica
I20240503 08:12:41.581831 27849 catalog_manager.cc:4830] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): tablet 6bc2a8940e5b42eb8152be26a9952662 (table rebalance_test_table_4 [id=acb2583756a845cab3b0cb05c246fe65]) successfully deleted
I20240503 08:12:41.582129 31267 raft_consensus.cc:2258] T 6bc2a8940e5b42eb8152be26a9952662 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:41.582846 31267 ts_tablet_manager.cc:1875] T 6bc2a8940e5b42eb8152be26a9952662 P 1b411c6bb25549368041773d7b710d48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:41.583112 31268 raft_consensus.cc:2229] T 48746a3d20534795b50aae78bfd660fe P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:41.583317 31268 raft_consensus.cc:2258] T 48746a3d20534795b50aae78bfd660fe P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:41.587810 31268 ts_tablet_manager.cc:1875] T 48746a3d20534795b50aae78bfd660fe P d09a11a03989421b9f28019dc1c14cc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:41.595520 31269 ts_tablet_manager.cc:1888] T c6498d83236c48f88bd21c303c9c9ab3 P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:41.595698 31269 log.cc:1196] T c6498d83236c48f88bd21c303c9c9ab3 P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/c6498d83236c48f88bd21c303c9c9ab3
I20240503 08:12:41.596127 31269 ts_tablet_manager.cc:1909] T c6498d83236c48f88bd21c303c9c9ab3 P 833582ff68c94269be364ebd187a75db: Deleting consensus metadata
I20240503 08:12:41.597047 31269 tablet_replica.cc:330] T 6bc2a8940e5b42eb8152be26a9952662 P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:12:41.597257 31269 raft_consensus.cc:2229] T 6bc2a8940e5b42eb8152be26a9952662 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:41.597505 31269 raft_consensus.cc:2258] T 6bc2a8940e5b42eb8152be26a9952662 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:41.598090 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet c6498d83236c48f88bd21c303c9c9ab3 (table rebalance_test_table_4 [id=acb2583756a845cab3b0cb05c246fe65]) successfully deleted
I20240503 08:12:41.598666 31409 tablet_replica.cc:330] T c6498d83236c48f88bd21c303c9c9ab3 P aae860d7ae2b4002bf7794c278f9cc56: stopping tablet replica
I20240503 08:12:41.599329 31409 raft_consensus.cc:2229] T c6498d83236c48f88bd21c303c9c9ab3 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:41.599720 31409 raft_consensus.cc:2258] T c6498d83236c48f88bd21c303c9c9ab3 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:41.600286 31409 ts_tablet_manager.cc:1875] T c6498d83236c48f88bd21c303c9c9ab3 P aae860d7ae2b4002bf7794c278f9cc56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:41.602257 31269 ts_tablet_manager.cc:1875] T 6bc2a8940e5b42eb8152be26a9952662 P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:41.604650 31267 ts_tablet_manager.cc:1888] T 6bc2a8940e5b42eb8152be26a9952662 P 1b411c6bb25549368041773d7b710d48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:41.604771 31267 log.cc:1196] T 6bc2a8940e5b42eb8152be26a9952662 P 1b411c6bb25549368041773d7b710d48: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/6bc2a8940e5b42eb8152be26a9952662
I20240503 08:12:41.605191 31267 ts_tablet_manager.cc:1909] T 6bc2a8940e5b42eb8152be26a9952662 P 1b411c6bb25549368041773d7b710d48: Deleting consensus metadata
I20240503 08:12:41.606225 31268 ts_tablet_manager.cc:1888] T 48746a3d20534795b50aae78bfd660fe P d09a11a03989421b9f28019dc1c14cc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:41.606392 31268 log.cc:1196] T 48746a3d20534795b50aae78bfd660fe P d09a11a03989421b9f28019dc1c14cc2: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/wals/48746a3d20534795b50aae78bfd660fe
I20240503 08:12:41.608091 27848 catalog_manager.cc:4830] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): tablet 6bc2a8940e5b42eb8152be26a9952662 (table rebalance_test_table_4 [id=acb2583756a845cab3b0cb05c246fe65]) successfully deleted
I20240503 08:12:41.608268 31268 ts_tablet_manager.cc:1909] T 48746a3d20534795b50aae78bfd660fe P d09a11a03989421b9f28019dc1c14cc2: Deleting consensus metadata
I20240503 08:12:41.609064 27849 catalog_manager.cc:4830] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): tablet 48746a3d20534795b50aae78bfd660fe (table rebalance_test_table_4 [id=acb2583756a845cab3b0cb05c246fe65]) successfully deleted
I20240503 08:12:41.615061 27859 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_extra_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:41.625130 31269 ts_tablet_manager.cc:1888] T 6bc2a8940e5b42eb8152be26a9952662 P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:41.625377 31269 log.cc:1196] T 6bc2a8940e5b42eb8152be26a9952662 P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/6bc2a8940e5b42eb8152be26a9952662
I20240503 08:12:41.626128 31269 ts_tablet_manager.cc:1909] T 6bc2a8940e5b42eb8152be26a9952662 P 833582ff68c94269be364ebd187a75db: Deleting consensus metadata
I20240503 08:12:41.627264 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet 6bc2a8940e5b42eb8152be26a9952662 (table rebalance_test_table_4 [id=acb2583756a845cab3b0cb05c246fe65]) successfully deleted
I20240503 08:12:41.628055 31269 tablet_replica.cc:330] T 48746a3d20534795b50aae78bfd660fe P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:12:41.628273 31269 raft_consensus.cc:2229] T 48746a3d20534795b50aae78bfd660fe P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:41.629879 31141 tablet_service.cc:1467] Processing CreateTablet for tablet d808d27e71a5486f9773ad4b239d1cfe (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=7358500a74314a5a8795ddb89337f666]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:41.629892 30873 tablet_service.cc:1467] Processing CreateTablet for tablet 6c2824aa878845e098f600f1fcfea087 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=7358500a74314a5a8795ddb89337f666]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:41.630300 31141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d808d27e71a5486f9773ad4b239d1cfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:41.630321 30873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c2824aa878845e098f600f1fcfea087. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:41.631631 31269 raft_consensus.cc:2258] T 48746a3d20534795b50aae78bfd660fe P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:41.632330 31269 ts_tablet_manager.cc:1875] T 48746a3d20534795b50aae78bfd660fe P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:41.634130 30874 tablet_service.cc:1467] Processing CreateTablet for tablet d808d27e71a5486f9773ad4b239d1cfe (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=7358500a74314a5a8795ddb89337f666]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:41.634503 30874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d808d27e71a5486f9773ad4b239d1cfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:41.637859 30741 tablet_service.cc:1467] Processing CreateTablet for tablet 6c2824aa878845e098f600f1fcfea087 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=7358500a74314a5a8795ddb89337f666]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:41.638295 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c2824aa878845e098f600f1fcfea087. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:41.642038 28373 tablet_service.cc:1467] Processing CreateTablet for tablet d808d27e71a5486f9773ad4b239d1cfe (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=7358500a74314a5a8795ddb89337f666]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:41.642468 28373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d808d27e71a5486f9773ad4b239d1cfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:41.649741 31418 tablet_bootstrap.cc:492] T 6c2824aa878845e098f600f1fcfea087 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:41.651922 31418 tablet_bootstrap.cc:654] T 6c2824aa878845e098f600f1fcfea087 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:41.654829 31269 ts_tablet_manager.cc:1888] T 48746a3d20534795b50aae78bfd660fe P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:41.654970 31269 log.cc:1196] T 48746a3d20534795b50aae78bfd660fe P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/48746a3d20534795b50aae78bfd660fe
I20240503 08:12:41.655402 31269 ts_tablet_manager.cc:1909] T 48746a3d20534795b50aae78bfd660fe P 833582ff68c94269be364ebd187a75db: Deleting consensus metadata
I20240503 08:12:41.657884 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet 48746a3d20534795b50aae78bfd660fe (table rebalance_test_table_4 [id=acb2583756a845cab3b0cb05c246fe65]) successfully deleted
I20240503 08:12:41.658210 31140 tablet_service.cc:1467] Processing CreateTablet for tablet 6c2824aa878845e098f600f1fcfea087 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=7358500a74314a5a8795ddb89337f666]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:41.658639 31140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c2824aa878845e098f600f1fcfea087. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:41.663277 31420 tablet_bootstrap.cc:492] T d808d27e71a5486f9773ad4b239d1cfe P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:41.663865 31418 tablet_bootstrap.cc:492] T 6c2824aa878845e098f600f1fcfea087 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:41.664049 31418 ts_tablet_manager.cc:1367] T 6c2824aa878845e098f600f1fcfea087 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.006s
I20240503 08:12:41.664845 31418 raft_consensus.cc:348] T 6c2824aa878845e098f600f1fcfea087 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:41.665153 31418 raft_consensus.cc:374] T 6c2824aa878845e098f600f1fcfea087 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:41.665182 31420 tablet_bootstrap.cc:654] T d808d27e71a5486f9773ad4b239d1cfe P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:41.665313 31418 raft_consensus.cc:729] T 6c2824aa878845e098f600f1fcfea087 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:41.665629 31418 consensus_queue.cc:260] T 6c2824aa878845e098f600f1fcfea087 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:41.666894 30873 tablet_service.cc:1467] Processing CreateTablet for tablet f83a4778ffe640ee907e0a73e58abb73 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=7358500a74314a5a8795ddb89337f666]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:41.666970 31418 ts_tablet_manager.cc:1398] T 6c2824aa878845e098f600f1fcfea087 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:12:41.667276 30873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f83a4778ffe640ee907e0a73e58abb73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:41.673556 31420 tablet_bootstrap.cc:492] T d808d27e71a5486f9773ad4b239d1cfe P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:41.673681 31420 ts_tablet_manager.cc:1367] T d808d27e71a5486f9773ad4b239d1cfe P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240503 08:12:41.674547 31420 raft_consensus.cc:348] T d808d27e71a5486f9773ad4b239d1cfe P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:41.674714 31420 raft_consensus.cc:374] T d808d27e71a5486f9773ad4b239d1cfe P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:41.674818 31420 raft_consensus.cc:729] T d808d27e71a5486f9773ad4b239d1cfe P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:41.675062 31420 consensus_queue.cc:260] T d808d27e71a5486f9773ad4b239d1cfe P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:41.675663 31420 ts_tablet_manager.cc:1398] T d808d27e71a5486f9773ad4b239d1cfe P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:41.678079 31008 tablet_service.cc:1467] Processing CreateTablet for tablet f83a4778ffe640ee907e0a73e58abb73 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=7358500a74314a5a8795ddb89337f666]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:41.678103 30740 tablet_service.cc:1467] Processing CreateTablet for tablet f83a4778ffe640ee907e0a73e58abb73 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=7358500a74314a5a8795ddb89337f666]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:41.678493 30740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f83a4778ffe640ee907e0a73e58abb73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:41.678514 31008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f83a4778ffe640ee907e0a73e58abb73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:41.688810 27859 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:41.695988 31426 tablet_bootstrap.cc:492] T f83a4778ffe640ee907e0a73e58abb73 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:41.696861 31388 tablet.cc:1722] T d472290d0d11409ba94b055065126aa8 P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:41.697470 31389 tablet.cc:1722] T d472290d0d11409ba94b055065126aa8 P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:41.698081 31426 tablet_bootstrap.cc:654] T f83a4778ffe640ee907e0a73e58abb73 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:41.704818 31392 tablet.cc:1722] T d472290d0d11409ba94b055065126aa8 P b00f244c69df401997ea99d492218a98: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:41.709275 27854 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table_new_name [id=959e4a8f2be647fa9701dd5c969b3c7d] alter complete (version 3)
I20240503 08:12:41.715597 31374 raft_consensus.cc:482] T d808d27e71a5486f9773ad4b239d1cfe P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:41.715746 31374 raft_consensus.cc:504] T d808d27e71a5486f9773ad4b239d1cfe P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:41.716583 31374 leader_election.cc:288] T d808d27e71a5486f9773ad4b239d1cfe P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:12:41.720542 31418 tablet_bootstrap.cc:492] T d808d27e71a5486f9773ad4b239d1cfe P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:41.722191 31418 tablet_bootstrap.cc:654] T d808d27e71a5486f9773ad4b239d1cfe P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:41.727921 31409 ts_tablet_manager.cc:1888] T c6498d83236c48f88bd21c303c9c9ab3 P aae860d7ae2b4002bf7794c278f9cc56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:41.730255 31409 log.cc:1196] T c6498d83236c48f88bd21c303c9c9ab3 P aae860d7ae2b4002bf7794c278f9cc56: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal/wals/c6498d83236c48f88bd21c303c9c9ab3
I20240503 08:12:41.730831 31409 ts_tablet_manager.cc:1909] T c6498d83236c48f88bd21c303c9c9ab3 P aae860d7ae2b4002bf7794c278f9cc56: Deleting consensus metadata
I20240503 08:12:41.731848 31409 tablet_replica.cc:330] T 48746a3d20534795b50aae78bfd660fe P aae860d7ae2b4002bf7794c278f9cc56: stopping tablet replica
I20240503 08:12:41.731882 27847 catalog_manager.cc:4830] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): tablet c6498d83236c48f88bd21c303c9c9ab3 (table rebalance_test_table_4 [id=acb2583756a845cab3b0cb05c246fe65]) successfully deleted
I20240503 08:12:41.732138 31409 raft_consensus.cc:2229] T 48746a3d20534795b50aae78bfd660fe P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:41.732359 31409 raft_consensus.cc:2258] T 48746a3d20534795b50aae78bfd660fe P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:41.733084 31409 ts_tablet_manager.cc:1875] T 48746a3d20534795b50aae78bfd660fe P aae860d7ae2b4002bf7794c278f9cc56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:41.747630 31426 tablet_bootstrap.cc:492] T f83a4778ffe640ee907e0a73e58abb73 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:41.747859 31409 ts_tablet_manager.cc:1888] T 48746a3d20534795b50aae78bfd660fe P aae860d7ae2b4002bf7794c278f9cc56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:41.747992 31409 log.cc:1196] T 48746a3d20534795b50aae78bfd660fe P aae860d7ae2b4002bf7794c278f9cc56: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal/wals/48746a3d20534795b50aae78bfd660fe
I20240503 08:12:41.748227 31426 ts_tablet_manager.cc:1367] T f83a4778ffe640ee907e0a73e58abb73 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.054s user 0.014s sys 0.014s
I20240503 08:12:41.749290 31409 ts_tablet_manager.cc:1909] T 48746a3d20534795b50aae78bfd660fe P aae860d7ae2b4002bf7794c278f9cc56: Deleting consensus metadata
I20240503 08:12:41.749338 31426 raft_consensus.cc:348] T f83a4778ffe640ee907e0a73e58abb73 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:41.750351 27847 catalog_manager.cc:4830] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): tablet 48746a3d20534795b50aae78bfd660fe (table rebalance_test_table_4 [id=acb2583756a845cab3b0cb05c246fe65]) successfully deleted
I20240503 08:12:41.750777 31426 raft_consensus.cc:374] T f83a4778ffe640ee907e0a73e58abb73 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:41.751016 31426 raft_consensus.cc:729] T f83a4778ffe640ee907e0a73e58abb73 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:41.751030 31431 tablet_bootstrap.cc:492] T 6c2824aa878845e098f600f1fcfea087 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:41.751444 31426 consensus_queue.cc:260] T f83a4778ffe640ee907e0a73e58abb73 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:41.752357 31426 ts_tablet_manager.cc:1398] T f83a4778ffe640ee907e0a73e58abb73 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 08:12:41.752830 31431 tablet_bootstrap.cc:654] T 6c2824aa878845e098f600f1fcfea087 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:41.753072 31426 tablet_bootstrap.cc:492] T 6c2824aa878845e098f600f1fcfea087 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:41.754860 31426 tablet_bootstrap.cc:654] T 6c2824aa878845e098f600f1fcfea087 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:41.755923 31418 tablet_bootstrap.cc:492] T d808d27e71a5486f9773ad4b239d1cfe P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:41.756146 31418 ts_tablet_manager.cc:1367] T d808d27e71a5486f9773ad4b239d1cfe P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.036s user 0.006s sys 0.013s
I20240503 08:12:41.756924 31418 raft_consensus.cc:348] T d808d27e71a5486f9773ad4b239d1cfe P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:41.757091 31418 raft_consensus.cc:374] T d808d27e71a5486f9773ad4b239d1cfe P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:41.757166 31418 raft_consensus.cc:729] T d808d27e71a5486f9773ad4b239d1cfe P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:41.757405 31418 consensus_queue.cc:260] T d808d27e71a5486f9773ad4b239d1cfe P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:41.758574 31418 ts_tablet_manager.cc:1398] T d808d27e71a5486f9773ad4b239d1cfe P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:12:41.758893 31418 tablet_bootstrap.cc:492] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:41.761101 31418 tablet_bootstrap.cc:654] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:41.762890 31161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d808d27e71a5486f9773ad4b239d1cfe" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:12:41.764405 31418 tablet_bootstrap.cc:492] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:41.764521 31418 ts_tablet_manager.cc:1367] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20240503 08:12:41.765030 31418 raft_consensus.cc:348] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:41.765167 31418 raft_consensus.cc:374] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:41.765283 31418 raft_consensus.cc:729] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:41.765478 31418 consensus_queue.cc:260] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
W20240503 08:12:41.765930 28326 leader_election.cc:341] T d808d27e71a5486f9773ad4b239d1cfe P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:41.766479 31418 ts_tablet_manager.cc:1398] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:41.775110 30894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d808d27e71a5486f9773ad4b239d1cfe" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:12:41.775321 30894 raft_consensus.cc:2454] T d808d27e71a5486f9773ad4b239d1cfe P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:12:41.775895 28326 leader_election.cc:302] T d808d27e71a5486f9773ad4b239d1cfe P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c, d09a11a03989421b9f28019dc1c14cc2; no voters: b00f244c69df401997ea99d492218a98
I20240503 08:12:41.776264 31374 raft_consensus.cc:2781] T d808d27e71a5486f9773ad4b239d1cfe P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:41.776367 31374 raft_consensus.cc:482] T d808d27e71a5486f9773ad4b239d1cfe P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:41.776448 31374 raft_consensus.cc:3037] T d808d27e71a5486f9773ad4b239d1cfe P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:41.779194 31374 raft_consensus.cc:504] T d808d27e71a5486f9773ad4b239d1cfe P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:41.779836 31374 leader_election.cc:288] T d808d27e71a5486f9773ad4b239d1cfe P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Requested vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:12:41.782151 31161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d808d27e71a5486f9773ad4b239d1cfe" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:12:41.782529 30894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d808d27e71a5486f9773ad4b239d1cfe" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:12:41.782708 30894 raft_consensus.cc:3037] T d808d27e71a5486f9773ad4b239d1cfe P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:41.785099 31428 tablet_bootstrap.cc:492] T f83a4778ffe640ee907e0a73e58abb73 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:41.785377 30894 raft_consensus.cc:2454] T d808d27e71a5486f9773ad4b239d1cfe P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:12:41.785903 28326 leader_election.cc:302] T d808d27e71a5486f9773ad4b239d1cfe P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:41.786159 31374 raft_consensus.cc:2781] T d808d27e71a5486f9773ad4b239d1cfe P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:41.786365 31374 raft_consensus.cc:686] T d808d27e71a5486f9773ad4b239d1cfe P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Becoming Leader. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Running, Role: LEADER
I20240503 08:12:41.786624 31374 consensus_queue.cc:237] T d808d27e71a5486f9773ad4b239d1cfe P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
W20240503 08:12:41.787725 28326 leader_election.cc:341] T d808d27e71a5486f9773ad4b239d1cfe P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:41.792160 27862 catalog_manager.cc:5484] T d808d27e71a5486f9773ad4b239d1cfe P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68). New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:12:41.793640 31428 tablet_bootstrap.cc:654] T f83a4778ffe640ee907e0a73e58abb73 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:41.794922 31426 tablet_bootstrap.cc:492] T 6c2824aa878845e098f600f1fcfea087 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:41.795132 31426 ts_tablet_manager.cc:1367] T 6c2824aa878845e098f600f1fcfea087 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.042s user 0.021s sys 0.010s
I20240503 08:12:41.795903 31426 raft_consensus.cc:348] T 6c2824aa878845e098f600f1fcfea087 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:41.796173 31426 raft_consensus.cc:374] T 6c2824aa878845e098f600f1fcfea087 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:41.796283 31426 raft_consensus.cc:729] T 6c2824aa878845e098f600f1fcfea087 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:41.796559 31426 consensus_queue.cc:260] T 6c2824aa878845e098f600f1fcfea087 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:41.797096 31431 tablet_bootstrap.cc:492] T 6c2824aa878845e098f600f1fcfea087 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:41.797580 31431 ts_tablet_manager.cc:1367] T 6c2824aa878845e098f600f1fcfea087 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.047s user 0.004s sys 0.004s
I20240503 08:12:41.798743 31431 raft_consensus.cc:348] T 6c2824aa878845e098f600f1fcfea087 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:41.799103 31426 ts_tablet_manager.cc:1398] T 6c2824aa878845e098f600f1fcfea087 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 08:12:41.801873 31431 raft_consensus.cc:374] T 6c2824aa878845e098f600f1fcfea087 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:41.801985 31431 raft_consensus.cc:729] T 6c2824aa878845e098f600f1fcfea087 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:41.802212 31431 consensus_queue.cc:260] T 6c2824aa878845e098f600f1fcfea087 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:41.803735 31379 raft_consensus.cc:482] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:41.803881 31379 raft_consensus.cc:504] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:41.804172 31431 ts_tablet_manager.cc:1398] T 6c2824aa878845e098f600f1fcfea087 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 08:12:41.804422 31431 tablet_bootstrap.cc:492] T d808d27e71a5486f9773ad4b239d1cfe P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:41.804733 31379 leader_election.cc:288] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:12:41.805455 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a4778ffe640ee907e0a73e58abb73" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:12:41.805658 30762 raft_consensus.cc:2454] T f83a4778ffe640ee907e0a73e58abb73 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:12:41.805927 31431 tablet_bootstrap.cc:654] T d808d27e71a5486f9773ad4b239d1cfe P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:41.806154 30829 leader_election.cc:302] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:41.806469 31379 raft_consensus.cc:2781] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:41.806562 31379 raft_consensus.cc:482] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:41.806627 31379 raft_consensus.cc:3037] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:41.809193 31379 raft_consensus.cc:504] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:41.810832 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a4778ffe640ee907e0a73e58abb73" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:12:41.811049 30762 raft_consensus.cc:3037] T f83a4778ffe640ee907e0a73e58abb73 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:41.814002 30762 raft_consensus.cc:2454] T f83a4778ffe640ee907e0a73e58abb73 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:12:41.814496 30829 leader_election.cc:302] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:41.814792 31309 raft_consensus.cc:2781] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:41.814919 31309 raft_consensus.cc:686] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Becoming Leader. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Running, Role: LEADER
I20240503 08:12:41.815176 31309 consensus_queue.cc:237] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:41.816919 31379 leader_election.cc:288] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Requested vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:12:41.803544 31428 tablet_bootstrap.cc:492] T f83a4778ffe640ee907e0a73e58abb73 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:41.821923 31428 ts_tablet_manager.cc:1367] T f83a4778ffe640ee907e0a73e58abb73 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.002s
I20240503 08:12:41.823585 31428 raft_consensus.cc:348] T f83a4778ffe640ee907e0a73e58abb73 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:41.823835 31428 raft_consensus.cc:374] T f83a4778ffe640ee907e0a73e58abb73 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:41.823985 31428 raft_consensus.cc:729] T f83a4778ffe640ee907e0a73e58abb73 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:41.824172 27859 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:41.824409 27859 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:41.824379 31428 consensus_queue.cc:260] T f83a4778ffe640ee907e0a73e58abb73 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:41.827840 27862 catalog_manager.cc:5484] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c reported cstate change: term changed from 0 to 1, leader changed from <none> to ce1496a47d084365ad786cd09e472a2c (127.20.230.70). New cstate: current_term: 1 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:12:41.828333 31428 ts_tablet_manager.cc:1398] T f83a4778ffe640ee907e0a73e58abb73 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240503 08:12:41.828868 27859 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet d472290d0d11409ba94b055065126aa8
I20240503 08:12:41.830042 31140 tablet_service.cc:1514] Processing DeleteTablet for tablet d472290d0d11409ba94b055065126aa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:41 UTC) from {username='slave'} at 127.0.0.1:45814
I20240503 08:12:41.831393 30741 tablet_service.cc:1514] Processing DeleteTablet for tablet d472290d0d11409ba94b055065126aa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:41 UTC) from {username='slave'} at 127.0.0.1:46794
I20240503 08:12:41.832700 31008 tablet_service.cc:1514] Processing DeleteTablet for tablet d472290d0d11409ba94b055065126aa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:41 UTC) from {username='slave'} at 127.0.0.1:47020
I20240503 08:12:41.838528 31301 raft_consensus.cc:482] T f83a4778ffe640ee907e0a73e58abb73 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:41.838696 31301 raft_consensus.cc:504] T f83a4778ffe640ee907e0a73e58abb73 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:41.840512 31301 leader_election.cc:288] T f83a4778ffe640ee907e0a73e58abb73 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:12:41.842305 31431 tablet_bootstrap.cc:492] T d808d27e71a5486f9773ad4b239d1cfe P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:41.842540 31431 ts_tablet_manager.cc:1367] T d808d27e71a5486f9773ad4b239d1cfe P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.038s user 0.022s sys 0.013s
I20240503 08:12:41.843355 31431 raft_consensus.cc:348] T d808d27e71a5486f9773ad4b239d1cfe P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:41.843640 31431 raft_consensus.cc:374] T d808d27e71a5486f9773ad4b239d1cfe P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:41.843953 31431 raft_consensus.cc:729] T d808d27e71a5486f9773ad4b239d1cfe P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:41.844262 31431 consensus_queue.cc:260] T d808d27e71a5486f9773ad4b239d1cfe P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:41.845507 31431 ts_tablet_manager.cc:1398] T d808d27e71a5486f9773ad4b239d1cfe P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:12:41.846663 31028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a4778ffe640ee907e0a73e58abb73" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:41.846829 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a4778ffe640ee907e0a73e58abb73" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:12:41.846908 31028 raft_consensus.cc:2454] T f83a4778ffe640ee907e0a73e58abb73 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:12:41.847132 31450 tablet_replica.cc:330] T d472290d0d11409ba94b055065126aa8 P 6d1d7de19459435fbde1999e3dfd226b: stopping tablet replica
I20240503 08:12:41.847608 31450 raft_consensus.cc:2229] T d472290d0d11409ba94b055065126aa8 P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:41.849025 31450 raft_consensus.cc:2258] T d472290d0d11409ba94b055065126aa8 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:41.849922 31449 tablet_replica.cc:330] T d472290d0d11409ba94b055065126aa8 P b00f244c69df401997ea99d492218a98: stopping tablet replica
I20240503 08:12:41.850360 31449 raft_consensus.cc:2229] T d472290d0d11409ba94b055065126aa8 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:41.850669 31449 raft_consensus.cc:2258] T d472290d0d11409ba94b055065126aa8 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:41.851125 31450 ts_tablet_manager.cc:1875] T d472290d0d11409ba94b055065126aa8 P 6d1d7de19459435fbde1999e3dfd226b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:41.852070 31449 ts_tablet_manager.cc:1875] T d472290d0d11409ba94b055065126aa8 P b00f244c69df401997ea99d492218a98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:41.858700 31449 ts_tablet_manager.cc:1888] T d472290d0d11409ba94b055065126aa8 P b00f244c69df401997ea99d492218a98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:41.858902 31449 log.cc:1196] T d472290d0d11409ba94b055065126aa8 P b00f244c69df401997ea99d492218a98: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal/wals/d472290d0d11409ba94b055065126aa8
I20240503 08:12:41.859442 31449 ts_tablet_manager.cc:1909] T d472290d0d11409ba94b055065126aa8 P b00f244c69df401997ea99d492218a98: Deleting consensus metadata
I20240503 08:12:41.861039 27846 catalog_manager.cc:4830] TS b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): tablet d472290d0d11409ba94b055065126aa8 (table rebalancer_dynamic_table_new_name [id=959e4a8f2be647fa9701dd5c969b3c7d]) successfully deleted
I20240503 08:12:41.867436 31451 tablet_replica.cc:330] T d472290d0d11409ba94b055065126aa8 P ced21ca983b749d0b4d4f6a0e2426bf8: stopping tablet replica
I20240503 08:12:41.867794 31451 raft_consensus.cc:2229] T d472290d0d11409ba94b055065126aa8 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:41.868100 31451 raft_consensus.cc:2258] T d472290d0d11409ba94b055065126aa8 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:41.869050 31450 ts_tablet_manager.cc:1888] T d472290d0d11409ba94b055065126aa8 P 6d1d7de19459435fbde1999e3dfd226b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:41.869190 31450 log.cc:1196] T d472290d0d11409ba94b055065126aa8 P 6d1d7de19459435fbde1999e3dfd226b: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal/wals/d472290d0d11409ba94b055065126aa8
I20240503 08:12:41.869717 31450 ts_tablet_manager.cc:1909] T d472290d0d11409ba94b055065126aa8 P 6d1d7de19459435fbde1999e3dfd226b: Deleting consensus metadata
I20240503 08:12:41.871440 27848 catalog_manager.cc:4830] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): tablet d472290d0d11409ba94b055065126aa8 (table rebalancer_dynamic_table_new_name [id=959e4a8f2be647fa9701dd5c969b3c7d]) successfully deleted
I20240503 08:12:41.871726 31451 ts_tablet_manager.cc:1875] T d472290d0d11409ba94b055065126aa8 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:41.871855 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a4778ffe640ee907e0a73e58abb73" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:12:41.872123 30762 raft_consensus.cc:2379] T f83a4778ffe640ee907e0a73e58abb73 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ced21ca983b749d0b4d4f6a0e2426bf8 in current term 1: Already voted for candidate ce1496a47d084365ad786cd09e472a2c in this term.
I20240503 08:12:41.872933 30894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a4778ffe640ee907e0a73e58abb73" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:12:41.873627 30959 leader_election.cc:302] T f83a4778ffe640ee907e0a73e58abb73 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8; no voters: 6d1d7de19459435fbde1999e3dfd226b, ce1496a47d084365ad786cd09e472a2c
I20240503 08:12:41.874135 31301 raft_consensus.cc:3037] T f83a4778ffe640ee907e0a73e58abb73 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:41.877060 31301 raft_consensus.cc:2726] T f83a4778ffe640ee907e0a73e58abb73 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:12:41.878412 31451 ts_tablet_manager.cc:1888] T d472290d0d11409ba94b055065126aa8 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:41.878557 31451 log.cc:1196] T d472290d0d11409ba94b055065126aa8 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal/wals/d472290d0d11409ba94b055065126aa8
I20240503 08:12:41.879153 31451 ts_tablet_manager.cc:1909] T d472290d0d11409ba94b055065126aa8 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting consensus metadata
I20240503 08:12:41.879598 31368 raft_consensus.cc:482] T 6c2824aa878845e098f600f1fcfea087 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:41.879756 31368 raft_consensus.cc:504] T 6c2824aa878845e098f600f1fcfea087 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:41.880613 31368 leader_election.cc:288] T 6c2824aa878845e098f600f1fcfea087 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:41.880757 27847 catalog_manager.cc:4830] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): tablet d472290d0d11409ba94b055065126aa8 (table rebalancer_dynamic_table_new_name [id=959e4a8f2be647fa9701dd5c969b3c7d]) successfully deleted
I20240503 08:12:41.882388 31161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c2824aa878845e098f600f1fcfea087" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:12:41.882639 31161 raft_consensus.cc:2454] T 6c2824aa878845e098f600f1fcfea087 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:12:41.883877 30694 leader_election.cc:302] T 6c2824aa878845e098f600f1fcfea087 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, b00f244c69df401997ea99d492218a98; no voters:
I20240503 08:12:41.884148 31368 raft_consensus.cc:2781] T 6c2824aa878845e098f600f1fcfea087 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:41.884269 31368 raft_consensus.cc:482] T 6c2824aa878845e098f600f1fcfea087 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:41.884356 31368 raft_consensus.cc:3037] T 6c2824aa878845e098f600f1fcfea087 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:41.887219 31368 raft_consensus.cc:504] T 6c2824aa878845e098f600f1fcfea087 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:41.888636 31308 raft_consensus.cc:482] T d808d27e71a5486f9773ad4b239d1cfe P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:41.888659 30894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c2824aa878845e098f600f1fcfea087" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:12:41.888765 31308 raft_consensus.cc:504] T d808d27e71a5486f9773ad4b239d1cfe P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:41.888864 30894 raft_consensus.cc:2454] T 6c2824aa878845e098f600f1fcfea087 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:12:41.889564 31308 leader_election.cc:288] T d808d27e71a5486f9773ad4b239d1cfe P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:12:41.890236 31161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c2824aa878845e098f600f1fcfea087" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:12:41.890440 31161 raft_consensus.cc:3037] T 6c2824aa878845e098f600f1fcfea087 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:41.891217 30894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c2824aa878845e098f600f1fcfea087" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:12:41.891388 30894 raft_consensus.cc:3037] T 6c2824aa878845e098f600f1fcfea087 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:41.893352 31161 raft_consensus.cc:2454] T 6c2824aa878845e098f600f1fcfea087 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:12:41.894065 30694 leader_election.cc:302] T 6c2824aa878845e098f600f1fcfea087 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, b00f244c69df401997ea99d492218a98; no voters:
I20240503 08:12:41.894716 31287 raft_consensus.cc:2781] T 6c2824aa878845e098f600f1fcfea087 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:41.896809 30894 raft_consensus.cc:2454] T 6c2824aa878845e098f600f1fcfea087 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
W20240503 08:12:41.898375 28439 tablet.cc:2369] T d808d27e71a5486f9773ad4b239d1cfe P d09a11a03989421b9f28019dc1c14cc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:12:41.905526 31368 leader_election.cc:288] T 6c2824aa878845e098f600f1fcfea087 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Requested vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:41.925880 31287 raft_consensus.cc:686] T 6c2824aa878845e098f600f1fcfea087 P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Becoming Leader. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Running, Role: LEADER
I20240503 08:12:41.926263 31287 consensus_queue.cc:237] T 6c2824aa878845e098f600f1fcfea087 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:41.928982 28392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d808d27e71a5486f9773ad4b239d1cfe" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:41.931041 30894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d808d27e71a5486f9773ad4b239d1cfe" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:12:41.931330 30894 raft_consensus.cc:2379] T d808d27e71a5486f9773ad4b239d1cfe P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b00f244c69df401997ea99d492218a98 in current term 1: Already voted for candidate d09a11a03989421b9f28019dc1c14cc2 in this term.
I20240503 08:12:41.931921 31094 leader_election.cc:302] T d808d27e71a5486f9773ad4b239d1cfe P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b00f244c69df401997ea99d492218a98; no voters: ce1496a47d084365ad786cd09e472a2c, d09a11a03989421b9f28019dc1c14cc2
I20240503 08:12:41.932199 31308 raft_consensus.cc:3037] T d808d27e71a5486f9773ad4b239d1cfe P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:41.931989 27859 catalog_manager.cc:5484] T 6c2824aa878845e098f600f1fcfea087 P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69). New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:12:41.934557 31308 raft_consensus.cc:2726] T d808d27e71a5486f9773ad4b239d1cfe P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:12:41.943325 27859 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:41.950462 31008 tablet_service.cc:1467] Processing CreateTablet for tablet 9fff870834d04fd59f77f14c3db9f3a1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=88d4f1885e1240c4bd1ea1cf956cbee2]), partition=
I20240503 08:12:41.950887 31008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fff870834d04fd59f77f14c3db9f3a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:41.954653 31140 tablet_service.cc:1467] Processing CreateTablet for tablet 9fff870834d04fd59f77f14c3db9f3a1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=88d4f1885e1240c4bd1ea1cf956cbee2]), partition=
I20240503 08:12:41.955014 31140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fff870834d04fd59f77f14c3db9f3a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:41.955471 28242 tablet_service.cc:1467] Processing CreateTablet for tablet 9fff870834d04fd59f77f14c3db9f3a1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=88d4f1885e1240c4bd1ea1cf956cbee2]), partition=
I20240503 08:12:41.955802 28242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fff870834d04fd59f77f14c3db9f3a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:41.958637 31428 tablet_bootstrap.cc:492] T 9fff870834d04fd59f77f14c3db9f3a1 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:41.959818 31428 tablet_bootstrap.cc:654] T 9fff870834d04fd59f77f14c3db9f3a1 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:41.960608 31465 tablet_bootstrap.cc:492] T 9fff870834d04fd59f77f14c3db9f3a1 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:41.962116 31465 tablet_bootstrap.cc:654] T 9fff870834d04fd59f77f14c3db9f3a1 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:41.963909 31431 tablet_bootstrap.cc:492] T 9fff870834d04fd59f77f14c3db9f3a1 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:41.963948 31465 tablet_bootstrap.cc:492] T 9fff870834d04fd59f77f14c3db9f3a1 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:41.964071 31465 ts_tablet_manager.cc:1367] T 9fff870834d04fd59f77f14c3db9f3a1 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:12:41.964792 31465 raft_consensus.cc:348] T 9fff870834d04fd59f77f14c3db9f3a1 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:41.965005 31465 raft_consensus.cc:374] T 9fff870834d04fd59f77f14c3db9f3a1 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:41.965080 31465 raft_consensus.cc:729] T 9fff870834d04fd59f77f14c3db9f3a1 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:41.965232 31431 tablet_bootstrap.cc:654] T 9fff870834d04fd59f77f14c3db9f3a1 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:41.965284 31465 consensus_queue.cc:260] T 9fff870834d04fd59f77f14c3db9f3a1 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:41.965863 31465 ts_tablet_manager.cc:1398] T 9fff870834d04fd59f77f14c3db9f3a1 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:41.971503 31428 tablet_bootstrap.cc:492] T 9fff870834d04fd59f77f14c3db9f3a1 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:41.971724 31428 ts_tablet_manager.cc:1367] T 9fff870834d04fd59f77f14c3db9f3a1 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.006s
I20240503 08:12:41.972481 31428 raft_consensus.cc:348] T 9fff870834d04fd59f77f14c3db9f3a1 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:41.972705 31428 raft_consensus.cc:374] T 9fff870834d04fd59f77f14c3db9f3a1 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:41.972771 31428 raft_consensus.cc:729] T 9fff870834d04fd59f77f14c3db9f3a1 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:41.972985 31428 consensus_queue.cc:260] T 9fff870834d04fd59f77f14c3db9f3a1 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:41.973491 31428 ts_tablet_manager.cc:1398] T 9fff870834d04fd59f77f14c3db9f3a1 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:12:41.977695 31431 tablet_bootstrap.cc:492] T 9fff870834d04fd59f77f14c3db9f3a1 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:41.977849 31431 ts_tablet_manager.cc:1367] T 9fff870834d04fd59f77f14c3db9f3a1 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.001s
I20240503 08:12:41.978516 31431 raft_consensus.cc:348] T 9fff870834d04fd59f77f14c3db9f3a1 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:41.978698 31431 raft_consensus.cc:374] T 9fff870834d04fd59f77f14c3db9f3a1 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:41.978776 31431 raft_consensus.cc:729] T 9fff870834d04fd59f77f14c3db9f3a1 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:41.978993 31431 consensus_queue.cc:260] T 9fff870834d04fd59f77f14c3db9f3a1 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:41.981674 31431 ts_tablet_manager.cc:1398] T 9fff870834d04fd59f77f14c3db9f3a1 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
W20240503 08:12:41.993237 28309 tablet.cc:2369] T 9fff870834d04fd59f77f14c3db9f3a1 P aae860d7ae2b4002bf7794c278f9cc56: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:12:42.075282 27862 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240503 08:12:42.075474 27862 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240503 08:12:42.078644 27862 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet 6f19592b330546b2b4935646ff7b0304
I20240503 08:12:42.079141 27862 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet c19f22d2bc0d47e3a8217b2c97760348
I20240503 08:12:42.079385 28112 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f19592b330546b2b4935646ff7b0304 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:42 UTC) from {username='slave'} at 127.0.0.1:57912
I20240503 08:12:42.079546 27982 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f19592b330546b2b4935646ff7b0304 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:42 UTC) from {username='slave'} at 127.0.0.1:55610
I20240503 08:12:42.079599 27862 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet 1856d5641d3f48698f5401a1d157369d
I20240503 08:12:42.079669 31267 tablet_replica.cc:330] T 6f19592b330546b2b4935646ff7b0304 P 1b411c6bb25549368041773d7b710d48: stopping tablet replica
I20240503 08:12:42.079860 31269 tablet_replica.cc:330] T 6f19592b330546b2b4935646ff7b0304 P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:12:42.079903 31267 raft_consensus.cc:2229] T 6f19592b330546b2b4935646ff7b0304 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:42.080065 31269 raft_consensus.cc:2229] T 6f19592b330546b2b4935646ff7b0304 P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:42.080154 31267 raft_consensus.cc:2258] T 6f19592b330546b2b4935646ff7b0304 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:42.080432 31269 raft_consensus.cc:2258] T 6f19592b330546b2b4935646ff7b0304 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:42.080703 28373 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f19592b330546b2b4935646ff7b0304 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:42 UTC) from {username='slave'} at 127.0.0.1:34786
I20240503 08:12:42.080935 31267 ts_tablet_manager.cc:1875] T 6f19592b330546b2b4935646ff7b0304 P 1b411c6bb25549368041773d7b710d48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:42.080991 28112 tablet_service.cc:1514] Processing DeleteTablet for tablet 1856d5641d3f48698f5401a1d157369d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:42 UTC) from {username='slave'} at 127.0.0.1:57912
I20240503 08:12:42.081080 27981 tablet_service.cc:1514] Processing DeleteTablet for tablet 1856d5641d3f48698f5401a1d157369d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:42 UTC) from {username='slave'} at 127.0.0.1:55610
I20240503 08:12:42.081151 31269 ts_tablet_manager.cc:1875] T 6f19592b330546b2b4935646ff7b0304 P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:42.081331 27982 tablet_service.cc:1514] Processing DeleteTablet for tablet c19f22d2bc0d47e3a8217b2c97760348 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:42 UTC) from {username='slave'} at 127.0.0.1:55610
I20240503 08:12:42.081753 31268 tablet_replica.cc:330] T 6f19592b330546b2b4935646ff7b0304 P d09a11a03989421b9f28019dc1c14cc2: stopping tablet replica
I20240503 08:12:42.082029 31268 raft_consensus.cc:2229] T 6f19592b330546b2b4935646ff7b0304 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:42.082072 28372 tablet_service.cc:1514] Processing DeleteTablet for tablet 1856d5641d3f48698f5401a1d157369d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:42 UTC) from {username='slave'} at 127.0.0.1:34786
I20240503 08:12:42.082264 31268 raft_consensus.cc:2258] T 6f19592b330546b2b4935646ff7b0304 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:42.082597 28373 tablet_service.cc:1514] Processing DeleteTablet for tablet c19f22d2bc0d47e3a8217b2c97760348 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:42 UTC) from {username='slave'} at 127.0.0.1:34786
I20240503 08:12:42.082943 31268 ts_tablet_manager.cc:1875] T 6f19592b330546b2b4935646ff7b0304 P d09a11a03989421b9f28019dc1c14cc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:42.082948 28242 tablet_service.cc:1514] Processing DeleteTablet for tablet c19f22d2bc0d47e3a8217b2c97760348 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:42 UTC) from {username='slave'} at 127.0.0.1:41960
I20240503 08:12:42.083216 31409 tablet_replica.cc:330] T c19f22d2bc0d47e3a8217b2c97760348 P aae860d7ae2b4002bf7794c278f9cc56: stopping tablet replica
I20240503 08:12:42.083406 31409 raft_consensus.cc:2229] T c19f22d2bc0d47e3a8217b2c97760348 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:42.083639 31409 raft_consensus.cc:2258] T c19f22d2bc0d47e3a8217b2c97760348 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:42.084322 31409 ts_tablet_manager.cc:1875] T c19f22d2bc0d47e3a8217b2c97760348 P aae860d7ae2b4002bf7794c278f9cc56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:42.088954 31267 ts_tablet_manager.cc:1888] T 6f19592b330546b2b4935646ff7b0304 P 1b411c6bb25549368041773d7b710d48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:42.089210 31267 log.cc:1196] T 6f19592b330546b2b4935646ff7b0304 P 1b411c6bb25549368041773d7b710d48: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/6f19592b330546b2b4935646ff7b0304
I20240503 08:12:42.089792 31267 ts_tablet_manager.cc:1909] T 6f19592b330546b2b4935646ff7b0304 P 1b411c6bb25549368041773d7b710d48: Deleting consensus metadata
I20240503 08:12:42.090891 31268 ts_tablet_manager.cc:1888] T 6f19592b330546b2b4935646ff7b0304 P d09a11a03989421b9f28019dc1c14cc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:42.091091 31268 log.cc:1196] T 6f19592b330546b2b4935646ff7b0304 P d09a11a03989421b9f28019dc1c14cc2: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/wals/6f19592b330546b2b4935646ff7b0304
I20240503 08:12:42.091058 27848 catalog_manager.cc:4830] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): tablet 6f19592b330546b2b4935646ff7b0304 (table rebalance_test_table_6 [id=5cde77bbf6c84845aaa35fc722360975]) successfully deleted
I20240503 08:12:42.091461 31267 tablet_replica.cc:330] T 1856d5641d3f48698f5401a1d157369d P 1b411c6bb25549368041773d7b710d48: stopping tablet replica
I20240503 08:12:42.091589 31268 ts_tablet_manager.cc:1909] T 6f19592b330546b2b4935646ff7b0304 P d09a11a03989421b9f28019dc1c14cc2: Deleting consensus metadata
I20240503 08:12:42.091717 31267 raft_consensus.cc:2229] T 1856d5641d3f48698f5401a1d157369d P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:42.092011 31267 raft_consensus.cc:2258] T 1856d5641d3f48698f5401a1d157369d P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:42.092649 27849 catalog_manager.cc:4830] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): tablet 6f19592b330546b2b4935646ff7b0304 (table rebalance_test_table_6 [id=5cde77bbf6c84845aaa35fc722360975]) successfully deleted
I20240503 08:12:42.092876 31267 ts_tablet_manager.cc:1875] T 1856d5641d3f48698f5401a1d157369d P 1b411c6bb25549368041773d7b710d48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:42.092911 31268 tablet_replica.cc:330] T 1856d5641d3f48698f5401a1d157369d P d09a11a03989421b9f28019dc1c14cc2: stopping tablet replica
I20240503 08:12:42.093096 31268 raft_consensus.cc:2229] T 1856d5641d3f48698f5401a1d157369d P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:42.093345 31268 raft_consensus.cc:2258] T 1856d5641d3f48698f5401a1d157369d P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:42.094743 31268 ts_tablet_manager.cc:1875] T 1856d5641d3f48698f5401a1d157369d P d09a11a03989421b9f28019dc1c14cc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:42.097268 31409 ts_tablet_manager.cc:1888] T c19f22d2bc0d47e3a8217b2c97760348 P aae860d7ae2b4002bf7794c278f9cc56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:42.097473 31409 log.cc:1196] T c19f22d2bc0d47e3a8217b2c97760348 P aae860d7ae2b4002bf7794c278f9cc56: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal/wals/c19f22d2bc0d47e3a8217b2c97760348
I20240503 08:12:42.097976 31409 ts_tablet_manager.cc:1909] T c19f22d2bc0d47e3a8217b2c97760348 P aae860d7ae2b4002bf7794c278f9cc56: Deleting consensus metadata
I20240503 08:12:42.098043 31269 ts_tablet_manager.cc:1888] T 6f19592b330546b2b4935646ff7b0304 P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:42.098196 31269 log.cc:1196] T 6f19592b330546b2b4935646ff7b0304 P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/6f19592b330546b2b4935646ff7b0304
I20240503 08:12:42.098717 31269 ts_tablet_manager.cc:1909] T 6f19592b330546b2b4935646ff7b0304 P 833582ff68c94269be364ebd187a75db: Deleting consensus metadata
I20240503 08:12:42.099258 31267 ts_tablet_manager.cc:1888] T 1856d5641d3f48698f5401a1d157369d P 1b411c6bb25549368041773d7b710d48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:42.099825 31269 tablet_replica.cc:330] T 1856d5641d3f48698f5401a1d157369d P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:12:42.099931 31267 log.cc:1196] T 1856d5641d3f48698f5401a1d157369d P 1b411c6bb25549368041773d7b710d48: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/1856d5641d3f48698f5401a1d157369d
I20240503 08:12:42.099977 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet 6f19592b330546b2b4935646ff7b0304 (table rebalance_test_table_6 [id=5cde77bbf6c84845aaa35fc722360975]) successfully deleted
I20240503 08:12:42.100092 31269 raft_consensus.cc:2229] T 1856d5641d3f48698f5401a1d157369d P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:42.100452 31269 raft_consensus.cc:2258] T 1856d5641d3f48698f5401a1d157369d P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:42.100787 27847 catalog_manager.cc:4830] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): tablet c19f22d2bc0d47e3a8217b2c97760348 (table rebalance_test_table_6 [id=5cde77bbf6c84845aaa35fc722360975]) successfully deleted
I20240503 08:12:42.101032 31267 ts_tablet_manager.cc:1909] T 1856d5641d3f48698f5401a1d157369d P 1b411c6bb25549368041773d7b710d48: Deleting consensus metadata
I20240503 08:12:42.101280 31269 ts_tablet_manager.cc:1875] T 1856d5641d3f48698f5401a1d157369d P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:42.101737 31268 ts_tablet_manager.cc:1888] T 1856d5641d3f48698f5401a1d157369d P d09a11a03989421b9f28019dc1c14cc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:42.101941 31268 log.cc:1196] T 1856d5641d3f48698f5401a1d157369d P d09a11a03989421b9f28019dc1c14cc2: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/wals/1856d5641d3f48698f5401a1d157369d
I20240503 08:12:42.102202 27848 catalog_manager.cc:4830] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): tablet 1856d5641d3f48698f5401a1d157369d (table rebalance_test_table_6 [id=5cde77bbf6c84845aaa35fc722360975]) successfully deleted
I20240503 08:12:42.102442 31268 ts_tablet_manager.cc:1909] T 1856d5641d3f48698f5401a1d157369d P d09a11a03989421b9f28019dc1c14cc2: Deleting consensus metadata
I20240503 08:12:42.103319 31268 tablet_replica.cc:330] T c19f22d2bc0d47e3a8217b2c97760348 P d09a11a03989421b9f28019dc1c14cc2: stopping tablet replica
I20240503 08:12:42.103591 31268 raft_consensus.cc:2229] T c19f22d2bc0d47e3a8217b2c97760348 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:42.103667 27849 catalog_manager.cc:4830] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): tablet 1856d5641d3f48698f5401a1d157369d (table rebalance_test_table_6 [id=5cde77bbf6c84845aaa35fc722360975]) successfully deleted
I20240503 08:12:42.103891 31268 raft_consensus.cc:2258] T c19f22d2bc0d47e3a8217b2c97760348 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:42.104652 31268 ts_tablet_manager.cc:1875] T c19f22d2bc0d47e3a8217b2c97760348 P d09a11a03989421b9f28019dc1c14cc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:42.107259 31269 ts_tablet_manager.cc:1888] T 1856d5641d3f48698f5401a1d157369d P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:42.107409 31269 log.cc:1196] T 1856d5641d3f48698f5401a1d157369d P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/1856d5641d3f48698f5401a1d157369d
I20240503 08:12:42.107715 31269 ts_tablet_manager.cc:1909] T 1856d5641d3f48698f5401a1d157369d P 833582ff68c94269be364ebd187a75db: Deleting consensus metadata
I20240503 08:12:42.109148 31269 tablet_replica.cc:330] T c19f22d2bc0d47e3a8217b2c97760348 P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:12:42.109273 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet 1856d5641d3f48698f5401a1d157369d (table rebalance_test_table_6 [id=5cde77bbf6c84845aaa35fc722360975]) successfully deleted
I20240503 08:12:42.109340 31269 raft_consensus.cc:2229] T c19f22d2bc0d47e3a8217b2c97760348 P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:42.109351 31268 ts_tablet_manager.cc:1888] T c19f22d2bc0d47e3a8217b2c97760348 P d09a11a03989421b9f28019dc1c14cc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:42.109472 31268 log.cc:1196] T c19f22d2bc0d47e3a8217b2c97760348 P d09a11a03989421b9f28019dc1c14cc2: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/wals/c19f22d2bc0d47e3a8217b2c97760348
I20240503 08:12:42.109648 31269 raft_consensus.cc:2258] T c19f22d2bc0d47e3a8217b2c97760348 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:42.109864 31268 ts_tablet_manager.cc:1909] T c19f22d2bc0d47e3a8217b2c97760348 P d09a11a03989421b9f28019dc1c14cc2: Deleting consensus metadata
I20240503 08:12:42.110256 31269 ts_tablet_manager.cc:1875] T c19f22d2bc0d47e3a8217b2c97760348 P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:42.110739 27849 catalog_manager.cc:4830] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): tablet c19f22d2bc0d47e3a8217b2c97760348 (table rebalance_test_table_6 [id=5cde77bbf6c84845aaa35fc722360975]) successfully deleted
I20240503 08:12:42.114866 31269 ts_tablet_manager.cc:1888] T c19f22d2bc0d47e3a8217b2c97760348 P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:42.115000 31269 log.cc:1196] T c19f22d2bc0d47e3a8217b2c97760348 P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/c19f22d2bc0d47e3a8217b2c97760348
I20240503 08:12:42.115392 31269 ts_tablet_manager.cc:1909] T c19f22d2bc0d47e3a8217b2c97760348 P 833582ff68c94269be364ebd187a75db: Deleting consensus metadata
I20240503 08:12:42.116173 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet c19f22d2bc0d47e3a8217b2c97760348 (table rebalance_test_table_6 [id=5cde77bbf6c84845aaa35fc722360975]) successfully deleted
I20240503 08:12:42.202649 31283 raft_consensus.cc:482] T 9fff870834d04fd59f77f14c3db9f3a1 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:42.202809 31283 raft_consensus.cc:504] T 9fff870834d04fd59f77f14c3db9f3a1 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:42.203719 31283 leader_election.cc:288] T 9fff870834d04fd59f77f14c3db9f3a1 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:42.207909 31309 consensus_queue.cc:1035] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:42.212177 31028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fff870834d04fd59f77f14c3db9f3a1" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:42.212420 31028 raft_consensus.cc:2454] T 9fff870834d04fd59f77f14c3db9f3a1 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:12:42.212869 28197 leader_election.cc:302] T 9fff870834d04fd59f77f14c3db9f3a1 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:42.213176 30652 raft_consensus.cc:2781] T 9fff870834d04fd59f77f14c3db9f3a1 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:42.213304 30652 raft_consensus.cc:482] T 9fff870834d04fd59f77f14c3db9f3a1 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:42.213433 30652 raft_consensus.cc:3037] T 9fff870834d04fd59f77f14c3db9f3a1 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:42.216117 30652 raft_consensus.cc:504] T 9fff870834d04fd59f77f14c3db9f3a1 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:42.216657 31161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fff870834d04fd59f77f14c3db9f3a1" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:12:42.216889 31161 raft_consensus.cc:2454] T 9fff870834d04fd59f77f14c3db9f3a1 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:12:42.217274 30652 leader_election.cc:288] T 9fff870834d04fd59f77f14c3db9f3a1 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Requested vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:42.217281 31028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fff870834d04fd59f77f14c3db9f3a1" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:12:42.217536 31028 raft_consensus.cc:3037] T 9fff870834d04fd59f77f14c3db9f3a1 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:42.217842 31161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fff870834d04fd59f77f14c3db9f3a1" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:12:42.218006 31161 raft_consensus.cc:3037] T 9fff870834d04fd59f77f14c3db9f3a1 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:42.220216 31028 raft_consensus.cc:2454] T 9fff870834d04fd59f77f14c3db9f3a1 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:42.220469 31161 raft_consensus.cc:2454] T 9fff870834d04fd59f77f14c3db9f3a1 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:42.221056 28197 leader_election.cc:302] T 9fff870834d04fd59f77f14c3db9f3a1 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:42.221391 30652 raft_consensus.cc:2781] T 9fff870834d04fd59f77f14c3db9f3a1 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:42.221532 30652 raft_consensus.cc:686] T 9fff870834d04fd59f77f14c3db9f3a1 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Becoming Leader. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Running, Role: LEADER
I20240503 08:12:42.221789 30652 consensus_queue.cc:237] T 9fff870834d04fd59f77f14c3db9f3a1 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:42.236332 31379 consensus_queue.cc:1035] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:42.237344 27862 catalog_manager.cc:5484] T 9fff870834d04fd59f77f14c3db9f3a1 P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: term changed from 0 to 1, leader changed from <none> to aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67). New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:12:42.280064 31374 consensus_queue.cc:1035] T d808d27e71a5486f9773ad4b239d1cfe P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:42.281819 31374 consensus_queue.cc:1035] T d808d27e71a5486f9773ad4b239d1cfe P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:42.338757 31287 consensus_queue.cc:1035] T 6c2824aa878845e098f600f1fcfea087 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:42.342603 31287 consensus_queue.cc:1035] T 6c2824aa878845e098f600f1fcfea087 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:42.360321 27862 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:42.365866 31161 raft_consensus.cc:1261] T 9fff870834d04fd59f77f14c3db9f3a1 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:42.365917 31028 raft_consensus.cc:1261] T 9fff870834d04fd59f77f14c3db9f3a1 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:42.366492 30652 consensus_queue.cc:1035] T 9fff870834d04fd59f77f14c3db9f3a1 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:42.366745 31283 consensus_queue.cc:1035] T 9fff870834d04fd59f77f14c3db9f3a1 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:42.370193 27859 catalog_manager.cc:5576] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915) does not have the latest schema for tablet 9fff870834d04fd59f77f14c3db9f3a1 (table rebalancer_dynamic_table [id=88d4f1885e1240c4bd1ea1cf956cbee2]). Expected version 1 got 0
I20240503 08:12:42.372426 31487 tablet.cc:1722] T 9fff870834d04fd59f77f14c3db9f3a1 P aae860d7ae2b4002bf7794c278f9cc56: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:42.373064 31488 tablet.cc:1722] T 9fff870834d04fd59f77f14c3db9f3a1 P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:42.373544 31488 tablet.cc:1620] T 9fff870834d04fd59f77f14c3db9f3a1 P ced21ca983b749d0b4d4f6a0e2426bf8: MemRowSet was empty: no flush needed.
I20240503 08:12:42.374956 31487 tablet.cc:1620] T 9fff870834d04fd59f77f14c3db9f3a1 P aae860d7ae2b4002bf7794c278f9cc56: MemRowSet was empty: no flush needed.
I20240503 08:12:42.377368 27859 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=88d4f1885e1240c4bd1ea1cf956cbee2] alter complete (version 1)
I20240503 08:12:42.381912 27862 catalog_manager.cc:5576] TS b00f244c69df401997ea99d492218a98 (127.20.230.72:40893) does not have the latest schema for tablet 9fff870834d04fd59f77f14c3db9f3a1 (table rebalancer_dynamic_table [id=88d4f1885e1240c4bd1ea1cf956cbee2]). Expected version 1 got 0
I20240503 08:12:42.383220 31489 tablet.cc:1722] T 9fff870834d04fd59f77f14c3db9f3a1 P b00f244c69df401997ea99d492218a98: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:42.383827 31489 tablet.cc:1620] T 9fff870834d04fd59f77f14c3db9f3a1 P b00f244c69df401997ea99d492218a98: MemRowSet was empty: no flush needed.
I20240503 08:12:42.384402 27862 catalog_manager.cc:5576] TS b00f244c69df401997ea99d492218a98 (127.20.230.72:40893) does not have the latest schema for tablet 9fff870834d04fd59f77f14c3db9f3a1 (table rebalancer_dynamic_table [id=88d4f1885e1240c4bd1ea1cf956cbee2]). Expected version 1 got 0
I20240503 08:12:42.484731 27862 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:42.485321 27856 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_extra_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:42.494294 31487 tablet.cc:1722] T 9fff870834d04fd59f77f14c3db9f3a1 P aae860d7ae2b4002bf7794c278f9cc56: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:42.494794 31487 tablet.cc:1620] T 9fff870834d04fd59f77f14c3db9f3a1 P aae860d7ae2b4002bf7794c278f9cc56: MemRowSet was empty: no flush needed.
I20240503 08:12:42.495275 30873 tablet_service.cc:1467] Processing CreateTablet for tablet b016e31e76854807af0cfaed6ec49071 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=1c7e1d7fc1be4aa4883c657bf13f0b9f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:42.495705 30873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b016e31e76854807af0cfaed6ec49071. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:42.495657 30741 tablet_service.cc:1467] Processing CreateTablet for tablet b016e31e76854807af0cfaed6ec49071 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=1c7e1d7fc1be4aa4883c657bf13f0b9f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:42.496136 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b016e31e76854807af0cfaed6ec49071. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:42.501703 31491 tablet_bootstrap.cc:492] T b016e31e76854807af0cfaed6ec49071 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:42.502202 27982 tablet_service.cc:1467] Processing CreateTablet for tablet b016e31e76854807af0cfaed6ec49071 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=1c7e1d7fc1be4aa4883c657bf13f0b9f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:42.502686 27982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b016e31e76854807af0cfaed6ec49071. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:42.504896 31491 tablet_bootstrap.cc:654] T b016e31e76854807af0cfaed6ec49071 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:42.511206 31488 tablet.cc:1722] T 9fff870834d04fd59f77f14c3db9f3a1 P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:42.511696 31488 tablet.cc:1620] T 9fff870834d04fd59f77f14c3db9f3a1 P ced21ca983b749d0b4d4f6a0e2426bf8: MemRowSet was empty: no flush needed.
I20240503 08:12:42.512564 31008 tablet_service.cc:1467] Processing CreateTablet for tablet d24e437f369b4df49e6363f3f492dc46 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=1c7e1d7fc1be4aa4883c657bf13f0b9f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:42.512960 31008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d24e437f369b4df49e6363f3f492dc46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:42.513487 28373 tablet_service.cc:1467] Processing CreateTablet for tablet d24e437f369b4df49e6363f3f492dc46 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=1c7e1d7fc1be4aa4883c657bf13f0b9f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:42.513938 28373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d24e437f369b4df49e6363f3f492dc46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:42.514070 30873 tablet_service.cc:1467] Processing CreateTablet for tablet 8c714f29abb140379b57e08eb927e2b7 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=1c7e1d7fc1be4aa4883c657bf13f0b9f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:42.514421 30873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c714f29abb140379b57e08eb927e2b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:42.518622 31140 tablet_service.cc:1467] Processing CreateTablet for tablet d24e437f369b4df49e6363f3f492dc46 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=1c7e1d7fc1be4aa4883c657bf13f0b9f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:42.519017 31140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d24e437f369b4df49e6363f3f492dc46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:42.521883 31141 tablet_service.cc:1467] Processing CreateTablet for tablet 8c714f29abb140379b57e08eb927e2b7 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=1c7e1d7fc1be4aa4883c657bf13f0b9f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:42.522245 31141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c714f29abb140379b57e08eb927e2b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:42.528802 28242 tablet_service.cc:1467] Processing CreateTablet for tablet 8c714f29abb140379b57e08eb927e2b7 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=1c7e1d7fc1be4aa4883c657bf13f0b9f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:42.529174 28242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c714f29abb140379b57e08eb927e2b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:42.536787 27862 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=88d4f1885e1240c4bd1ea1cf956cbee2] alter complete (version 2)
I20240503 08:12:42.541391 31489 tablet.cc:1722] T 9fff870834d04fd59f77f14c3db9f3a1 P b00f244c69df401997ea99d492218a98: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:42.541944 31489 tablet.cc:1620] T 9fff870834d04fd59f77f14c3db9f3a1 P b00f244c69df401997ea99d492218a98: MemRowSet was empty: no flush needed.
I20240503 08:12:42.545140 31495 tablet_bootstrap.cc:492] T b016e31e76854807af0cfaed6ec49071 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:42.547662 31494 tablet_bootstrap.cc:492] T b016e31e76854807af0cfaed6ec49071 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:42.549592 31495 tablet_bootstrap.cc:654] T b016e31e76854807af0cfaed6ec49071 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:42.549984 31494 tablet_bootstrap.cc:654] T b016e31e76854807af0cfaed6ec49071 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:42.550993 31496 tablet_bootstrap.cc:492] T d24e437f369b4df49e6363f3f492dc46 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:42.551430 31497 tablet_bootstrap.cc:492] T d24e437f369b4df49e6363f3f492dc46 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:42.552994 31497 tablet_bootstrap.cc:654] T d24e437f369b4df49e6363f3f492dc46 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:42.553969 31494 tablet_bootstrap.cc:492] T b016e31e76854807af0cfaed6ec49071 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:42.554158 31494 ts_tablet_manager.cc:1367] T b016e31e76854807af0cfaed6ec49071 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240503 08:12:42.554172 31498 tablet_bootstrap.cc:492] T 8c714f29abb140379b57e08eb927e2b7 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:42.554852 31496 tablet_bootstrap.cc:654] T d24e437f369b4df49e6363f3f492dc46 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:42.555058 31494 raft_consensus.cc:348] T b016e31e76854807af0cfaed6ec49071 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:42.555357 31494 raft_consensus.cc:374] T b016e31e76854807af0cfaed6ec49071 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:42.555477 31494 raft_consensus.cc:729] T b016e31e76854807af0cfaed6ec49071 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:42.555778 31494 consensus_queue.cc:260] T b016e31e76854807af0cfaed6ec49071 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:42.556607 31494 ts_tablet_manager.cc:1398] T b016e31e76854807af0cfaed6ec49071 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:42.556995 31498 tablet_bootstrap.cc:654] T 8c714f29abb140379b57e08eb927e2b7 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:42.558076 31495 tablet_bootstrap.cc:492] T b016e31e76854807af0cfaed6ec49071 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:42.558295 31495 ts_tablet_manager.cc:1367] T b016e31e76854807af0cfaed6ec49071 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.002s
I20240503 08:12:42.558569 31493 tablet_bootstrap.cc:492] T d24e437f369b4df49e6363f3f492dc46 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:42.560115 31493 tablet_bootstrap.cc:654] T d24e437f369b4df49e6363f3f492dc46 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:42.563676 31497 tablet_bootstrap.cc:492] T d24e437f369b4df49e6363f3f492dc46 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:42.563903 31497 ts_tablet_manager.cc:1367] T d24e437f369b4df49e6363f3f492dc46 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.003s
I20240503 08:12:42.564431 31491 tablet_bootstrap.cc:492] T b016e31e76854807af0cfaed6ec49071 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:42.564644 31491 ts_tablet_manager.cc:1367] T b016e31e76854807af0cfaed6ec49071 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.063s user 0.014s sys 0.009s
I20240503 08:12:42.564682 31497 raft_consensus.cc:348] T d24e437f369b4df49e6363f3f492dc46 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:42.565712 31495 raft_consensus.cc:348] T b016e31e76854807af0cfaed6ec49071 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:42.565986 31495 raft_consensus.cc:374] T b016e31e76854807af0cfaed6ec49071 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:42.566099 31495 raft_consensus.cc:729] T b016e31e76854807af0cfaed6ec49071 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:42.566190 31497 raft_consensus.cc:374] T d24e437f369b4df49e6363f3f492dc46 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:42.566319 31497 raft_consensus.cc:729] T d24e437f369b4df49e6363f3f492dc46 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:42.566593 31495 consensus_queue.cc:260] T b016e31e76854807af0cfaed6ec49071 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:42.567003 31497 consensus_queue.cc:260] T d24e437f369b4df49e6363f3f492dc46 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:42.568338 31491 raft_consensus.cc:348] T b016e31e76854807af0cfaed6ec49071 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:42.568683 31491 raft_consensus.cc:374] T b016e31e76854807af0cfaed6ec49071 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:42.568807 31491 raft_consensus.cc:729] T b016e31e76854807af0cfaed6ec49071 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:42.569550 31491 consensus_queue.cc:260] T b016e31e76854807af0cfaed6ec49071 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:42.570798 31491 ts_tablet_manager.cc:1398] T b016e31e76854807af0cfaed6ec49071 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 08:12:42.571069 31491 tablet_bootstrap.cc:492] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:42.571631 31498 tablet_bootstrap.cc:492] T 8c714f29abb140379b57e08eb927e2b7 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:42.572070 31497 ts_tablet_manager.cc:1398] T d24e437f369b4df49e6363f3f492dc46 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240503 08:12:42.571803 31498 ts_tablet_manager.cc:1367] T 8c714f29abb140379b57e08eb927e2b7 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.000s
I20240503 08:12:42.572901 31491 tablet_bootstrap.cc:654] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:42.574151 31498 raft_consensus.cc:348] T 8c714f29abb140379b57e08eb927e2b7 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:42.574544 31498 raft_consensus.cc:374] T 8c714f29abb140379b57e08eb927e2b7 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:42.574661 31498 raft_consensus.cc:729] T 8c714f29abb140379b57e08eb927e2b7 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:42.574924 31498 consensus_queue.cc:260] T 8c714f29abb140379b57e08eb927e2b7 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:42.575464 31498 ts_tablet_manager.cc:1398] T 8c714f29abb140379b57e08eb927e2b7 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:12:42.575528 31493 tablet_bootstrap.cc:492] T d24e437f369b4df49e6363f3f492dc46 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:42.575675 31493 ts_tablet_manager.cc:1367] T d24e437f369b4df49e6363f3f492dc46 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.007s
I20240503 08:12:42.575995 31491 tablet_bootstrap.cc:492] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:42.576138 31491 ts_tablet_manager.cc:1367] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240503 08:12:42.575222 31495 ts_tablet_manager.cc:1398] T b016e31e76854807af0cfaed6ec49071 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20240503 08:12:42.576678 31493 raft_consensus.cc:348] T d24e437f369b4df49e6363f3f492dc46 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:42.576874 31493 raft_consensus.cc:374] T d24e437f369b4df49e6363f3f492dc46 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:42.576957 31493 raft_consensus.cc:729] T d24e437f369b4df49e6363f3f492dc46 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:42.577090 31497 tablet_bootstrap.cc:492] T 8c714f29abb140379b57e08eb927e2b7 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:42.577139 31493 consensus_queue.cc:260] T d24e437f369b4df49e6363f3f492dc46 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:42.577644 31493 ts_tablet_manager.cc:1398] T d24e437f369b4df49e6363f3f492dc46 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:12:42.578014 31309 raft_consensus.cc:482] T b016e31e76854807af0cfaed6ec49071 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:42.578140 31309 raft_consensus.cc:504] T b016e31e76854807af0cfaed6ec49071 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:42.578593 31497 tablet_bootstrap.cc:654] T 8c714f29abb140379b57e08eb927e2b7 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:42.578774 31309 leader_election.cc:288] T b016e31e76854807af0cfaed6ec49071 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:12:42.580232 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b016e31e76854807af0cfaed6ec49071" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:12:42.580435 30762 raft_consensus.cc:2454] T b016e31e76854807af0cfaed6ec49071 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:12:42.580904 30829 leader_election.cc:302] T b016e31e76854807af0cfaed6ec49071 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:42.581138 31379 raft_consensus.cc:2781] T b016e31e76854807af0cfaed6ec49071 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:42.581218 31379 raft_consensus.cc:482] T b016e31e76854807af0cfaed6ec49071 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:42.581280 31379 raft_consensus.cc:3037] T b016e31e76854807af0cfaed6ec49071 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:42.583938 31379 raft_consensus.cc:504] T b016e31e76854807af0cfaed6ec49071 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:42.584178 27862 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240503 08:12:42.584368 27862 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240503 08:12:42.586577 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b016e31e76854807af0cfaed6ec49071" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:12:42.586747 30762 raft_consensus.cc:3037] T b016e31e76854807af0cfaed6ec49071 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:42.587105 31496 tablet_bootstrap.cc:492] T d24e437f369b4df49e6363f3f492dc46 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:42.587252 31496 ts_tablet_manager.cc:1367] T d24e437f369b4df49e6363f3f492dc46 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.002s
I20240503 08:12:42.588017 31496 raft_consensus.cc:348] T d24e437f369b4df49e6363f3f492dc46 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:42.588201 31496 raft_consensus.cc:374] T d24e437f369b4df49e6363f3f492dc46 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:42.588307 31496 raft_consensus.cc:729] T d24e437f369b4df49e6363f3f492dc46 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:42.588553 31496 consensus_queue.cc:260] T d24e437f369b4df49e6363f3f492dc46 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:42.589367 30762 raft_consensus.cc:2454] T b016e31e76854807af0cfaed6ec49071 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:12:42.589649 31496 ts_tablet_manager.cc:1398] T d24e437f369b4df49e6363f3f492dc46 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:42.590049 30829 leader_election.cc:302] T b016e31e76854807af0cfaed6ec49071 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:42.590292 27862 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet b8560620faf54b51ba4c92d8cf95be2d
I20240503 08:12:42.590777 31309 raft_consensus.cc:2781] T b016e31e76854807af0cfaed6ec49071 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:42.590889 31309 raft_consensus.cc:686] T b016e31e76854807af0cfaed6ec49071 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Becoming Leader. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Running, Role: LEADER
I20240503 08:12:42.590978 28242 tablet_service.cc:1514] Processing DeleteTablet for tablet b8560620faf54b51ba4c92d8cf95be2d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:42 UTC) from {username='slave'} at 127.0.0.1:41960
I20240503 08:12:42.591109 31309 consensus_queue.cc:237] T b016e31e76854807af0cfaed6ec49071 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:42.591224 31409 tablet_replica.cc:330] T b8560620faf54b51ba4c92d8cf95be2d P aae860d7ae2b4002bf7794c278f9cc56: stopping tablet replica
I20240503 08:12:42.591449 31409 raft_consensus.cc:2229] T b8560620faf54b51ba4c92d8cf95be2d P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:42.591647 31409 raft_consensus.cc:2258] T b8560620faf54b51ba4c92d8cf95be2d P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:42.592379 31409 ts_tablet_manager.cc:1875] T b8560620faf54b51ba4c92d8cf95be2d P aae860d7ae2b4002bf7794c278f9cc56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:42.594933 27862 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet 582128d9dc884a9ab5ecc3e1c90e535e
I20240503 08:12:42.595266 28112 tablet_service.cc:1514] Processing DeleteTablet for tablet b8560620faf54b51ba4c92d8cf95be2d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:42 UTC) from {username='slave'} at 127.0.0.1:57912
I20240503 08:12:42.595530 31267 tablet_replica.cc:330] T b8560620faf54b51ba4c92d8cf95be2d P 1b411c6bb25549368041773d7b710d48: stopping tablet replica
I20240503 08:12:42.595734 31267 raft_consensus.cc:2229] T b8560620faf54b51ba4c92d8cf95be2d P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:42.595957 31267 raft_consensus.cc:2258] T b8560620faf54b51ba4c92d8cf95be2d P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:42.596300 27862 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet 2fbd9559f45645fd8df45491295224ba
I20240503 08:12:42.596689 31267 ts_tablet_manager.cc:1875] T b8560620faf54b51ba4c92d8cf95be2d P 1b411c6bb25549368041773d7b710d48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:42.597168 31379 leader_election.cc:288] T b016e31e76854807af0cfaed6ec49071 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Requested vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:12:42.598542 28373 tablet_service.cc:1514] Processing DeleteTablet for tablet 582128d9dc884a9ab5ecc3e1c90e535e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:42 UTC) from {username='slave'} at 127.0.0.1:34786
I20240503 08:12:42.598796 28372 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fbd9559f45645fd8df45491295224ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:42 UTC) from {username='slave'} at 127.0.0.1:34786
I20240503 08:12:42.598992 31268 tablet_replica.cc:330] T 582128d9dc884a9ab5ecc3e1c90e535e P d09a11a03989421b9f28019dc1c14cc2: stopping tablet replica
I20240503 08:12:42.599192 31268 raft_consensus.cc:2229] T 582128d9dc884a9ab5ecc3e1c90e535e P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:42.599391 31268 raft_consensus.cc:2258] T 582128d9dc884a9ab5ecc3e1c90e535e P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:42.600013 31268 ts_tablet_manager.cc:1875] T 582128d9dc884a9ab5ecc3e1c90e535e P d09a11a03989421b9f28019dc1c14cc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:42.601538 31409 ts_tablet_manager.cc:1888] T b8560620faf54b51ba4c92d8cf95be2d P aae860d7ae2b4002bf7794c278f9cc56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:42.601673 31409 log.cc:1196] T b8560620faf54b51ba4c92d8cf95be2d P aae860d7ae2b4002bf7794c278f9cc56: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal/wals/b8560620faf54b51ba4c92d8cf95be2d
I20240503 08:12:42.601943 28112 tablet_service.cc:1514] Processing DeleteTablet for tablet 582128d9dc884a9ab5ecc3e1c90e535e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:42 UTC) from {username='slave'} at 127.0.0.1:57912
I20240503 08:12:42.602084 31409 ts_tablet_manager.cc:1909] T b8560620faf54b51ba4c92d8cf95be2d P aae860d7ae2b4002bf7794c278f9cc56: Deleting consensus metadata
I20240503 08:12:42.602865 28242 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fbd9559f45645fd8df45491295224ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:42 UTC) from {username='slave'} at 127.0.0.1:41960
I20240503 08:12:42.603251 31409 tablet_replica.cc:330] T 2fbd9559f45645fd8df45491295224ba P aae860d7ae2b4002bf7794c278f9cc56: stopping tablet replica
I20240503 08:12:42.603425 31409 raft_consensus.cc:2229] T 2fbd9559f45645fd8df45491295224ba P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:42.603616 31409 raft_consensus.cc:2258] T 2fbd9559f45645fd8df45491295224ba P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:42.604204 31409 ts_tablet_manager.cc:1875] T 2fbd9559f45645fd8df45491295224ba P aae860d7ae2b4002bf7794c278f9cc56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:42.604205 27847 catalog_manager.cc:4830] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): tablet b8560620faf54b51ba4c92d8cf95be2d (table rebalance_test_table_8 [id=53655b23fc8b423298fafe1b2c13a1f7]) successfully deleted
I20240503 08:12:42.607234 31491 raft_consensus.cc:348] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:42.607452 31491 raft_consensus.cc:374] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:42.607511 31491 raft_consensus.cc:729] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:42.607708 31491 consensus_queue.cc:260] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:42.608191 31491 ts_tablet_manager.cc:1398] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.032s user 0.002s sys 0.000s
I20240503 08:12:42.613955 31267 ts_tablet_manager.cc:1888] T b8560620faf54b51ba4c92d8cf95be2d P 1b411c6bb25549368041773d7b710d48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
W20240503 08:12:42.614715 27938 consensus_peers.cc:485] T b8560620faf54b51ba4c92d8cf95be2d P 833582ff68c94269be364ebd187a75db -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:42.616353 27982 tablet_service.cc:1514] Processing DeleteTablet for tablet b8560620faf54b51ba4c92d8cf95be2d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:42 UTC) from {username='slave'} at 127.0.0.1:55610
I20240503 08:12:42.616452 31486 raft_consensus.cc:482] T 8c714f29abb140379b57e08eb927e2b7 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:42.616607 31486 raft_consensus.cc:504] T 8c714f29abb140379b57e08eb927e2b7 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:42.617344 31486 leader_election.cc:288] T 8c714f29abb140379b57e08eb927e2b7 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:42.618518 31161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c714f29abb140379b57e08eb927e2b7" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
W20240503 08:12:42.619092 28196 leader_election.cc:341] T 8c714f29abb140379b57e08eb927e2b7 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:42.622684 27981 tablet_service.cc:1514] Processing DeleteTablet for tablet 582128d9dc884a9ab5ecc3e1c90e535e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:42 UTC) from {username='slave'} at 127.0.0.1:55610
I20240503 08:12:42.623159 27981 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fbd9559f45645fd8df45491295224ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:42 UTC) from {username='slave'} at 127.0.0.1:55610
I20240503 08:12:42.626240 27856 catalog_manager.cc:5484] T b016e31e76854807af0cfaed6ec49071 P ce1496a47d084365ad786cd09e472a2c reported cstate change: term changed from 0 to 1, leader changed from <none> to ce1496a47d084365ad786cd09e472a2c (127.20.230.70). New cstate: current_term: 1 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:12:42.629698 31497 tablet_bootstrap.cc:492] T 8c714f29abb140379b57e08eb927e2b7 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:42.629921 31497 ts_tablet_manager.cc:1367] T 8c714f29abb140379b57e08eb927e2b7 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.053s user 0.004s sys 0.000s
I20240503 08:12:42.631601 31497 raft_consensus.cc:348] T 8c714f29abb140379b57e08eb927e2b7 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:42.631876 31497 raft_consensus.cc:374] T 8c714f29abb140379b57e08eb927e2b7 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:42.632009 31497 raft_consensus.cc:729] T 8c714f29abb140379b57e08eb927e2b7 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:42.632359 31268 ts_tablet_manager.cc:1888] T 582128d9dc884a9ab5ecc3e1c90e535e P d09a11a03989421b9f28019dc1c14cc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:42.632349 31409 ts_tablet_manager.cc:1888] T 2fbd9559f45645fd8df45491295224ba P aae860d7ae2b4002bf7794c278f9cc56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:42.632344 31497 consensus_queue.cc:260] T 8c714f29abb140379b57e08eb927e2b7 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:42.632498 31409 log.cc:1196] T 2fbd9559f45645fd8df45491295224ba P aae860d7ae2b4002bf7794c278f9cc56: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal/wals/2fbd9559f45645fd8df45491295224ba
I20240503 08:12:42.632740 31268 log.cc:1196] T 582128d9dc884a9ab5ecc3e1c90e535e P d09a11a03989421b9f28019dc1c14cc2: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/wals/582128d9dc884a9ab5ecc3e1c90e535e
I20240503 08:12:42.632966 31267 log.cc:1196] T b8560620faf54b51ba4c92d8cf95be2d P 1b411c6bb25549368041773d7b710d48: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/b8560620faf54b51ba4c92d8cf95be2d
I20240503 08:12:42.633011 31409 ts_tablet_manager.cc:1909] T 2fbd9559f45645fd8df45491295224ba P aae860d7ae2b4002bf7794c278f9cc56: Deleting consensus metadata
I20240503 08:12:42.634076 27847 catalog_manager.cc:4830] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): tablet 2fbd9559f45645fd8df45491295224ba (table rebalance_test_table_8 [id=53655b23fc8b423298fafe1b2c13a1f7]) successfully deleted
I20240503 08:12:42.634733 31497 ts_tablet_manager.cc:1398] T 8c714f29abb140379b57e08eb927e2b7 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 08:12:42.635677 31309 raft_consensus.cc:482] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:42.635860 31309 raft_consensus.cc:504] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:42.636117 31267 ts_tablet_manager.cc:1909] T b8560620faf54b51ba4c92d8cf95be2d P 1b411c6bb25549368041773d7b710d48: Deleting consensus metadata
I20240503 08:12:42.636369 31268 ts_tablet_manager.cc:1909] T 582128d9dc884a9ab5ecc3e1c90e535e P d09a11a03989421b9f28019dc1c14cc2: Deleting consensus metadata
I20240503 08:12:42.636672 31309 leader_election.cc:288] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:42.642661 27856 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:42.645090 27848 catalog_manager.cc:4830] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): tablet b8560620faf54b51ba4c92d8cf95be2d (table rebalance_test_table_8 [id=53655b23fc8b423298fafe1b2c13a1f7]) successfully deleted
I20240503 08:12:42.645825 27849 catalog_manager.cc:4830] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): tablet 582128d9dc884a9ab5ecc3e1c90e535e (table rebalance_test_table_8 [id=53655b23fc8b423298fafe1b2c13a1f7]) successfully deleted
I20240503 08:12:42.646097 31267 tablet_replica.cc:330] T 582128d9dc884a9ab5ecc3e1c90e535e P 1b411c6bb25549368041773d7b710d48: stopping tablet replica
I20240503 08:12:42.646153 31268 tablet_replica.cc:330] T 2fbd9559f45645fd8df45491295224ba P d09a11a03989421b9f28019dc1c14cc2: stopping tablet replica
I20240503 08:12:42.646292 31267 raft_consensus.cc:2229] T 582128d9dc884a9ab5ecc3e1c90e535e P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:42.646517 31267 raft_consensus.cc:2258] T 582128d9dc884a9ab5ecc3e1c90e535e P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:42.647126 31267 ts_tablet_manager.cc:1875] T 582128d9dc884a9ab5ecc3e1c90e535e P 1b411c6bb25549368041773d7b710d48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:42.652609 31487 tablet.cc:1722] T 9fff870834d04fd59f77f14c3db9f3a1 P aae860d7ae2b4002bf7794c278f9cc56: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:42.657634 31267 ts_tablet_manager.cc:1888] T 582128d9dc884a9ab5ecc3e1c90e535e P 1b411c6bb25549368041773d7b710d48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:42.657820 31267 log.cc:1196] T 582128d9dc884a9ab5ecc3e1c90e535e P 1b411c6bb25549368041773d7b710d48: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/582128d9dc884a9ab5ecc3e1c90e535e
I20240503 08:12:42.657824 31511 tablet_replica.cc:330] T b8560620faf54b51ba4c92d8cf95be2d P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:12:42.658353 31267 ts_tablet_manager.cc:1909] T 582128d9dc884a9ab5ecc3e1c90e535e P 1b411c6bb25549368041773d7b710d48: Deleting consensus metadata
I20240503 08:12:42.658627 31511 raft_consensus.cc:2229] T b8560620faf54b51ba4c92d8cf95be2d P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:42.659008 31511 raft_consensus.cc:2258] T b8560620faf54b51ba4c92d8cf95be2d P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:42.659344 27848 catalog_manager.cc:4830] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): tablet 582128d9dc884a9ab5ecc3e1c90e535e (table rebalance_test_table_8 [id=53655b23fc8b423298fafe1b2c13a1f7]) successfully deleted
W20240503 08:12:42.660604 27939 consensus_peers.cc:485] T 582128d9dc884a9ab5ecc3e1c90e535e P 833582ff68c94269be364ebd187a75db -> Peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): Couldn't send request to peer d09a11a03989421b9f28019dc1c14cc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 582128d9dc884a9ab5ecc3e1c90e535e. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:42.660775 31511 ts_tablet_manager.cc:1875] T b8560620faf54b51ba4c92d8cf95be2d P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:42.664697 28261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c714f29abb140379b57e08eb927e2b7" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:12:42.664916 28261 raft_consensus.cc:2454] T 8c714f29abb140379b57e08eb927e2b7 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:12:42.665539 30894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c714f29abb140379b57e08eb927e2b7" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:12:42.665755 30894 raft_consensus.cc:2454] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:12:42.666199 28196 leader_election.cc:302] T 8c714f29abb140379b57e08eb927e2b7 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, ce1496a47d084365ad786cd09e472a2c; no voters: b00f244c69df401997ea99d492218a98
I20240503 08:12:42.666716 30652 raft_consensus.cc:2781] T 8c714f29abb140379b57e08eb927e2b7 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:42.666817 30652 raft_consensus.cc:482] T 8c714f29abb140379b57e08eb927e2b7 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:42.666889 30652 raft_consensus.cc:3037] T 8c714f29abb140379b57e08eb927e2b7 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:42.668998 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b016e31e76854807af0cfaed6ec49071" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:12:42.669134 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b016e31e76854807af0cfaed6ec49071" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:12:42.669229 28002 raft_consensus.cc:2454] T b016e31e76854807af0cfaed6ec49071 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:12:42.669286 31511 ts_tablet_manager.cc:1888] T b8560620faf54b51ba4c92d8cf95be2d P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:42.669214 30652 raft_consensus.cc:504] T 8c714f29abb140379b57e08eb927e2b7 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:42.669387 31511 log.cc:1196] T b8560620faf54b51ba4c92d8cf95be2d P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/b8560620faf54b51ba4c92d8cf95be2d
I20240503 08:12:42.669816 30652 leader_election.cc:288] T 8c714f29abb140379b57e08eb927e2b7 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Requested vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:42.669862 31511 ts_tablet_manager.cc:1909] T b8560620faf54b51ba4c92d8cf95be2d P 833582ff68c94269be364ebd187a75db: Deleting consensus metadata
I20240503 08:12:42.670085 31488 tablet.cc:1722] T 9fff870834d04fd59f77f14c3db9f3a1 P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:42.670744 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet b8560620faf54b51ba4c92d8cf95be2d (table rebalance_test_table_8 [id=53655b23fc8b423298fafe1b2c13a1f7]) successfully deleted
I20240503 08:12:42.671464 31511 tablet_replica.cc:330] T 582128d9dc884a9ab5ecc3e1c90e535e P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:12:42.671913 31161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c714f29abb140379b57e08eb927e2b7" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:12:42.672118 31161 raft_consensus.cc:3037] T 8c714f29abb140379b57e08eb927e2b7 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:42.672667 30828 leader_election.cc:302] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:42.673038 31509 raft_consensus.cc:2781] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:42.673121 31509 raft_consensus.cc:482] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:42.673205 31509 raft_consensus.cc:3037] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:42.673604 30894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c714f29abb140379b57e08eb927e2b7" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:12:42.674432 31268 raft_consensus.cc:2229] T 2fbd9559f45645fd8df45491295224ba P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:42.674497 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c714f29abb140379b57e08eb927e2b7" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:12:42.674639 31489 tablet.cc:1722] T 9fff870834d04fd59f77f14c3db9f3a1 P b00f244c69df401997ea99d492218a98: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:42.674976 31268 raft_consensus.cc:2258] T 2fbd9559f45645fd8df45491295224ba P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:42.675715 31268 ts_tablet_manager.cc:1875] T 2fbd9559f45645fd8df45491295224ba P d09a11a03989421b9f28019dc1c14cc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:42.677042 31509 raft_consensus.cc:504] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:42.677304 31161 raft_consensus.cc:2454] T 8c714f29abb140379b57e08eb927e2b7 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:42.677698 28196 leader_election.cc:302] T 8c714f29abb140379b57e08eb927e2b7 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, b00f244c69df401997ea99d492218a98; no voters:
I20240503 08:12:42.677891 30894 raft_consensus.cc:2379] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate aae860d7ae2b4002bf7794c278f9cc56 in current term 1: Already voted for candidate ce1496a47d084365ad786cd09e472a2c in this term.
I20240503 08:12:42.678130 31490 raft_consensus.cc:2781] T 8c714f29abb140379b57e08eb927e2b7 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:42.678254 31490 raft_consensus.cc:686] T 8c714f29abb140379b57e08eb927e2b7 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Becoming Leader. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Running, Role: LEADER
I20240503 08:12:42.678521 31490 consensus_queue.cc:237] T 8c714f29abb140379b57e08eb927e2b7 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:42.679591 31509 leader_election.cc:288] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:42.680500 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c714f29abb140379b57e08eb927e2b7" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:12:42.680716 31159 raft_consensus.cc:2379] T 8c714f29abb140379b57e08eb927e2b7 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ce1496a47d084365ad786cd09e472a2c in current term 1: Already voted for candidate aae860d7ae2b4002bf7794c278f9cc56 in this term.
I20240503 08:12:42.681564 31268 ts_tablet_manager.cc:1888] T 2fbd9559f45645fd8df45491295224ba P d09a11a03989421b9f28019dc1c14cc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:42.681730 31268 log.cc:1196] T 2fbd9559f45645fd8df45491295224ba P d09a11a03989421b9f28019dc1c14cc2: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/wals/2fbd9559f45645fd8df45491295224ba
I20240503 08:12:42.682196 31268 ts_tablet_manager.cc:1909] T 2fbd9559f45645fd8df45491295224ba P d09a11a03989421b9f28019dc1c14cc2: Deleting consensus metadata
I20240503 08:12:42.683147 27849 catalog_manager.cc:4830] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): tablet 2fbd9559f45645fd8df45491295224ba (table rebalance_test_table_8 [id=53655b23fc8b423298fafe1b2c13a1f7]) successfully deleted
I20240503 08:12:42.683526 28261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c714f29abb140379b57e08eb927e2b7" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:12:42.682884 27856 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table_new_name [id=88d4f1885e1240c4bd1ea1cf956cbee2] alter complete (version 3)
I20240503 08:12:42.684125 30828 leader_election.cc:302] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c; no voters: aae860d7ae2b4002bf7794c278f9cc56, b00f244c69df401997ea99d492218a98
I20240503 08:12:42.684404 31511 raft_consensus.cc:2229] T 582128d9dc884a9ab5ecc3e1c90e535e P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:42.684721 31509 raft_consensus.cc:2726] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20240503 08:12:42.682498 27938 consensus_peers.cc:485] T 582128d9dc884a9ab5ecc3e1c90e535e P 833582ff68c94269be364ebd187a75db -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 582128d9dc884a9ab5ecc3e1c90e535e. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:42.686582 31511 raft_consensus.cc:2258] T 582128d9dc884a9ab5ecc3e1c90e535e P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:42.686834 27859 catalog_manager.cc:5484] T 8c714f29abb140379b57e08eb927e2b7 P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: term changed from 0 to 1, leader changed from <none> to aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67). New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:12:42.687288 31511 ts_tablet_manager.cc:1875] T 582128d9dc884a9ab5ecc3e1c90e535e P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:42.692171 31511 ts_tablet_manager.cc:1888] T 582128d9dc884a9ab5ecc3e1c90e535e P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:42.692281 31511 log.cc:1196] T 582128d9dc884a9ab5ecc3e1c90e535e P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/582128d9dc884a9ab5ecc3e1c90e535e
I20240503 08:12:42.692593 31511 ts_tablet_manager.cc:1909] T 582128d9dc884a9ab5ecc3e1c90e535e P 833582ff68c94269be364ebd187a75db: Deleting consensus metadata
I20240503 08:12:42.693230 31511 tablet_replica.cc:330] T 2fbd9559f45645fd8df45491295224ba P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:12:42.693316 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet 582128d9dc884a9ab5ecc3e1c90e535e (table rebalance_test_table_8 [id=53655b23fc8b423298fafe1b2c13a1f7]) successfully deleted
I20240503 08:12:42.693434 31511 raft_consensus.cc:2229] T 2fbd9559f45645fd8df45491295224ba P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:42.693631 31511 raft_consensus.cc:2258] T 2fbd9559f45645fd8df45491295224ba P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:42.694144 31511 ts_tablet_manager.cc:1875] T 2fbd9559f45645fd8df45491295224ba P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:42.697870 31511 ts_tablet_manager.cc:1888] T 2fbd9559f45645fd8df45491295224ba P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:42.697969 31511 log.cc:1196] T 2fbd9559f45645fd8df45491295224ba P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/2fbd9559f45645fd8df45491295224ba
I20240503 08:12:42.698256 31511 ts_tablet_manager.cc:1909] T 2fbd9559f45645fd8df45491295224ba P 833582ff68c94269be364ebd187a75db: Deleting consensus metadata
I20240503 08:12:42.698953 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet 2fbd9559f45645fd8df45491295224ba (table rebalance_test_table_8 [id=53655b23fc8b423298fafe1b2c13a1f7]) successfully deleted
I20240503 08:12:42.720125 31370 raft_consensus.cc:482] T d24e437f369b4df49e6363f3f492dc46 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:42.720276 31370 raft_consensus.cc:504] T d24e437f369b4df49e6363f3f492dc46 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:42.720911 31370 leader_election.cc:288] T d24e437f369b4df49e6363f3f492dc46 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:42.721460 31161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24e437f369b4df49e6363f3f492dc46" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:12:42.721690 31161 raft_consensus.cc:2454] T d24e437f369b4df49e6363f3f492dc46 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:12:42.722067 28326 leader_election.cc:302] T d24e437f369b4df49e6363f3f492dc46 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:42.722350 31370 raft_consensus.cc:2781] T d24e437f369b4df49e6363f3f492dc46 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:42.722508 31370 raft_consensus.cc:482] T d24e437f369b4df49e6363f3f492dc46 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:42.722618 31370 raft_consensus.cc:3037] T d24e437f369b4df49e6363f3f492dc46 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:42.724723 31370 raft_consensus.cc:504] T d24e437f369b4df49e6363f3f492dc46 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:42.725734 31370 leader_election.cc:288] T d24e437f369b4df49e6363f3f492dc46 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Requested vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:42.726259 31161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24e437f369b4df49e6363f3f492dc46" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:12:42.726745 31161 raft_consensus.cc:3037] T d24e437f369b4df49e6363f3f492dc46 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:42.727595 31028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24e437f369b4df49e6363f3f492dc46" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:42.727653 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24e437f369b4df49e6363f3f492dc46" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:12:42.727833 31028 raft_consensus.cc:2454] T d24e437f369b4df49e6363f3f492dc46 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:12:42.728828 31161 raft_consensus.cc:2454] T d24e437f369b4df49e6363f3f492dc46 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:12:42.729437 28326 leader_election.cc:302] T d24e437f369b4df49e6363f3f492dc46 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b00f244c69df401997ea99d492218a98, d09a11a03989421b9f28019dc1c14cc2; no voters: ced21ca983b749d0b4d4f6a0e2426bf8
I20240503 08:12:42.729701 31370 raft_consensus.cc:2781] T d24e437f369b4df49e6363f3f492dc46 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:42.729890 31370 raft_consensus.cc:686] T d24e437f369b4df49e6363f3f492dc46 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Becoming Leader. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Running, Role: LEADER
I20240503 08:12:42.730374 31370 consensus_queue.cc:237] T d24e437f369b4df49e6363f3f492dc46 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:42.733366 27859 catalog_manager.cc:5484] T d24e437f369b4df49e6363f3f492dc46 P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68). New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:12:42.787220 27859 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:42.787444 27859 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:42.790279 27859 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet 9fff870834d04fd59f77f14c3db9f3a1
I20240503 08:12:42.791026 31008 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fff870834d04fd59f77f14c3db9f3a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:42 UTC) from {username='slave'} at 127.0.0.1:47020
I20240503 08:12:42.791128 31140 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fff870834d04fd59f77f14c3db9f3a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:42 UTC) from {username='slave'} at 127.0.0.1:45814
I20240503 08:12:42.791143 28242 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fff870834d04fd59f77f14c3db9f3a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:42 UTC) from {username='slave'} at 127.0.0.1:41960
I20240503 08:12:42.791390 31409 tablet_replica.cc:330] T 9fff870834d04fd59f77f14c3db9f3a1 P aae860d7ae2b4002bf7794c278f9cc56: stopping tablet replica
I20240503 08:12:42.791625 31409 raft_consensus.cc:2229] T 9fff870834d04fd59f77f14c3db9f3a1 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:42.791755 31521 tablet_replica.cc:330] T 9fff870834d04fd59f77f14c3db9f3a1 P b00f244c69df401997ea99d492218a98: stopping tablet replica
I20240503 08:12:42.791885 31520 tablet_replica.cc:330] T 9fff870834d04fd59f77f14c3db9f3a1 P ced21ca983b749d0b4d4f6a0e2426bf8: stopping tablet replica
I20240503 08:12:42.791946 31521 raft_consensus.cc:2229] T 9fff870834d04fd59f77f14c3db9f3a1 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:42.791947 31409 raft_consensus.cc:2258] T 9fff870834d04fd59f77f14c3db9f3a1 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:42.792034 31520 raft_consensus.cc:2229] T 9fff870834d04fd59f77f14c3db9f3a1 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:42.792151 31521 raft_consensus.cc:2258] T 9fff870834d04fd59f77f14c3db9f3a1 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:42.792218 31520 raft_consensus.cc:2258] T 9fff870834d04fd59f77f14c3db9f3a1 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:42.792873 31409 ts_tablet_manager.cc:1875] T 9fff870834d04fd59f77f14c3db9f3a1 P aae860d7ae2b4002bf7794c278f9cc56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:42.792971 31521 ts_tablet_manager.cc:1875] T 9fff870834d04fd59f77f14c3db9f3a1 P b00f244c69df401997ea99d492218a98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:42.793010 31520 ts_tablet_manager.cc:1875] T 9fff870834d04fd59f77f14c3db9f3a1 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:42.797327 31521 ts_tablet_manager.cc:1888] T 9fff870834d04fd59f77f14c3db9f3a1 P b00f244c69df401997ea99d492218a98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:42.797441 31521 log.cc:1196] T 9fff870834d04fd59f77f14c3db9f3a1 P b00f244c69df401997ea99d492218a98: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal/wals/9fff870834d04fd59f77f14c3db9f3a1
I20240503 08:12:42.797824 31521 ts_tablet_manager.cc:1909] T 9fff870834d04fd59f77f14c3db9f3a1 P b00f244c69df401997ea99d492218a98: Deleting consensus metadata
I20240503 08:12:42.798888 27846 catalog_manager.cc:4830] TS b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): tablet 9fff870834d04fd59f77f14c3db9f3a1 (table rebalancer_dynamic_table_new_name [id=88d4f1885e1240c4bd1ea1cf956cbee2]) successfully deleted
I20240503 08:12:42.798955 31409 ts_tablet_manager.cc:1888] T 9fff870834d04fd59f77f14c3db9f3a1 P aae860d7ae2b4002bf7794c278f9cc56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:42.798969 31520 ts_tablet_manager.cc:1888] T 9fff870834d04fd59f77f14c3db9f3a1 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:42.799070 31520 log.cc:1196] T 9fff870834d04fd59f77f14c3db9f3a1 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal/wals/9fff870834d04fd59f77f14c3db9f3a1
I20240503 08:12:42.799077 31409 log.cc:1196] T 9fff870834d04fd59f77f14c3db9f3a1 P aae860d7ae2b4002bf7794c278f9cc56: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal/wals/9fff870834d04fd59f77f14c3db9f3a1
I20240503 08:12:42.799463 31409 ts_tablet_manager.cc:1909] T 9fff870834d04fd59f77f14c3db9f3a1 P aae860d7ae2b4002bf7794c278f9cc56: Deleting consensus metadata
I20240503 08:12:42.799499 31520 ts_tablet_manager.cc:1909] T 9fff870834d04fd59f77f14c3db9f3a1 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting consensus metadata
I20240503 08:12:42.800386 27847 catalog_manager.cc:4830] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): tablet 9fff870834d04fd59f77f14c3db9f3a1 (table rebalancer_dynamic_table_new_name [id=88d4f1885e1240c4bd1ea1cf956cbee2]) successfully deleted
I20240503 08:12:42.800587 27847 catalog_manager.cc:4830] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): tablet 9fff870834d04fd59f77f14c3db9f3a1 (table rebalancer_dynamic_table_new_name [id=88d4f1885e1240c4bd1ea1cf956cbee2]) successfully deleted
I20240503 08:12:42.892371 27859 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:42.899101 31008 tablet_service.cc:1467] Processing CreateTablet for tablet f607fd8558ca4619879e54220c4aa823 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f6a728e264764764a3692179ed402756]), partition=
I20240503 08:12:42.899446 30873 tablet_service.cc:1467] Processing CreateTablet for tablet f607fd8558ca4619879e54220c4aa823 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f6a728e264764764a3692179ed402756]), partition=
I20240503 08:12:42.899528 31008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f607fd8558ca4619879e54220c4aa823. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:42.899574 27982 tablet_service.cc:1467] Processing CreateTablet for tablet f607fd8558ca4619879e54220c4aa823 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f6a728e264764764a3692179ed402756]), partition=
I20240503 08:12:42.899788 30873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f607fd8558ca4619879e54220c4aa823. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:42.899865 27982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f607fd8558ca4619879e54220c4aa823. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:42.903913 31496 tablet_bootstrap.cc:492] T f607fd8558ca4619879e54220c4aa823 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:42.905259 31496 tablet_bootstrap.cc:654] T f607fd8558ca4619879e54220c4aa823 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:42.905656 31494 tablet_bootstrap.cc:492] T f607fd8558ca4619879e54220c4aa823 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:42.905671 31491 tablet_bootstrap.cc:492] T f607fd8558ca4619879e54220c4aa823 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:42.907042 31491 tablet_bootstrap.cc:654] T f607fd8558ca4619879e54220c4aa823 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:42.907065 31494 tablet_bootstrap.cc:654] T f607fd8558ca4619879e54220c4aa823 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:42.917357 31496 tablet_bootstrap.cc:492] T f607fd8558ca4619879e54220c4aa823 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:42.917539 31496 ts_tablet_manager.cc:1367] T f607fd8558ca4619879e54220c4aa823 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.006s
I20240503 08:12:42.917574 31494 tablet_bootstrap.cc:492] T f607fd8558ca4619879e54220c4aa823 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:42.917601 31491 tablet_bootstrap.cc:492] T f607fd8558ca4619879e54220c4aa823 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:42.917721 31491 ts_tablet_manager.cc:1367] T f607fd8558ca4619879e54220c4aa823 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20240503 08:12:42.917788 31494 ts_tablet_manager.cc:1367] T f607fd8558ca4619879e54220c4aa823 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.003s
I20240503 08:12:42.918536 31494 raft_consensus.cc:348] T f607fd8558ca4619879e54220c4aa823 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:42.918641 31491 raft_consensus.cc:348] T f607fd8558ca4619879e54220c4aa823 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:42.918825 31491 raft_consensus.cc:374] T f607fd8558ca4619879e54220c4aa823 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:42.918728 31496 raft_consensus.cc:348] T f607fd8558ca4619879e54220c4aa823 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:42.918849 31494 raft_consensus.cc:374] T f607fd8558ca4619879e54220c4aa823 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:42.918916 31494 raft_consensus.cc:729] T f607fd8558ca4619879e54220c4aa823 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:42.918942 31496 raft_consensus.cc:374] T f607fd8558ca4619879e54220c4aa823 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:42.919036 31496 raft_consensus.cc:729] T f607fd8558ca4619879e54220c4aa823 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:42.919063 31491 raft_consensus.cc:729] T f607fd8558ca4619879e54220c4aa823 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:42.919144 31494 consensus_queue.cc:260] T f607fd8558ca4619879e54220c4aa823 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:42.919291 31491 consensus_queue.cc:260] T f607fd8558ca4619879e54220c4aa823 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:42.919303 31496 consensus_queue.cc:260] T f607fd8558ca4619879e54220c4aa823 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:42.919651 31494 ts_tablet_manager.cc:1398] T f607fd8558ca4619879e54220c4aa823 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:12:42.919806 31491 ts_tablet_manager.cc:1398] T f607fd8558ca4619879e54220c4aa823 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:12:42.921030 31496 ts_tablet_manager.cc:1398] T f607fd8558ca4619879e54220c4aa823 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 08:12:42.932337 30944 raft_consensus.cc:482] T f607fd8558ca4619879e54220c4aa823 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:42.932464 30944 raft_consensus.cc:504] T f607fd8558ca4619879e54220c4aa823 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:42.933233 30944 leader_election.cc:288] T f607fd8558ca4619879e54220c4aa823 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:12:42.933383 31028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f607fd8558ca4619879e54220c4aa823" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:42.933403 30894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f607fd8558ca4619879e54220c4aa823" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:12:42.933611 31028 raft_consensus.cc:2454] T f607fd8558ca4619879e54220c4aa823 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:42.933611 30894 raft_consensus.cc:2454] T f607fd8558ca4619879e54220c4aa823 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:42.934001 27937 leader_election.cc:302] T f607fd8558ca4619879e54220c4aa823 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:42.934259 30944 raft_consensus.cc:2781] T f607fd8558ca4619879e54220c4aa823 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:42.934365 30944 raft_consensus.cc:482] T f607fd8558ca4619879e54220c4aa823 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:42.934449 30944 raft_consensus.cc:3037] T f607fd8558ca4619879e54220c4aa823 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:42.936813 30944 raft_consensus.cc:504] T f607fd8558ca4619879e54220c4aa823 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:42.937398 30944 leader_election.cc:288] T f607fd8558ca4619879e54220c4aa823 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Requested vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:12:42.937703 31028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f607fd8558ca4619879e54220c4aa823" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:12:42.937810 30894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f607fd8558ca4619879e54220c4aa823" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:12:42.937889 31028 raft_consensus.cc:3037] T f607fd8558ca4619879e54220c4aa823 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:42.937999 30894 raft_consensus.cc:3037] T f607fd8558ca4619879e54220c4aa823 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:42.939766 31028 raft_consensus.cc:2454] T f607fd8558ca4619879e54220c4aa823 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:42.939782 30894 raft_consensus.cc:2454] T f607fd8558ca4619879e54220c4aa823 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:42.940101 27936 leader_election.cc:302] T f607fd8558ca4619879e54220c4aa823 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:42.940356 30944 raft_consensus.cc:2781] T f607fd8558ca4619879e54220c4aa823 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:42.940483 30944 raft_consensus.cc:686] T f607fd8558ca4619879e54220c4aa823 P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Becoming Leader. State: Replica: 833582ff68c94269be364ebd187a75db, State: Running, Role: LEADER
I20240503 08:12:42.940717 30944 consensus_queue.cc:237] T f607fd8558ca4619879e54220c4aa823 P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:42.943809 27855 catalog_manager.cc:5484] T f607fd8558ca4619879e54220c4aa823 P 833582ff68c94269be364ebd187a75db reported cstate change: term changed from 0 to 1, leader changed from <none> to 833582ff68c94269be364ebd187a75db (127.20.230.65). New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:12:43.007783 28003 raft_consensus.cc:3037] T b016e31e76854807af0cfaed6ec49071 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:43.010300 31509 consensus_queue.cc:1035] T b016e31e76854807af0cfaed6ec49071 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:43.014446 31509 consensus_queue.cc:1035] T b016e31e76854807af0cfaed6ec49071 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:43.056630 27855 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:43.062964 30894 raft_consensus.cc:1261] T f607fd8558ca4619879e54220c4aa823 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:43.063025 31028 raft_consensus.cc:1261] T f607fd8558ca4619879e54220c4aa823 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:43.063704 30686 consensus_queue.cc:1035] T f607fd8558ca4619879e54220c4aa823 P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:43.064982 30686 consensus_queue.cc:1035] T f607fd8558ca4619879e54220c4aa823 P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:43.068868 27855 catalog_manager.cc:5576] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747) does not have the latest schema for tablet f607fd8558ca4619879e54220c4aa823 (table rebalancer_dynamic_table [id=f6a728e264764764a3692179ed402756]). Expected version 1 got 0
I20240503 08:12:43.071492 31533 tablet.cc:1722] T f607fd8558ca4619879e54220c4aa823 P 833582ff68c94269be364ebd187a75db: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:43.072778 31533 tablet.cc:1620] T f607fd8558ca4619879e54220c4aa823 P 833582ff68c94269be364ebd187a75db: MemRowSet was empty: no flush needed.
I20240503 08:12:43.072863 31488 tablet.cc:1722] T f607fd8558ca4619879e54220c4aa823 P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:43.073311 31488 tablet.cc:1620] T f607fd8558ca4619879e54220c4aa823 P ced21ca983b749d0b4d4f6a0e2426bf8: MemRowSet was empty: no flush needed.
I20240503 08:12:43.073941 27855 catalog_manager.cc:5576] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915) does not have the latest schema for tablet f607fd8558ca4619879e54220c4aa823 (table rebalancer_dynamic_table [id=f6a728e264764764a3692179ed402756]). Expected version 1 got 0
I20240503 08:12:43.078651 27855 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=f6a728e264764764a3692179ed402756] alter complete (version 1)
I20240503 08:12:43.079252 31534 tablet.cc:1722] T f607fd8558ca4619879e54220c4aa823 P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:43.080329 31534 tablet.cc:1620] T f607fd8558ca4619879e54220c4aa823 P ce1496a47d084365ad786cd09e472a2c: MemRowSet was empty: no flush needed.
I20240503 08:12:43.082511 27855 catalog_manager.cc:5576] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747) does not have the latest schema for tablet f607fd8558ca4619879e54220c4aa823 (table rebalancer_dynamic_table [id=f6a728e264764764a3692179ed402756]). Expected version 1 got 0
I20240503 08:12:43.104619 27855 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240503 08:12:43.104812 27855 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240503 08:12:43.108158 27855 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet cdc2d9cfc64d42ff9a9b30bb04f4433e
I20240503 08:12:43.108889 28372 tablet_service.cc:1514] Processing DeleteTablet for tablet cdc2d9cfc64d42ff9a9b30bb04f4433e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:43 UTC) from {username='slave'} at 127.0.0.1:34786
I20240503 08:12:43.108963 28112 tablet_service.cc:1514] Processing DeleteTablet for tablet cdc2d9cfc64d42ff9a9b30bb04f4433e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:43 UTC) from {username='slave'} at 127.0.0.1:57912
I20240503 08:12:43.108750 27855 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet 5c293c45cb5e45bebc486f0418a612ed
I20240503 08:12:43.109088 27982 tablet_service.cc:1514] Processing DeleteTablet for tablet cdc2d9cfc64d42ff9a9b30bb04f4433e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:43 UTC) from {username='slave'} at 127.0.0.1:55610
I20240503 08:12:43.109158 31268 tablet_replica.cc:330] T cdc2d9cfc64d42ff9a9b30bb04f4433e P d09a11a03989421b9f28019dc1c14cc2: stopping tablet replica
I20240503 08:12:43.109200 31267 tablet_replica.cc:330] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 1b411c6bb25549368041773d7b710d48: stopping tablet replica
I20240503 08:12:43.109345 31268 raft_consensus.cc:2229] T cdc2d9cfc64d42ff9a9b30bb04f4433e P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:43.109426 31267 raft_consensus.cc:2229] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:43.109515 27855 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet f9d5fb461eca4d7ca0b2f0e81e9823bc
I20240503 08:12:43.109606 31268 raft_consensus.cc:2258] T cdc2d9cfc64d42ff9a9b30bb04f4433e P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:43.109663 31267 raft_consensus.cc:2258] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:43.110291 31268 ts_tablet_manager.cc:1875] T cdc2d9cfc64d42ff9a9b30bb04f4433e P d09a11a03989421b9f28019dc1c14cc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:43.110376 31267 ts_tablet_manager.cc:1875] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 1b411c6bb25549368041773d7b710d48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:43.110428 28112 tablet_service.cc:1514] Processing DeleteTablet for tablet f9d5fb461eca4d7ca0b2f0e81e9823bc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:43 UTC) from {username='slave'} at 127.0.0.1:57912
I20240503 08:12:43.113895 28242 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c293c45cb5e45bebc486f0418a612ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:43 UTC) from {username='slave'} at 127.0.0.1:41960
I20240503 08:12:43.114171 31409 tablet_replica.cc:330] T 5c293c45cb5e45bebc486f0418a612ed P aae860d7ae2b4002bf7794c278f9cc56: stopping tablet replica
I20240503 08:12:43.114383 31409 raft_consensus.cc:2229] T 5c293c45cb5e45bebc486f0418a612ed P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:43.114516 28372 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c293c45cb5e45bebc486f0418a612ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:43 UTC) from {username='slave'} at 127.0.0.1:34786
I20240503 08:12:43.114583 31267 ts_tablet_manager.cc:1888] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 1b411c6bb25549368041773d7b710d48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:43.114602 31409 raft_consensus.cc:2258] T 5c293c45cb5e45bebc486f0418a612ed P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:43.114691 31267 log.cc:1196] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 1b411c6bb25549368041773d7b710d48: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/cdc2d9cfc64d42ff9a9b30bb04f4433e
I20240503 08:12:43.114917 27981 tablet_service.cc:1514] Processing DeleteTablet for tablet f9d5fb461eca4d7ca0b2f0e81e9823bc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:43 UTC) from {username='slave'} at 127.0.0.1:55610
I20240503 08:12:43.115036 31267 ts_tablet_manager.cc:1909] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 1b411c6bb25549368041773d7b710d48: Deleting consensus metadata
I20240503 08:12:43.115226 28372 tablet_service.cc:1514] Processing DeleteTablet for tablet f9d5fb461eca4d7ca0b2f0e81e9823bc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:43 UTC) from {username='slave'} at 127.0.0.1:34786
I20240503 08:12:43.115322 31409 ts_tablet_manager.cc:1875] T 5c293c45cb5e45bebc486f0418a612ed P aae860d7ae2b4002bf7794c278f9cc56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:43.115787 31267 tablet_replica.cc:330] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 1b411c6bb25549368041773d7b710d48: stopping tablet replica
I20240503 08:12:43.115908 27848 catalog_manager.cc:4830] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): tablet cdc2d9cfc64d42ff9a9b30bb04f4433e (table rebalance_test_table_10 [id=2d7e406e0e184a5faee52035a9d992d7]) successfully deleted
I20240503 08:12:43.115957 31267 raft_consensus.cc:2229] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:43.116171 31268 ts_tablet_manager.cc:1888] T cdc2d9cfc64d42ff9a9b30bb04f4433e P d09a11a03989421b9f28019dc1c14cc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:43.116214 31267 raft_consensus.cc:2258] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:43.116268 31268 log.cc:1196] T cdc2d9cfc64d42ff9a9b30bb04f4433e P d09a11a03989421b9f28019dc1c14cc2: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/wals/cdc2d9cfc64d42ff9a9b30bb04f4433e
I20240503 08:12:43.116664 31268 ts_tablet_manager.cc:1909] T cdc2d9cfc64d42ff9a9b30bb04f4433e P d09a11a03989421b9f28019dc1c14cc2: Deleting consensus metadata
I20240503 08:12:43.116693 31267 ts_tablet_manager.cc:1875] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 1b411c6bb25549368041773d7b710d48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:43.117460 31268 tablet_replica.cc:330] T 5c293c45cb5e45bebc486f0418a612ed P d09a11a03989421b9f28019dc1c14cc2: stopping tablet replica
I20240503 08:12:43.117653 31268 raft_consensus.cc:2229] T 5c293c45cb5e45bebc486f0418a612ed P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:43.117605 27849 catalog_manager.cc:4830] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): tablet cdc2d9cfc64d42ff9a9b30bb04f4433e (table rebalance_test_table_10 [id=2d7e406e0e184a5faee52035a9d992d7]) successfully deleted
I20240503 08:12:43.117913 31511 tablet_replica.cc:330] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:12:43.118005 31268 raft_consensus.cc:2258] T 5c293c45cb5e45bebc486f0418a612ed P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:43.118104 31511 raft_consensus.cc:2229] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:43.118364 27982 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c293c45cb5e45bebc486f0418a612ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:43 UTC) from {username='slave'} at 127.0.0.1:55610
I20240503 08:12:43.118384 31511 raft_consensus.cc:2258] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:43.118609 31268 ts_tablet_manager.cc:1875] T 5c293c45cb5e45bebc486f0418a612ed P d09a11a03989421b9f28019dc1c14cc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:43.119113 31511 ts_tablet_manager.cc:1875] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:43.123509 31409 ts_tablet_manager.cc:1888] T 5c293c45cb5e45bebc486f0418a612ed P aae860d7ae2b4002bf7794c278f9cc56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:43.123672 31409 log.cc:1196] T 5c293c45cb5e45bebc486f0418a612ed P aae860d7ae2b4002bf7794c278f9cc56: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal/wals/5c293c45cb5e45bebc486f0418a612ed
I20240503 08:12:43.124123 31268 ts_tablet_manager.cc:1888] T 5c293c45cb5e45bebc486f0418a612ed P d09a11a03989421b9f28019dc1c14cc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:43.124168 31267 ts_tablet_manager.cc:1888] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 1b411c6bb25549368041773d7b710d48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:43.124194 31409 ts_tablet_manager.cc:1909] T 5c293c45cb5e45bebc486f0418a612ed P aae860d7ae2b4002bf7794c278f9cc56: Deleting consensus metadata
I20240503 08:12:43.124277 31268 log.cc:1196] T 5c293c45cb5e45bebc486f0418a612ed P d09a11a03989421b9f28019dc1c14cc2: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/wals/5c293c45cb5e45bebc486f0418a612ed
I20240503 08:12:43.124629 31268 ts_tablet_manager.cc:1909] T 5c293c45cb5e45bebc486f0418a612ed P d09a11a03989421b9f28019dc1c14cc2: Deleting consensus metadata
I20240503 08:12:43.125106 31267 log.cc:1196] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 1b411c6bb25549368041773d7b710d48: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/f9d5fb461eca4d7ca0b2f0e81e9823bc
I20240503 08:12:43.125491 31268 tablet_replica.cc:330] T f9d5fb461eca4d7ca0b2f0e81e9823bc P d09a11a03989421b9f28019dc1c14cc2: stopping tablet replica
I20240503 08:12:43.125694 31268 raft_consensus.cc:2229] T f9d5fb461eca4d7ca0b2f0e81e9823bc P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:43.125715 31511 ts_tablet_manager.cc:1888] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:43.125747 27849 catalog_manager.cc:4830] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): tablet 5c293c45cb5e45bebc486f0418a612ed (table rebalance_test_table_10 [id=2d7e406e0e184a5faee52035a9d992d7]) successfully deleted
I20240503 08:12:43.125934 31267 ts_tablet_manager.cc:1909] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 1b411c6bb25549368041773d7b710d48: Deleting consensus metadata
I20240503 08:12:43.125941 31268 raft_consensus.cc:2258] T f9d5fb461eca4d7ca0b2f0e81e9823bc P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:43.125960 31511 log.cc:1196] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/cdc2d9cfc64d42ff9a9b30bb04f4433e
I20240503 08:12:43.126252 27847 catalog_manager.cc:4830] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): tablet 5c293c45cb5e45bebc486f0418a612ed (table rebalance_test_table_10 [id=2d7e406e0e184a5faee52035a9d992d7]) successfully deleted
I20240503 08:12:43.126322 31511 ts_tablet_manager.cc:1909] T cdc2d9cfc64d42ff9a9b30bb04f4433e P 833582ff68c94269be364ebd187a75db: Deleting consensus metadata
I20240503 08:12:43.126518 31268 ts_tablet_manager.cc:1875] T f9d5fb461eca4d7ca0b2f0e81e9823bc P d09a11a03989421b9f28019dc1c14cc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:43.126941 27848 catalog_manager.cc:4830] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): tablet f9d5fb461eca4d7ca0b2f0e81e9823bc (table rebalance_test_table_10 [id=2d7e406e0e184a5faee52035a9d992d7]) successfully deleted
I20240503 08:12:43.127167 31511 tablet_replica.cc:330] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:12:43.127274 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet cdc2d9cfc64d42ff9a9b30bb04f4433e (table rebalance_test_table_10 [id=2d7e406e0e184a5faee52035a9d992d7]) successfully deleted
I20240503 08:12:43.127360 31511 raft_consensus.cc:2229] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:43.127574 31511 raft_consensus.cc:2258] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:43.128051 31511 ts_tablet_manager.cc:1875] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:43.130148 31486 consensus_queue.cc:1035] T 8c714f29abb140379b57e08eb927e2b7 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:43.131206 31268 ts_tablet_manager.cc:1888] T f9d5fb461eca4d7ca0b2f0e81e9823bc P d09a11a03989421b9f28019dc1c14cc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:43.131304 31268 log.cc:1196] T f9d5fb461eca4d7ca0b2f0e81e9823bc P d09a11a03989421b9f28019dc1c14cc2: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/wals/f9d5fb461eca4d7ca0b2f0e81e9823bc
I20240503 08:12:43.131672 31268 ts_tablet_manager.cc:1909] T f9d5fb461eca4d7ca0b2f0e81e9823bc P d09a11a03989421b9f28019dc1c14cc2: Deleting consensus metadata
I20240503 08:12:43.132213 31511 ts_tablet_manager.cc:1888] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:43.132367 31511 log.cc:1196] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/f9d5fb461eca4d7ca0b2f0e81e9823bc
I20240503 08:12:43.132687 31511 ts_tablet_manager.cc:1909] T f9d5fb461eca4d7ca0b2f0e81e9823bc P 833582ff68c94269be364ebd187a75db: Deleting consensus metadata
I20240503 08:12:43.132915 27849 catalog_manager.cc:4830] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): tablet f9d5fb461eca4d7ca0b2f0e81e9823bc (table rebalance_test_table_10 [id=2d7e406e0e184a5faee52035a9d992d7]) successfully deleted
I20240503 08:12:43.134438 30652 consensus_queue.cc:1035] T 8c714f29abb140379b57e08eb927e2b7 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:43.135331 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet f9d5fb461eca4d7ca0b2f0e81e9823bc (table rebalance_test_table_10 [id=2d7e406e0e184a5faee52035a9d992d7]) successfully deleted
I20240503 08:12:43.136535 31511 tablet_replica.cc:330] T 5c293c45cb5e45bebc486f0418a612ed P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:12:43.137125 31511 raft_consensus.cc:2229] T 5c293c45cb5e45bebc486f0418a612ed P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:43.137308 31511 raft_consensus.cc:2258] T 5c293c45cb5e45bebc486f0418a612ed P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:43.137933 31511 ts_tablet_manager.cc:1875] T 5c293c45cb5e45bebc486f0418a612ed P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:43.141597 31028 raft_consensus.cc:3037] T d24e437f369b4df49e6363f3f492dc46 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:43.142422 31511 ts_tablet_manager.cc:1888] T 5c293c45cb5e45bebc486f0418a612ed P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:43.142581 31511 log.cc:1196] T 5c293c45cb5e45bebc486f0418a612ed P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/5c293c45cb5e45bebc486f0418a612ed
I20240503 08:12:43.142931 31511 ts_tablet_manager.cc:1909] T 5c293c45cb5e45bebc486f0418a612ed P 833582ff68c94269be364ebd187a75db: Deleting consensus metadata
I20240503 08:12:43.143687 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet 5c293c45cb5e45bebc486f0418a612ed (table rebalance_test_table_10 [id=2d7e406e0e184a5faee52035a9d992d7]) successfully deleted
I20240503 08:12:43.144368 31374 consensus_queue.cc:1035] T d24e437f369b4df49e6363f3f492dc46 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:43.149135 31370 consensus_queue.cc:1035] T d24e437f369b4df49e6363f3f492dc46 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:43.181252 27856 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:43.187963 31533 tablet.cc:1722] T f607fd8558ca4619879e54220c4aa823 P 833582ff68c94269be364ebd187a75db: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:43.188494 31533 tablet.cc:1620] T f607fd8558ca4619879e54220c4aa823 P 833582ff68c94269be364ebd187a75db: MemRowSet was empty: no flush needed.
I20240503 08:12:43.189572 31488 tablet.cc:1722] T f607fd8558ca4619879e54220c4aa823 P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:43.190043 31488 tablet.cc:1620] T f607fd8558ca4619879e54220c4aa823 P ced21ca983b749d0b4d4f6a0e2426bf8: MemRowSet was empty: no flush needed.
I20240503 08:12:43.197407 27856 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=f6a728e264764764a3692179ed402756] alter complete (version 2)
I20240503 08:12:43.204507 31534 tablet.cc:1722] T f607fd8558ca4619879e54220c4aa823 P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:43.204952 31534 tablet.cc:1620] T f607fd8558ca4619879e54220c4aa823 P ce1496a47d084365ad786cd09e472a2c: MemRowSet was empty: no flush needed.
I20240503 08:12:43.241189 27856 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_extra_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:43.248867 31008 tablet_service.cc:1467] Processing CreateTablet for tablet 4ed8026daee840c6b31ca7d989ed25f0 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=a80c6942ff454bcfb6a142a36fb80741]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:43.249176 31007 tablet_service.cc:1467] Processing CreateTablet for tablet e7b54127ad4245838c4cc05071378472 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=a80c6942ff454bcfb6a142a36fb80741]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:43.249373 31008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ed8026daee840c6b31ca7d989ed25f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:43.249568 31007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7b54127ad4245838c4cc05071378472. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:43.249629 30873 tablet_service.cc:1467] Processing CreateTablet for tablet e7b54127ad4245838c4cc05071378472 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=a80c6942ff454bcfb6a142a36fb80741]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:43.250018 30873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7b54127ad4245838c4cc05071378472. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:43.250126 28112 tablet_service.cc:1467] Processing CreateTablet for tablet 4ed8026daee840c6b31ca7d989ed25f0 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=a80c6942ff454bcfb6a142a36fb80741]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:43.250576 28112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ed8026daee840c6b31ca7d989ed25f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:43.253823 30741 tablet_service.cc:1467] Processing CreateTablet for tablet a829381b44c3494ab29ddcf21df2f9e1 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=a80c6942ff454bcfb6a142a36fb80741]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:43.253752 27982 tablet_service.cc:1467] Processing CreateTablet for tablet 4ed8026daee840c6b31ca7d989ed25f0 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=a80c6942ff454bcfb6a142a36fb80741]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:43.254225 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a829381b44c3494ab29ddcf21df2f9e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:43.254308 27982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ed8026daee840c6b31ca7d989ed25f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:43.253901 27981 tablet_service.cc:1467] Processing CreateTablet for tablet e7b54127ad4245838c4cc05071378472 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=a80c6942ff454bcfb6a142a36fb80741]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:43.258224 27981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7b54127ad4245838c4cc05071378472. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:43.258632 31140 tablet_service.cc:1467] Processing CreateTablet for tablet a829381b44c3494ab29ddcf21df2f9e1 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=a80c6942ff454bcfb6a142a36fb80741]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:43.259022 31140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a829381b44c3494ab29ddcf21df2f9e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:43.260104 31494 tablet_bootstrap.cc:492] T 4ed8026daee840c6b31ca7d989ed25f0 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:43.261693 31494 tablet_bootstrap.cc:654] T 4ed8026daee840c6b31ca7d989ed25f0 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:43.262002 31496 tablet_bootstrap.cc:492] T 4ed8026daee840c6b31ca7d989ed25f0 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:43.262351 30874 tablet_service.cc:1467] Processing CreateTablet for tablet a829381b44c3494ab29ddcf21df2f9e1 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=a80c6942ff454bcfb6a142a36fb80741]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:43.262732 30874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a829381b44c3494ab29ddcf21df2f9e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:43.263535 31496 tablet_bootstrap.cc:654] T 4ed8026daee840c6b31ca7d989ed25f0 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:43.267571 31494 tablet_bootstrap.cc:492] T 4ed8026daee840c6b31ca7d989ed25f0 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:43.269109 31494 ts_tablet_manager.cc:1367] T 4ed8026daee840c6b31ca7d989ed25f0 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.003s
I20240503 08:12:43.269934 31494 raft_consensus.cc:348] T 4ed8026daee840c6b31ca7d989ed25f0 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:43.270141 31494 raft_consensus.cc:374] T 4ed8026daee840c6b31ca7d989ed25f0 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:43.270275 31494 raft_consensus.cc:729] T 4ed8026daee840c6b31ca7d989ed25f0 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:43.270542 31494 consensus_queue.cc:260] T 4ed8026daee840c6b31ca7d989ed25f0 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:43.271070 31494 ts_tablet_manager.cc:1398] T 4ed8026daee840c6b31ca7d989ed25f0 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:43.271410 31494 tablet_bootstrap.cc:492] T e7b54127ad4245838c4cc05071378472 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:43.274178 31496 tablet_bootstrap.cc:492] T 4ed8026daee840c6b31ca7d989ed25f0 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:43.274334 31496 ts_tablet_manager.cc:1367] T 4ed8026daee840c6b31ca7d989ed25f0 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.002s
I20240503 08:12:43.274957 31491 tablet_bootstrap.cc:492] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:43.275074 31496 raft_consensus.cc:348] T 4ed8026daee840c6b31ca7d989ed25f0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:43.275236 31544 tablet_bootstrap.cc:492] T a829381b44c3494ab29ddcf21df2f9e1 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:43.275254 31496 raft_consensus.cc:374] T 4ed8026daee840c6b31ca7d989ed25f0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:43.275337 31496 raft_consensus.cc:729] T 4ed8026daee840c6b31ca7d989ed25f0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:43.275555 31496 consensus_queue.cc:260] T 4ed8026daee840c6b31ca7d989ed25f0 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:43.276497 31491 tablet_bootstrap.cc:654] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:43.276674 31544 tablet_bootstrap.cc:654] T a829381b44c3494ab29ddcf21df2f9e1 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:43.276674 31496 ts_tablet_manager.cc:1398] T 4ed8026daee840c6b31ca7d989ed25f0 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:43.276969 31496 tablet_bootstrap.cc:492] T e7b54127ad4245838c4cc05071378472 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:43.278443 31496 tablet_bootstrap.cc:654] T e7b54127ad4245838c4cc05071378472 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:43.282168 31544 tablet_bootstrap.cc:492] T a829381b44c3494ab29ddcf21df2f9e1 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:43.282198 31491 tablet_bootstrap.cc:492] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:43.282367 31491 ts_tablet_manager.cc:1367] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.002s
I20240503 08:12:43.282428 31544 ts_tablet_manager.cc:1367] T a829381b44c3494ab29ddcf21df2f9e1 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.001s
I20240503 08:12:43.282660 31494 tablet_bootstrap.cc:654] T e7b54127ad4245838c4cc05071378472 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:43.283087 31491 raft_consensus.cc:348] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:43.283316 31491 raft_consensus.cc:374] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:43.283380 31491 raft_consensus.cc:729] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:43.283604 31491 consensus_queue.cc:260] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:43.283624 31544 raft_consensus.cc:348] T a829381b44c3494ab29ddcf21df2f9e1 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:43.283814 31544 raft_consensus.cc:374] T a829381b44c3494ab29ddcf21df2f9e1 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:43.283908 31544 raft_consensus.cc:729] T a829381b44c3494ab29ddcf21df2f9e1 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:43.284130 31544 consensus_queue.cc:260] T a829381b44c3494ab29ddcf21df2f9e1 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:43.284951 31544 ts_tablet_manager.cc:1398] T a829381b44c3494ab29ddcf21df2f9e1 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:43.286159 31491 ts_tablet_manager.cc:1398] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 08:12:43.286499 31491 tablet_bootstrap.cc:492] T a829381b44c3494ab29ddcf21df2f9e1 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:43.288483 31496 tablet_bootstrap.cc:492] T e7b54127ad4245838c4cc05071378472 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:43.288702 31496 ts_tablet_manager.cc:1367] T e7b54127ad4245838c4cc05071378472 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.002s
I20240503 08:12:43.289283 31551 tablet_bootstrap.cc:492] T a829381b44c3494ab29ddcf21df2f9e1 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:43.289608 31496 raft_consensus.cc:348] T e7b54127ad4245838c4cc05071378472 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:43.289904 31496 raft_consensus.cc:374] T e7b54127ad4245838c4cc05071378472 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:43.290020 31496 raft_consensus.cc:729] T e7b54127ad4245838c4cc05071378472 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:43.290323 31496 consensus_queue.cc:260] T e7b54127ad4245838c4cc05071378472 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:43.290954 31551 tablet_bootstrap.cc:654] T a829381b44c3494ab29ddcf21df2f9e1 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:43.291100 31491 tablet_bootstrap.cc:654] T a829381b44c3494ab29ddcf21df2f9e1 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:43.294356 31496 ts_tablet_manager.cc:1398] T e7b54127ad4245838c4cc05071378472 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 08:12:43.295969 31491 tablet_bootstrap.cc:492] T a829381b44c3494ab29ddcf21df2f9e1 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:43.298028 31491 ts_tablet_manager.cc:1367] T a829381b44c3494ab29ddcf21df2f9e1 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240503 08:12:43.299167 31551 tablet_bootstrap.cc:492] T a829381b44c3494ab29ddcf21df2f9e1 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:43.299327 31551 ts_tablet_manager.cc:1367] T a829381b44c3494ab29ddcf21df2f9e1 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20240503 08:12:43.299381 31491 raft_consensus.cc:348] T a829381b44c3494ab29ddcf21df2f9e1 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:43.300101 31551 raft_consensus.cc:348] T a829381b44c3494ab29ddcf21df2f9e1 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:43.300321 31551 raft_consensus.cc:374] T a829381b44c3494ab29ddcf21df2f9e1 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:43.300396 31551 raft_consensus.cc:729] T a829381b44c3494ab29ddcf21df2f9e1 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:43.300623 31551 consensus_queue.cc:260] T a829381b44c3494ab29ddcf21df2f9e1 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:43.303036 31491 raft_consensus.cc:374] T a829381b44c3494ab29ddcf21df2f9e1 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:43.303193 31491 raft_consensus.cc:729] T a829381b44c3494ab29ddcf21df2f9e1 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:43.303436 27859 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:43.303480 31491 consensus_queue.cc:260] T a829381b44c3494ab29ddcf21df2f9e1 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:43.303894 31551 ts_tablet_manager.cc:1398] T a829381b44c3494ab29ddcf21df2f9e1 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:43.304939 31491 ts_tablet_manager.cc:1398] T a829381b44c3494ab29ddcf21df2f9e1 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 08:12:43.307238 31494 tablet_bootstrap.cc:492] T e7b54127ad4245838c4cc05071378472 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:43.307420 31494 ts_tablet_manager.cc:1367] T e7b54127ad4245838c4cc05071378472 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.002s
I20240503 08:12:43.307979 31494 raft_consensus.cc:348] T e7b54127ad4245838c4cc05071378472 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:43.308161 31494 raft_consensus.cc:374] T e7b54127ad4245838c4cc05071378472 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:43.308239 31494 raft_consensus.cc:729] T e7b54127ad4245838c4cc05071378472 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:43.308447 31494 consensus_queue.cc:260] T e7b54127ad4245838c4cc05071378472 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:43.308959 31494 ts_tablet_manager.cc:1398] T e7b54127ad4245838c4cc05071378472 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:12:43.311951 31541 tablet_bootstrap.cc:492] T 4ed8026daee840c6b31ca7d989ed25f0 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:43.313665 31541 tablet_bootstrap.cc:654] T 4ed8026daee840c6b31ca7d989ed25f0 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:43.315773 31541 tablet_bootstrap.cc:492] T 4ed8026daee840c6b31ca7d989ed25f0 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:43.315907 31541 ts_tablet_manager.cc:1367] T 4ed8026daee840c6b31ca7d989ed25f0 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:12:43.316660 31541 raft_consensus.cc:348] T 4ed8026daee840c6b31ca7d989ed25f0 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:43.316887 31541 raft_consensus.cc:374] T 4ed8026daee840c6b31ca7d989ed25f0 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:43.317003 31541 raft_consensus.cc:729] T 4ed8026daee840c6b31ca7d989ed25f0 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:43.317255 31541 consensus_queue.cc:260] T 4ed8026daee840c6b31ca7d989ed25f0 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:43.317271 31533 tablet.cc:1722] T f607fd8558ca4619879e54220c4aa823 P 833582ff68c94269be364ebd187a75db: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:43.317711 31541 ts_tablet_manager.cc:1398] T 4ed8026daee840c6b31ca7d989ed25f0 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:43.318049 31534 tablet.cc:1722] T f607fd8558ca4619879e54220c4aa823 P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:43.323071 31488 tablet.cc:1722] T f607fd8558ca4619879e54220c4aa823 P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:43.325544 31509 raft_consensus.cc:482] T a829381b44c3494ab29ddcf21df2f9e1 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:43.325685 31509 raft_consensus.cc:504] T a829381b44c3494ab29ddcf21df2f9e1 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:43.326282 31509 leader_election.cc:288] T a829381b44c3494ab29ddcf21df2f9e1 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:43.326400 31309 raft_consensus.cc:482] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:43.326548 31309 raft_consensus.cc:504] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:43.326622 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a829381b44c3494ab29ddcf21df2f9e1" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:12:43.326674 31161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a829381b44c3494ab29ddcf21df2f9e1" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:12:43.326828 30762 raft_consensus.cc:2454] T a829381b44c3494ab29ddcf21df2f9e1 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:12:43.326861 31161 raft_consensus.cc:2454] T a829381b44c3494ab29ddcf21df2f9e1 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:12:43.327114 31309 leader_election.cc:288] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:12:43.327250 30829 leader_election.cc:302] T a829381b44c3494ab29ddcf21df2f9e1 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:43.327531 31309 raft_consensus.cc:2781] T a829381b44c3494ab29ddcf21df2f9e1 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:43.327651 31309 raft_consensus.cc:482] T a829381b44c3494ab29ddcf21df2f9e1 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:43.327735 31309 raft_consensus.cc:3037] T a829381b44c3494ab29ddcf21df2f9e1 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:43.327724 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7b54127ad4245838c4cc05071378472" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:12:43.327895 28003 raft_consensus.cc:2454] T e7b54127ad4245838c4cc05071378472 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:12:43.328284 30830 leader_election.cc:302] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:43.328614 31509 raft_consensus.cc:2781] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:43.328737 31509 raft_consensus.cc:482] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:43.328824 31509 raft_consensus.cc:3037] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:43.330495 31309 raft_consensus.cc:504] T a829381b44c3494ab29ddcf21df2f9e1 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:43.331445 31309 leader_election.cc:288] T a829381b44c3494ab29ddcf21df2f9e1 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Requested vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:43.331403 31161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a829381b44c3494ab29ddcf21df2f9e1" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:12:43.331343 31509 raft_consensus.cc:504] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:43.331635 31161 raft_consensus.cc:3037] T a829381b44c3494ab29ddcf21df2f9e1 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:43.332125 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a829381b44c3494ab29ddcf21df2f9e1" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:12:43.332151 31028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7b54127ad4245838c4cc05071378472" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:43.332319 30762 raft_consensus.cc:3037] T a829381b44c3494ab29ddcf21df2f9e1 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:43.332363 31028 raft_consensus.cc:2454] T e7b54127ad4245838c4cc05071378472 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:12:43.332423 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7b54127ad4245838c4cc05071378472" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:12:43.332588 31027 raft_consensus.cc:3037] T e7b54127ad4245838c4cc05071378472 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:43.334163 31161 raft_consensus.cc:2454] T a829381b44c3494ab29ddcf21df2f9e1 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:12:43.334342 27856 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table_new_name [id=f6a728e264764764a3692179ed402756] alter complete (version 3)
I20240503 08:12:43.334694 30827 leader_election.cc:302] T a829381b44c3494ab29ddcf21df2f9e1 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:43.334779 30762 raft_consensus.cc:2454] T a829381b44c3494ab29ddcf21df2f9e1 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:12:43.335070 31027 raft_consensus.cc:2454] T e7b54127ad4245838c4cc05071378472 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:12:43.335413 31509 leader_election.cc:288] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Requested vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:12:43.335503 30828 leader_election.cc:302] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:43.335592 31509 raft_consensus.cc:2781] T a829381b44c3494ab29ddcf21df2f9e1 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:43.337342 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7b54127ad4245838c4cc05071378472" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:12:43.337555 28003 raft_consensus.cc:3037] T e7b54127ad4245838c4cc05071378472 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:43.337965 31309 raft_consensus.cc:2781] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:43.340778 28003 raft_consensus.cc:2454] T e7b54127ad4245838c4cc05071378472 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:12:43.341425 31309 raft_consensus.cc:686] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Becoming Leader. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Running, Role: LEADER
I20240503 08:12:43.341727 31309 consensus_queue.cc:237] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:43.342898 31509 raft_consensus.cc:686] T a829381b44c3494ab29ddcf21df2f9e1 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Becoming Leader. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Running, Role: LEADER
I20240503 08:12:43.343148 31509 consensus_queue.cc:237] T a829381b44c3494ab29ddcf21df2f9e1 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:43.345700 27862 catalog_manager.cc:5484] T a829381b44c3494ab29ddcf21df2f9e1 P ce1496a47d084365ad786cd09e472a2c reported cstate change: term changed from 0 to 1, leader changed from <none> to ce1496a47d084365ad786cd09e472a2c (127.20.230.70). New cstate: current_term: 1 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:12:43.349629 27862 catalog_manager.cc:5484] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c reported cstate change: term changed from 0 to 1, leader changed from <none> to ce1496a47d084365ad786cd09e472a2c (127.20.230.70). New cstate: current_term: 1 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:12:43.442159 27855 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:43.442377 27855 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:43.445312 27855 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet f607fd8558ca4619879e54220c4aa823
I20240503 08:12:43.446244 31007 tablet_service.cc:1514] Processing DeleteTablet for tablet f607fd8558ca4619879e54220c4aa823 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:43 UTC) from {username='slave'} at 127.0.0.1:47020
I20240503 08:12:43.446261 27982 tablet_service.cc:1514] Processing DeleteTablet for tablet f607fd8558ca4619879e54220c4aa823 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:43 UTC) from {username='slave'} at 127.0.0.1:55610
I20240503 08:12:43.446593 30874 tablet_service.cc:1514] Processing DeleteTablet for tablet f607fd8558ca4619879e54220c4aa823 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:43 UTC) from {username='slave'} at 127.0.0.1:34426
I20240503 08:12:43.447002 31511 tablet_replica.cc:330] T f607fd8558ca4619879e54220c4aa823 P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:12:43.447237 31511 raft_consensus.cc:2229] T f607fd8558ca4619879e54220c4aa823 P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:43.447603 31511 raft_consensus.cc:2258] T f607fd8558ca4619879e54220c4aa823 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:43.448635 31511 ts_tablet_manager.cc:1875] T f607fd8558ca4619879e54220c4aa823 P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:43.453403 31559 tablet_replica.cc:330] T f607fd8558ca4619879e54220c4aa823 P ced21ca983b749d0b4d4f6a0e2426bf8: stopping tablet replica
I20240503 08:12:43.453531 31560 tablet_replica.cc:330] T f607fd8558ca4619879e54220c4aa823 P ce1496a47d084365ad786cd09e472a2c: stopping tablet replica
I20240503 08:12:43.453624 31559 raft_consensus.cc:2229] T f607fd8558ca4619879e54220c4aa823 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:43.453809 31560 raft_consensus.cc:2229] T f607fd8558ca4619879e54220c4aa823 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:43.453907 31559 raft_consensus.cc:2258] T f607fd8558ca4619879e54220c4aa823 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:43.454102 31560 raft_consensus.cc:2258] T f607fd8558ca4619879e54220c4aa823 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:43.455005 31560 ts_tablet_manager.cc:1875] T f607fd8558ca4619879e54220c4aa823 P ce1496a47d084365ad786cd09e472a2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:43.455181 31559 ts_tablet_manager.cc:1875] T f607fd8558ca4619879e54220c4aa823 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:43.455214 31511 ts_tablet_manager.cc:1888] T f607fd8558ca4619879e54220c4aa823 P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:43.455389 31511 log.cc:1196] T f607fd8558ca4619879e54220c4aa823 P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/f607fd8558ca4619879e54220c4aa823
I20240503 08:12:43.455796 31511 ts_tablet_manager.cc:1909] T f607fd8558ca4619879e54220c4aa823 P 833582ff68c94269be364ebd187a75db: Deleting consensus metadata
I20240503 08:12:43.456856 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet f607fd8558ca4619879e54220c4aa823 (table rebalancer_dynamic_table_new_name [id=f6a728e264764764a3692179ed402756]) successfully deleted
I20240503 08:12:43.460065 31560 ts_tablet_manager.cc:1888] T f607fd8558ca4619879e54220c4aa823 P ce1496a47d084365ad786cd09e472a2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:43.460085 31559 ts_tablet_manager.cc:1888] T f607fd8558ca4619879e54220c4aa823 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:43.460176 31560 log.cc:1196] T f607fd8558ca4619879e54220c4aa823 P ce1496a47d084365ad786cd09e472a2c: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal/wals/f607fd8558ca4619879e54220c4aa823
I20240503 08:12:43.460186 31559 log.cc:1196] T f607fd8558ca4619879e54220c4aa823 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal/wals/f607fd8558ca4619879e54220c4aa823
I20240503 08:12:43.460526 31560 ts_tablet_manager.cc:1909] T f607fd8558ca4619879e54220c4aa823 P ce1496a47d084365ad786cd09e472a2c: Deleting consensus metadata
I20240503 08:12:43.460511 31559 ts_tablet_manager.cc:1909] T f607fd8558ca4619879e54220c4aa823 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting consensus metadata
I20240503 08:12:43.461500 27847 catalog_manager.cc:4830] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): tablet f607fd8558ca4619879e54220c4aa823 (table rebalancer_dynamic_table_new_name [id=f6a728e264764764a3692179ed402756]) successfully deleted
I20240503 08:12:43.461500 27846 catalog_manager.cc:4830] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): tablet f607fd8558ca4619879e54220c4aa823 (table rebalancer_dynamic_table_new_name [id=f6a728e264764764a3692179ed402756]) successfully deleted
I20240503 08:12:43.549470 27855 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:43.556901 31140 tablet_service.cc:1467] Processing CreateTablet for tablet 52014c83a92b420a8651668b9d5df87f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bd215262ff114f8c9f8b510aea62bfcd]), partition=
I20240503 08:12:43.557168 28112 tablet_service.cc:1467] Processing CreateTablet for tablet 52014c83a92b420a8651668b9d5df87f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bd215262ff114f8c9f8b510aea62bfcd]), partition=
I20240503 08:12:43.557201 30874 tablet_service.cc:1467] Processing CreateTablet for tablet 52014c83a92b420a8651668b9d5df87f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bd215262ff114f8c9f8b510aea62bfcd]), partition=
I20240503 08:12:43.557328 31140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52014c83a92b420a8651668b9d5df87f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:43.557477 28112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52014c83a92b420a8651668b9d5df87f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:43.557572 30874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52014c83a92b420a8651668b9d5df87f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:43.561460 31541 tablet_bootstrap.cc:492] T 52014c83a92b420a8651668b9d5df87f P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:43.562728 31541 tablet_bootstrap.cc:654] T 52014c83a92b420a8651668b9d5df87f P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:43.563535 31544 tablet_bootstrap.cc:492] T 52014c83a92b420a8651668b9d5df87f P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:43.563766 31491 tablet_bootstrap.cc:492] T 52014c83a92b420a8651668b9d5df87f P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:43.565066 31544 tablet_bootstrap.cc:654] T 52014c83a92b420a8651668b9d5df87f P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:43.565208 31491 tablet_bootstrap.cc:654] T 52014c83a92b420a8651668b9d5df87f P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:43.566990 31491 tablet_bootstrap.cc:492] T 52014c83a92b420a8651668b9d5df87f P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:43.567134 31491 ts_tablet_manager.cc:1367] T 52014c83a92b420a8651668b9d5df87f P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:43.567683 31491 raft_consensus.cc:348] T 52014c83a92b420a8651668b9d5df87f P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:43.567868 31491 raft_consensus.cc:374] T 52014c83a92b420a8651668b9d5df87f P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:43.568018 31491 raft_consensus.cc:729] T 52014c83a92b420a8651668b9d5df87f P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:43.568303 31491 consensus_queue.cc:260] T 52014c83a92b420a8651668b9d5df87f P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:43.568814 31491 ts_tablet_manager.cc:1398] T 52014c83a92b420a8651668b9d5df87f P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:43.569188 31301 raft_consensus.cc:482] T 4ed8026daee840c6b31ca7d989ed25f0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:43.569371 31301 raft_consensus.cc:504] T 4ed8026daee840c6b31ca7d989ed25f0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:43.570207 31301 leader_election.cc:288] T 4ed8026daee840c6b31ca7d989ed25f0 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:43.572225 31544 tablet_bootstrap.cc:492] T 52014c83a92b420a8651668b9d5df87f P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:43.572391 31544 ts_tablet_manager.cc:1367] T 52014c83a92b420a8651668b9d5df87f P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20240503 08:12:43.573088 31544 raft_consensus.cc:348] T 52014c83a92b420a8651668b9d5df87f P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:43.573319 31544 raft_consensus.cc:374] T 52014c83a92b420a8651668b9d5df87f P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:43.573385 31544 raft_consensus.cc:729] T 52014c83a92b420a8651668b9d5df87f P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:43.573647 31544 consensus_queue.cc:260] T 52014c83a92b420a8651668b9d5df87f P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:43.574498 31541 tablet_bootstrap.cc:492] T 52014c83a92b420a8651668b9d5df87f P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:43.574685 31541 ts_tablet_manager.cc:1367] T 52014c83a92b420a8651668b9d5df87f P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.002s
I20240503 08:12:43.574785 31544 ts_tablet_manager.cc:1398] T 52014c83a92b420a8651668b9d5df87f P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:43.575407 31541 raft_consensus.cc:348] T 52014c83a92b420a8651668b9d5df87f P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:43.575627 31541 raft_consensus.cc:374] T 52014c83a92b420a8651668b9d5df87f P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:43.575696 31541 raft_consensus.cc:729] T 52014c83a92b420a8651668b9d5df87f P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:43.575902 31541 consensus_queue.cc:260] T 52014c83a92b420a8651668b9d5df87f P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:43.576494 31541 ts_tablet_manager.cc:1398] T 52014c83a92b420a8651668b9d5df87f P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:12:43.584350 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed8026daee840c6b31ca7d989ed25f0" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:12:43.584615 28003 raft_consensus.cc:2454] T 4ed8026daee840c6b31ca7d989ed25f0 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 0.
I20240503 08:12:43.584991 30962 leader_election.cc:302] T 4ed8026daee840c6b31ca7d989ed25f0 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:43.585273 31301 raft_consensus.cc:2781] T 4ed8026daee840c6b31ca7d989ed25f0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:43.585384 31301 raft_consensus.cc:482] T 4ed8026daee840c6b31ca7d989ed25f0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:43.585494 31301 raft_consensus.cc:3037] T 4ed8026daee840c6b31ca7d989ed25f0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:43.585987 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed8026daee840c6b31ca7d989ed25f0" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:12:43.586195 28132 raft_consensus.cc:2454] T 4ed8026daee840c6b31ca7d989ed25f0 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 0.
I20240503 08:12:43.587863 31301 raft_consensus.cc:504] T 4ed8026daee840c6b31ca7d989ed25f0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:43.588491 31301 leader_election.cc:288] T 4ed8026daee840c6b31ca7d989ed25f0 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 election: Requested vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:43.588835 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed8026daee840c6b31ca7d989ed25f0" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:12:43.588866 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed8026daee840c6b31ca7d989ed25f0" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:12:43.589006 28003 raft_consensus.cc:3037] T 4ed8026daee840c6b31ca7d989ed25f0 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:43.589038 28132 raft_consensus.cc:3037] T 4ed8026daee840c6b31ca7d989ed25f0 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:43.590782 28132 raft_consensus.cc:2454] T 4ed8026daee840c6b31ca7d989ed25f0 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 1.
I20240503 08:12:43.590898 28003 raft_consensus.cc:2454] T 4ed8026daee840c6b31ca7d989ed25f0 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 1.
I20240503 08:12:43.591125 30961 leader_election.cc:302] T 4ed8026daee840c6b31ca7d989ed25f0 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:43.591429 31301 raft_consensus.cc:2781] T 4ed8026daee840c6b31ca7d989ed25f0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:43.591830 31301 raft_consensus.cc:686] T 4ed8026daee840c6b31ca7d989ed25f0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 LEADER]: Becoming Leader. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Running, Role: LEADER
I20240503 08:12:43.592232 31301 consensus_queue.cc:237] T 4ed8026daee840c6b31ca7d989ed25f0 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:43.595340 27862 catalog_manager.cc:5484] T 4ed8026daee840c6b31ca7d989ed25f0 P ced21ca983b749d0b4d4f6a0e2426bf8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71). New cstate: current_term: 1 leader_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:12:43.610262 31308 raft_consensus.cc:482] T 52014c83a92b420a8651668b9d5df87f P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:43.610448 31308 raft_consensus.cc:504] T 52014c83a92b420a8651668b9d5df87f P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:43.611145 31308 leader_election.cc:288] T 52014c83a92b420a8651668b9d5df87f P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:12:43.613291 30894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52014c83a92b420a8651668b9d5df87f" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:12:43.613574 30894 raft_consensus.cc:2454] T 52014c83a92b420a8651668b9d5df87f P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 0.
I20240503 08:12:43.614001 31094 leader_election.cc:302] T 52014c83a92b420a8651668b9d5df87f P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:43.614360 31308 raft_consensus.cc:2781] T 52014c83a92b420a8651668b9d5df87f P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:43.614457 31308 raft_consensus.cc:482] T 52014c83a92b420a8651668b9d5df87f P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:43.614552 31308 raft_consensus.cc:3037] T 52014c83a92b420a8651668b9d5df87f P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:43.617110 31308 raft_consensus.cc:504] T 52014c83a92b420a8651668b9d5df87f P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:43.617871 31308 leader_election.cc:288] T 52014c83a92b420a8651668b9d5df87f P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 election: Requested vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:12:43.618124 30894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52014c83a92b420a8651668b9d5df87f" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:12:43.618314 30894 raft_consensus.cc:3037] T 52014c83a92b420a8651668b9d5df87f P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:43.618782 27855 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240503 08:12:43.618944 27855 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240503 08:12:43.621024 30894 raft_consensus.cc:2454] T 52014c83a92b420a8651668b9d5df87f P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 1.
I20240503 08:12:43.621392 31094 leader_election.cc:302] T 52014c83a92b420a8651668b9d5df87f P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:43.623955 31308 raft_consensus.cc:2781] T 52014c83a92b420a8651668b9d5df87f P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:43.624384 27855 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet 095d76252eb341c6906963c317224285
I20240503 08:12:43.624598 31308 raft_consensus.cc:686] T 52014c83a92b420a8651668b9d5df87f P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Becoming Leader. State: Replica: b00f244c69df401997ea99d492218a98, State: Running, Role: LEADER
I20240503 08:12:43.624920 27855 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet 333d4108d2ef4a809f5b53304b340899
I20240503 08:12:43.624886 31308 consensus_queue.cc:237] T 52014c83a92b420a8651668b9d5df87f P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:43.625252 27982 tablet_service.cc:1514] Processing DeleteTablet for tablet 095d76252eb341c6906963c317224285 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:43 UTC) from {username='slave'} at 127.0.0.1:55610
I20240503 08:12:43.625347 27855 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet 13cd6e71728040f5a939f212ccb75f0d
I20240503 08:12:43.625430 28112 tablet_service.cc:1514] Processing DeleteTablet for tablet 095d76252eb341c6906963c317224285 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:43 UTC) from {username='slave'} at 127.0.0.1:57912
I20240503 08:12:43.625505 31511 tablet_replica.cc:330] T 095d76252eb341c6906963c317224285 P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:12:43.625747 31511 raft_consensus.cc:2229] T 095d76252eb341c6906963c317224285 P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:43.625898 31267 tablet_replica.cc:330] T 095d76252eb341c6906963c317224285 P 1b411c6bb25549368041773d7b710d48: stopping tablet replica
I20240503 08:12:43.626112 31267 raft_consensus.cc:2229] T 095d76252eb341c6906963c317224285 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:43.626148 31511 raft_consensus.cc:2258] T 095d76252eb341c6906963c317224285 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:43.626353 31267 raft_consensus.cc:2258] T 095d76252eb341c6906963c317224285 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:43.626405 28242 tablet_service.cc:1514] Processing DeleteTablet for tablet 095d76252eb341c6906963c317224285 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:43 UTC) from {username='slave'} at 127.0.0.1:41960
I20240503 08:12:43.626886 31511 ts_tablet_manager.cc:1875] T 095d76252eb341c6906963c317224285 P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:43.626981 31267 ts_tablet_manager.cc:1875] T 095d76252eb341c6906963c317224285 P 1b411c6bb25549368041773d7b710d48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:43.629930 31409 tablet_replica.cc:330] T 095d76252eb341c6906963c317224285 P aae860d7ae2b4002bf7794c278f9cc56: stopping tablet replica
I20240503 08:12:43.629912 28372 tablet_service.cc:1514] Processing DeleteTablet for tablet 333d4108d2ef4a809f5b53304b340899 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:43 UTC) from {username='slave'} at 127.0.0.1:34786
I20240503 08:12:43.630164 31268 tablet_replica.cc:330] T 333d4108d2ef4a809f5b53304b340899 P d09a11a03989421b9f28019dc1c14cc2: stopping tablet replica
I20240503 08:12:43.630146 31409 raft_consensus.cc:2229] T 095d76252eb341c6906963c317224285 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:43.630189 28243 tablet_service.cc:1514] Processing DeleteTablet for tablet 13cd6e71728040f5a939f212ccb75f0d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:43 UTC) from {username='slave'} at 127.0.0.1:41960
I20240503 08:12:43.630380 31409 raft_consensus.cc:2258] T 095d76252eb341c6906963c317224285 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:43.630363 31268 raft_consensus.cc:2229] T 333d4108d2ef4a809f5b53304b340899 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:43.630678 31268 raft_consensus.cc:2258] T 333d4108d2ef4a809f5b53304b340899 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:43.631073 31409 ts_tablet_manager.cc:1875] T 095d76252eb341c6906963c317224285 P aae860d7ae2b4002bf7794c278f9cc56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:43.631249 28372 tablet_service.cc:1514] Processing DeleteTablet for tablet 13cd6e71728040f5a939f212ccb75f0d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:43 UTC) from {username='slave'} at 127.0.0.1:34786
I20240503 08:12:43.631330 31268 ts_tablet_manager.cc:1875] T 333d4108d2ef4a809f5b53304b340899 P d09a11a03989421b9f28019dc1c14cc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:43.631801 27981 tablet_service.cc:1514] Processing DeleteTablet for tablet 13cd6e71728040f5a939f212ccb75f0d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:43 UTC) from {username='slave'} at 127.0.0.1:55610
I20240503 08:12:43.634016 27982 tablet_service.cc:1514] Processing DeleteTablet for tablet 333d4108d2ef4a809f5b53304b340899 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:43 UTC) from {username='slave'} at 127.0.0.1:55610
I20240503 08:12:43.634441 28112 tablet_service.cc:1514] Processing DeleteTablet for tablet 333d4108d2ef4a809f5b53304b340899 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:43 UTC) from {username='slave'} at 127.0.0.1:57912
I20240503 08:12:43.636152 31267 ts_tablet_manager.cc:1888] T 095d76252eb341c6906963c317224285 P 1b411c6bb25549368041773d7b710d48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:43.636312 31267 log.cc:1196] T 095d76252eb341c6906963c317224285 P 1b411c6bb25549368041773d7b710d48: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/095d76252eb341c6906963c317224285
I20240503 08:12:43.636751 31267 ts_tablet_manager.cc:1909] T 095d76252eb341c6906963c317224285 P 1b411c6bb25549368041773d7b710d48: Deleting consensus metadata
I20240503 08:12:43.638518 31267 tablet_replica.cc:330] T 333d4108d2ef4a809f5b53304b340899 P 1b411c6bb25549368041773d7b710d48: stopping tablet replica
I20240503 08:12:43.638504 31268 ts_tablet_manager.cc:1888] T 333d4108d2ef4a809f5b53304b340899 P d09a11a03989421b9f28019dc1c14cc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:43.638592 27848 catalog_manager.cc:4830] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): tablet 095d76252eb341c6906963c317224285 (table rebalance_test_table_12 [id=bce17fe5bc2f48b1b1cada053caa5e61]) successfully deleted
I20240503 08:12:43.638674 31268 log.cc:1196] T 333d4108d2ef4a809f5b53304b340899 P d09a11a03989421b9f28019dc1c14cc2: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/wals/333d4108d2ef4a809f5b53304b340899
I20240503 08:12:43.638844 31267 raft_consensus.cc:2229] T 333d4108d2ef4a809f5b53304b340899 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:43.639065 31267 raft_consensus.cc:2258] T 333d4108d2ef4a809f5b53304b340899 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:43.639206 31268 ts_tablet_manager.cc:1909] T 333d4108d2ef4a809f5b53304b340899 P d09a11a03989421b9f28019dc1c14cc2: Deleting consensus metadata
I20240503 08:12:43.639642 31267 ts_tablet_manager.cc:1875] T 333d4108d2ef4a809f5b53304b340899 P 1b411c6bb25549368041773d7b710d48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:43.640134 31268 tablet_replica.cc:330] T 13cd6e71728040f5a939f212ccb75f0d P d09a11a03989421b9f28019dc1c14cc2: stopping tablet replica
I20240503 08:12:43.640224 27849 catalog_manager.cc:4830] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): tablet 333d4108d2ef4a809f5b53304b340899 (table rebalance_test_table_12 [id=bce17fe5bc2f48b1b1cada053caa5e61]) successfully deleted
I20240503 08:12:43.640404 31268 raft_consensus.cc:2229] T 13cd6e71728040f5a939f212ccb75f0d P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:43.640749 31268 raft_consensus.cc:2258] T 13cd6e71728040f5a939f212ccb75f0d P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:43.641152 31409 ts_tablet_manager.cc:1888] T 095d76252eb341c6906963c317224285 P aae860d7ae2b4002bf7794c278f9cc56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:43.641335 31409 log.cc:1196] T 095d76252eb341c6906963c317224285 P aae860d7ae2b4002bf7794c278f9cc56: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal/wals/095d76252eb341c6906963c317224285
I20240503 08:12:43.641393 31268 ts_tablet_manager.cc:1875] T 13cd6e71728040f5a939f212ccb75f0d P d09a11a03989421b9f28019dc1c14cc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:43.642541 31511 ts_tablet_manager.cc:1888] T 095d76252eb341c6906963c317224285 P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:43.642684 31511 log.cc:1196] T 095d76252eb341c6906963c317224285 P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/095d76252eb341c6906963c317224285
I20240503 08:12:43.643105 31511 ts_tablet_manager.cc:1909] T 095d76252eb341c6906963c317224285 P 833582ff68c94269be364ebd187a75db: Deleting consensus metadata
I20240503 08:12:43.643463 31409 ts_tablet_manager.cc:1909] T 095d76252eb341c6906963c317224285 P aae860d7ae2b4002bf7794c278f9cc56: Deleting consensus metadata
I20240503 08:12:43.644486 31409 tablet_replica.cc:330] T 13cd6e71728040f5a939f212ccb75f0d P aae860d7ae2b4002bf7794c278f9cc56: stopping tablet replica
I20240503 08:12:43.644537 27847 catalog_manager.cc:4830] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): tablet 095d76252eb341c6906963c317224285 (table rebalance_test_table_12 [id=bce17fe5bc2f48b1b1cada053caa5e61]) successfully deleted
I20240503 08:12:43.645311 28131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52014c83a92b420a8651668b9d5df87f" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:12:43.645393 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet 095d76252eb341c6906963c317224285 (table rebalance_test_table_12 [id=bce17fe5bc2f48b1b1cada053caa5e61]) successfully deleted
I20240503 08:12:43.645442 31409 raft_consensus.cc:2229] T 13cd6e71728040f5a939f212ccb75f0d P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:43.645483 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52014c83a92b420a8651668b9d5df87f" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:12:43.645670 31409 raft_consensus.cc:2258] T 13cd6e71728040f5a939f212ccb75f0d P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:43.645530 28131 raft_consensus.cc:3037] T 52014c83a92b420a8651668b9d5df87f P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:43.646327 31409 ts_tablet_manager.cc:1875] T 13cd6e71728040f5a939f212ccb75f0d P aae860d7ae2b4002bf7794c278f9cc56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:43.648810 28131 raft_consensus.cc:2454] T 52014c83a92b420a8651668b9d5df87f P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 1.
I20240503 08:12:43.650152 31511 tablet_replica.cc:330] T 13cd6e71728040f5a939f212ccb75f0d P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:12:43.650388 31511 raft_consensus.cc:2229] T 13cd6e71728040f5a939f212ccb75f0d P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:43.650821 31511 raft_consensus.cc:2258] T 13cd6e71728040f5a939f212ccb75f0d P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:43.651510 31511 ts_tablet_manager.cc:1875] T 13cd6e71728040f5a939f212ccb75f0d P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:43.653939 31267 ts_tablet_manager.cc:1888] T 333d4108d2ef4a809f5b53304b340899 P 1b411c6bb25549368041773d7b710d48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:43.654119 31267 log.cc:1196] T 333d4108d2ef4a809f5b53304b340899 P 1b411c6bb25549368041773d7b710d48: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/333d4108d2ef4a809f5b53304b340899
I20240503 08:12:43.654541 31267 ts_tablet_manager.cc:1909] T 333d4108d2ef4a809f5b53304b340899 P 1b411c6bb25549368041773d7b710d48: Deleting consensus metadata
I20240503 08:12:43.655506 27848 catalog_manager.cc:4830] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): tablet 333d4108d2ef4a809f5b53304b340899 (table rebalance_test_table_12 [id=bce17fe5bc2f48b1b1cada053caa5e61]) successfully deleted
I20240503 08:12:43.658368 31268 ts_tablet_manager.cc:1888] T 13cd6e71728040f5a939f212ccb75f0d P d09a11a03989421b9f28019dc1c14cc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:43.658495 31268 log.cc:1196] T 13cd6e71728040f5a939f212ccb75f0d P d09a11a03989421b9f28019dc1c14cc2: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/wals/13cd6e71728040f5a939f212ccb75f0d
I20240503 08:12:43.658835 31409 ts_tablet_manager.cc:1888] T 13cd6e71728040f5a939f212ccb75f0d P aae860d7ae2b4002bf7794c278f9cc56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:43.658922 31268 ts_tablet_manager.cc:1909] T 13cd6e71728040f5a939f212ccb75f0d P d09a11a03989421b9f28019dc1c14cc2: Deleting consensus metadata
I20240503 08:12:43.658967 31409 log.cc:1196] T 13cd6e71728040f5a939f212ccb75f0d P aae860d7ae2b4002bf7794c278f9cc56: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal/wals/13cd6e71728040f5a939f212ccb75f0d
I20240503 08:12:43.659374 31409 ts_tablet_manager.cc:1909] T 13cd6e71728040f5a939f212ccb75f0d P aae860d7ae2b4002bf7794c278f9cc56: Deleting consensus metadata
I20240503 08:12:43.659950 27849 catalog_manager.cc:4830] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): tablet 13cd6e71728040f5a939f212ccb75f0d (table rebalance_test_table_12 [id=bce17fe5bc2f48b1b1cada053caa5e61]) successfully deleted
I20240503 08:12:43.660334 27847 catalog_manager.cc:4830] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): tablet 13cd6e71728040f5a939f212ccb75f0d (table rebalance_test_table_12 [id=bce17fe5bc2f48b1b1cada053caa5e61]) successfully deleted
I20240503 08:12:43.666458 31511 ts_tablet_manager.cc:1888] T 13cd6e71728040f5a939f212ccb75f0d P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:43.666589 31511 log.cc:1196] T 13cd6e71728040f5a939f212ccb75f0d P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/13cd6e71728040f5a939f212ccb75f0d
I20240503 08:12:43.666982 31511 ts_tablet_manager.cc:1909] T 13cd6e71728040f5a939f212ccb75f0d P 833582ff68c94269be364ebd187a75db: Deleting consensus metadata
I20240503 08:12:43.667793 31511 tablet_replica.cc:330] T 333d4108d2ef4a809f5b53304b340899 P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:12:43.667623 27862 catalog_manager.cc:5484] T 52014c83a92b420a8651668b9d5df87f P b00f244c69df401997ea99d492218a98 reported cstate change: term changed from 0 to 1, leader changed from <none> to b00f244c69df401997ea99d492218a98 (127.20.230.72). New cstate: current_term: 1 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:12:43.667937 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet 13cd6e71728040f5a939f212ccb75f0d (table rebalance_test_table_12 [id=bce17fe5bc2f48b1b1cada053caa5e61]) successfully deleted
I20240503 08:12:43.669278 31511 raft_consensus.cc:2229] T 333d4108d2ef4a809f5b53304b340899 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:43.669517 31511 raft_consensus.cc:2258] T 333d4108d2ef4a809f5b53304b340899 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:43.670183 31511 ts_tablet_manager.cc:1875] T 333d4108d2ef4a809f5b53304b340899 P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:43.674201 31511 ts_tablet_manager.cc:1888] T 333d4108d2ef4a809f5b53304b340899 P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:43.674309 31511 log.cc:1196] T 333d4108d2ef4a809f5b53304b340899 P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/333d4108d2ef4a809f5b53304b340899
I20240503 08:12:43.674582 31511 ts_tablet_manager.cc:1909] T 333d4108d2ef4a809f5b53304b340899 P 833582ff68c94269be364ebd187a75db: Deleting consensus metadata
I20240503 08:12:43.675241 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet 333d4108d2ef4a809f5b53304b340899 (table rebalance_test_table_12 [id=bce17fe5bc2f48b1b1cada053caa5e61]) successfully deleted
I20240503 08:12:43.725317 31509 consensus_queue.cc:1035] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:43.729892 31555 consensus_queue.cc:1035] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:43.740504 31555 consensus_queue.cc:1035] T a829381b44c3494ab29ddcf21df2f9e1 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:43.744576 31555 consensus_queue.cc:1035] T a829381b44c3494ab29ddcf21df2f9e1 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:43.773398 27862 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:43.778613 30894 raft_consensus.cc:1261] T 52014c83a92b420a8651668b9d5df87f P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:43.778659 28131 raft_consensus.cc:1261] T 52014c83a92b420a8651668b9d5df87f P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:43.779163 31548 consensus_queue.cc:1035] T 52014c83a92b420a8651668b9d5df87f P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:43.779417 31308 consensus_queue.cc:1035] T 52014c83a92b420a8651668b9d5df87f P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:43.782732 27862 catalog_manager.cc:5576] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747) does not have the latest schema for tablet 52014c83a92b420a8651668b9d5df87f (table rebalancer_dynamic_table [id=bd215262ff114f8c9f8b510aea62bfcd]). Expected version 1 got 0
I20240503 08:12:43.784421 31581 tablet.cc:1722] T 52014c83a92b420a8651668b9d5df87f P b00f244c69df401997ea99d492218a98: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:43.785216 31581 tablet.cc:1620] T 52014c83a92b420a8651668b9d5df87f P b00f244c69df401997ea99d492218a98: MemRowSet was empty: no flush needed.
I20240503 08:12:43.785161 31534 tablet.cc:1722] T 52014c83a92b420a8651668b9d5df87f P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:43.785564 31534 tablet.cc:1620] T 52014c83a92b420a8651668b9d5df87f P ce1496a47d084365ad786cd09e472a2c: MemRowSet was empty: no flush needed.
I20240503 08:12:43.788512 27862 catalog_manager.cc:5576] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143) does not have the latest schema for tablet 52014c83a92b420a8651668b9d5df87f (table rebalancer_dynamic_table [id=bd215262ff114f8c9f8b510aea62bfcd]). Expected version 1 got 0
I20240503 08:12:43.791930 31583 tablet.cc:1722] T 52014c83a92b420a8651668b9d5df87f P 1b411c6bb25549368041773d7b710d48: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:43.792438 31583 tablet.cc:1620] T 52014c83a92b420a8651668b9d5df87f P 1b411c6bb25549368041773d7b710d48: MemRowSet was empty: no flush needed.
I20240503 08:12:43.794224 27855 catalog_manager.cc:5576] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143) does not have the latest schema for tablet 52014c83a92b420a8651668b9d5df87f (table rebalancer_dynamic_table [id=bd215262ff114f8c9f8b510aea62bfcd]). Expected version 1 got 0
I20240503 08:12:43.797305 27856 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=bd215262ff114f8c9f8b510aea62bfcd] alter complete (version 1)
I20240503 08:12:43.900365 27855 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:43.907449 31581 tablet.cc:1722] T 52014c83a92b420a8651668b9d5df87f P b00f244c69df401997ea99d492218a98: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:43.908289 31581 tablet.cc:1620] T 52014c83a92b420a8651668b9d5df87f P b00f244c69df401997ea99d492218a98: MemRowSet was empty: no flush needed.
I20240503 08:12:43.908447 31583 tablet.cc:1722] T 52014c83a92b420a8651668b9d5df87f P 1b411c6bb25549368041773d7b710d48: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:43.908905 31583 tablet.cc:1620] T 52014c83a92b420a8651668b9d5df87f P 1b411c6bb25549368041773d7b710d48: MemRowSet was empty: no flush needed.
I20240503 08:12:43.909240 31534 tablet.cc:1722] T 52014c83a92b420a8651668b9d5df87f P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:43.909698 31534 tablet.cc:1620] T 52014c83a92b420a8651668b9d5df87f P ce1496a47d084365ad786cd09e472a2c: MemRowSet was empty: no flush needed.
I20240503 08:12:43.916600 27855 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=bd215262ff114f8c9f8b510aea62bfcd] alter complete (version 2)
I20240503 08:12:44.019881 27856 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:44.026290 31581 tablet.cc:1722] T 52014c83a92b420a8651668b9d5df87f P b00f244c69df401997ea99d492218a98: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:44.026849 31534 tablet.cc:1722] T 52014c83a92b420a8651668b9d5df87f P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:44.027227 31583 tablet.cc:1722] T 52014c83a92b420a8651668b9d5df87f P 1b411c6bb25549368041773d7b710d48: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:44.034214 27862 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table_new_name [id=bd215262ff114f8c9f8b510aea62bfcd] alter complete (version 3)
I20240503 08:12:44.080226 31301 consensus_queue.cc:1035] T 4ed8026daee840c6b31ca7d989ed25f0 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:44.085147 31301 consensus_queue.cc:1035] T 4ed8026daee840c6b31ca7d989ed25f0 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:44.131768 27862 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240503 08:12:44.131981 27862 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240503 08:12:44.135329 27862 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet f1f2318ab9a24643a4007dc51769ae9c
I20240503 08:12:44.135797 27862 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet 53af98acddc84bd3b69e00db733379a8
I20240503 08:12:44.136013 28112 tablet_service.cc:1514] Processing DeleteTablet for tablet f1f2318ab9a24643a4007dc51769ae9c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:44 UTC) from {username='slave'} at 127.0.0.1:57912
I20240503 08:12:44.136159 28372 tablet_service.cc:1514] Processing DeleteTablet for tablet f1f2318ab9a24643a4007dc51769ae9c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:44 UTC) from {username='slave'} at 127.0.0.1:34786
I20240503 08:12:44.136282 31267 tablet_replica.cc:330] T f1f2318ab9a24643a4007dc51769ae9c P 1b411c6bb25549368041773d7b710d48: stopping tablet replica
I20240503 08:12:44.136477 31267 raft_consensus.cc:2229] T f1f2318ab9a24643a4007dc51769ae9c P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:44.136461 27982 tablet_service.cc:1514] Processing DeleteTablet for tablet f1f2318ab9a24643a4007dc51769ae9c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:44 UTC) from {username='slave'} at 127.0.0.1:55610
I20240503 08:12:44.136678 31268 tablet_replica.cc:330] T f1f2318ab9a24643a4007dc51769ae9c P d09a11a03989421b9f28019dc1c14cc2: stopping tablet replica
I20240503 08:12:44.136704 31511 tablet_replica.cc:330] T f1f2318ab9a24643a4007dc51769ae9c P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:12:44.136771 31267 raft_consensus.cc:2258] T f1f2318ab9a24643a4007dc51769ae9c P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:44.136878 31511 raft_consensus.cc:2229] T f1f2318ab9a24643a4007dc51769ae9c P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:44.136911 31268 raft_consensus.cc:2229] T f1f2318ab9a24643a4007dc51769ae9c P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:44.137116 31268 raft_consensus.cc:2258] T f1f2318ab9a24643a4007dc51769ae9c P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:44.137112 31511 raft_consensus.cc:2258] T f1f2318ab9a24643a4007dc51769ae9c P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:44.137440 31267 ts_tablet_manager.cc:1875] T f1f2318ab9a24643a4007dc51769ae9c P 1b411c6bb25549368041773d7b710d48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:44.137755 31511 ts_tablet_manager.cc:1875] T f1f2318ab9a24643a4007dc51769ae9c P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:44.137799 31268 ts_tablet_manager.cc:1875] T f1f2318ab9a24643a4007dc51769ae9c P d09a11a03989421b9f28019dc1c14cc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:44.141654 28243 tablet_service.cc:1514] Processing DeleteTablet for tablet 53af98acddc84bd3b69e00db733379a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:44 UTC) from {username='slave'} at 127.0.0.1:41960
I20240503 08:12:44.141757 27862 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet afc92633cbb644be92ab6480743761c1
I20240503 08:12:44.141978 31409 tablet_replica.cc:330] T 53af98acddc84bd3b69e00db733379a8 P aae860d7ae2b4002bf7794c278f9cc56: stopping tablet replica
I20240503 08:12:44.142067 31267 ts_tablet_manager.cc:1888] T f1f2318ab9a24643a4007dc51769ae9c P 1b411c6bb25549368041773d7b710d48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:44.142200 31267 log.cc:1196] T f1f2318ab9a24643a4007dc51769ae9c P 1b411c6bb25549368041773d7b710d48: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/f1f2318ab9a24643a4007dc51769ae9c
I20240503 08:12:44.142199 31409 raft_consensus.cc:2229] T 53af98acddc84bd3b69e00db733379a8 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:44.142452 28112 tablet_service.cc:1514] Processing DeleteTablet for tablet afc92633cbb644be92ab6480743761c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:44 UTC) from {username='slave'} at 127.0.0.1:57912
I20240503 08:12:44.142510 31409 raft_consensus.cc:2258] T 53af98acddc84bd3b69e00db733379a8 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:44.142825 31267 ts_tablet_manager.cc:1909] T f1f2318ab9a24643a4007dc51769ae9c P 1b411c6bb25549368041773d7b710d48: Deleting consensus metadata
I20240503 08:12:44.143052 27982 tablet_service.cc:1514] Processing DeleteTablet for tablet 53af98acddc84bd3b69e00db733379a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:44 UTC) from {username='slave'} at 127.0.0.1:55610
I20240503 08:12:44.143157 31409 ts_tablet_manager.cc:1875] T 53af98acddc84bd3b69e00db733379a8 P aae860d7ae2b4002bf7794c278f9cc56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:44.143893 27848 catalog_manager.cc:4830] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): tablet f1f2318ab9a24643a4007dc51769ae9c (table rebalance_test_table_14 [id=d2bc25b059a84de8a7559b7f9dcebadf]) successfully deleted
I20240503 08:12:44.144024 27856 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:44.144217 27856 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:44.144434 28373 tablet_service.cc:1514] Processing DeleteTablet for tablet afc92633cbb644be92ab6480743761c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:44 UTC) from {username='slave'} at 127.0.0.1:34786
I20240503 08:12:44.144955 31267 tablet_replica.cc:330] T afc92633cbb644be92ab6480743761c1 P 1b411c6bb25549368041773d7b710d48: stopping tablet replica
I20240503 08:12:44.145143 31267 raft_consensus.cc:2229] T afc92633cbb644be92ab6480743761c1 P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:44.145447 31267 raft_consensus.cc:2258] T afc92633cbb644be92ab6480743761c1 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:44.145606 28372 tablet_service.cc:1514] Processing DeleteTablet for tablet 53af98acddc84bd3b69e00db733379a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:44 UTC) from {username='slave'} at 127.0.0.1:34786
I20240503 08:12:44.146051 27982 tablet_service.cc:1514] Processing DeleteTablet for tablet afc92633cbb644be92ab6480743761c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:44 UTC) from {username='slave'} at 127.0.0.1:55610
I20240503 08:12:44.146107 31267 ts_tablet_manager.cc:1875] T afc92633cbb644be92ab6480743761c1 P 1b411c6bb25549368041773d7b710d48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:44.147433 31268 ts_tablet_manager.cc:1888] T f1f2318ab9a24643a4007dc51769ae9c P d09a11a03989421b9f28019dc1c14cc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:44.147624 31268 log.cc:1196] T f1f2318ab9a24643a4007dc51769ae9c P d09a11a03989421b9f28019dc1c14cc2: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/wals/f1f2318ab9a24643a4007dc51769ae9c
I20240503 08:12:44.146840 27862 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_extra_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:44.148036 31268 ts_tablet_manager.cc:1909] T f1f2318ab9a24643a4007dc51769ae9c P d09a11a03989421b9f28019dc1c14cc2: Deleting consensus metadata
I20240503 08:12:44.147498 27856 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet 52014c83a92b420a8651668b9d5df87f
I20240503 08:12:44.148808 31140 tablet_service.cc:1514] Processing DeleteTablet for tablet 52014c83a92b420a8651668b9d5df87f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:44 UTC) from {username='slave'} at 127.0.0.1:45814
I20240503 08:12:44.149150 27849 catalog_manager.cc:4830] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): tablet f1f2318ab9a24643a4007dc51769ae9c (table rebalance_test_table_14 [id=d2bc25b059a84de8a7559b7f9dcebadf]) successfully deleted
I20240503 08:12:44.149452 31268 tablet_replica.cc:330] T afc92633cbb644be92ab6480743761c1 P d09a11a03989421b9f28019dc1c14cc2: stopping tablet replica
I20240503 08:12:44.149641 31268 raft_consensus.cc:2229] T afc92633cbb644be92ab6480743761c1 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:44.149693 28112 tablet_service.cc:1514] Processing DeleteTablet for tablet 52014c83a92b420a8651668b9d5df87f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:44 UTC) from {username='slave'} at 127.0.0.1:57912
I20240503 08:12:44.149884 31511 ts_tablet_manager.cc:1888] T f1f2318ab9a24643a4007dc51769ae9c P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:44.149940 31268 raft_consensus.cc:2258] T afc92633cbb644be92ab6480743761c1 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:44.150022 31511 log.cc:1196] T f1f2318ab9a24643a4007dc51769ae9c P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/f1f2318ab9a24643a4007dc51769ae9c
I20240503 08:12:44.150450 31511 ts_tablet_manager.cc:1909] T f1f2318ab9a24643a4007dc51769ae9c P 833582ff68c94269be364ebd187a75db: Deleting consensus metadata
I20240503 08:12:44.150430 30874 tablet_service.cc:1514] Processing DeleteTablet for tablet 52014c83a92b420a8651668b9d5df87f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:44 UTC) from {username='slave'} at 127.0.0.1:34426
I20240503 08:12:44.150635 31268 ts_tablet_manager.cc:1875] T afc92633cbb644be92ab6480743761c1 P d09a11a03989421b9f28019dc1c14cc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:44.152208 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet f1f2318ab9a24643a4007dc51769ae9c (table rebalance_test_table_14 [id=d2bc25b059a84de8a7559b7f9dcebadf]) successfully deleted
I20240503 08:12:44.152936 31511 tablet_replica.cc:330] T 53af98acddc84bd3b69e00db733379a8 P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:12:44.153033 31267 ts_tablet_manager.cc:1888] T afc92633cbb644be92ab6480743761c1 P 1b411c6bb25549368041773d7b710d48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:44.153160 31267 log.cc:1196] T afc92633cbb644be92ab6480743761c1 P 1b411c6bb25549368041773d7b710d48: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/afc92633cbb644be92ab6480743761c1
I20240503 08:12:44.153205 31511 raft_consensus.cc:2229] T 53af98acddc84bd3b69e00db733379a8 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:44.153501 31511 raft_consensus.cc:2258] T 53af98acddc84bd3b69e00db733379a8 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:44.153635 31267 ts_tablet_manager.cc:1909] T afc92633cbb644be92ab6480743761c1 P 1b411c6bb25549368041773d7b710d48: Deleting consensus metadata
I20240503 08:12:44.154229 31511 ts_tablet_manager.cc:1875] T 53af98acddc84bd3b69e00db733379a8 P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:44.154697 27848 catalog_manager.cc:4830] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): tablet afc92633cbb644be92ab6480743761c1 (table rebalance_test_table_14 [id=d2bc25b059a84de8a7559b7f9dcebadf]) successfully deleted
I20240503 08:12:44.155007 31267 tablet_replica.cc:330] T 52014c83a92b420a8651668b9d5df87f P 1b411c6bb25549368041773d7b710d48: stopping tablet replica
I20240503 08:12:44.155229 31267 raft_consensus.cc:2229] T 52014c83a92b420a8651668b9d5df87f P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:44.155427 31267 raft_consensus.cc:2258] T 52014c83a92b420a8651668b9d5df87f P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:44.155622 31587 tablet_replica.cc:330] T 52014c83a92b420a8651668b9d5df87f P b00f244c69df401997ea99d492218a98: stopping tablet replica
I20240503 08:12:44.156283 31267 ts_tablet_manager.cc:1875] T 52014c83a92b420a8651668b9d5df87f P 1b411c6bb25549368041773d7b710d48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:44.156594 31587 raft_consensus.cc:2229] T 52014c83a92b420a8651668b9d5df87f P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:44.158010 31587 raft_consensus.cc:2258] T 52014c83a92b420a8651668b9d5df87f P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:44.158658 31409 ts_tablet_manager.cc:1888] T 53af98acddc84bd3b69e00db733379a8 P aae860d7ae2b4002bf7794c278f9cc56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:44.158802 31409 log.cc:1196] T 53af98acddc84bd3b69e00db733379a8 P aae860d7ae2b4002bf7794c278f9cc56: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal/wals/53af98acddc84bd3b69e00db733379a8
I20240503 08:12:44.159190 31409 ts_tablet_manager.cc:1909] T 53af98acddc84bd3b69e00db733379a8 P aae860d7ae2b4002bf7794c278f9cc56: Deleting consensus metadata
I20240503 08:12:44.159832 31587 ts_tablet_manager.cc:1875] T 52014c83a92b420a8651668b9d5df87f P b00f244c69df401997ea99d492218a98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:44.160086 27847 catalog_manager.cc:4830] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): tablet 53af98acddc84bd3b69e00db733379a8 (table rebalance_test_table_14 [id=d2bc25b059a84de8a7559b7f9dcebadf]) successfully deleted
I20240503 08:12:44.160596 31511 ts_tablet_manager.cc:1888] T 53af98acddc84bd3b69e00db733379a8 P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:44.160791 31511 log.cc:1196] T 53af98acddc84bd3b69e00db733379a8 P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/53af98acddc84bd3b69e00db733379a8
I20240503 08:12:44.161213 31511 ts_tablet_manager.cc:1909] T 53af98acddc84bd3b69e00db733379a8 P 833582ff68c94269be364ebd187a75db: Deleting consensus metadata
I20240503 08:12:44.162170 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet 53af98acddc84bd3b69e00db733379a8 (table rebalance_test_table_14 [id=d2bc25b059a84de8a7559b7f9dcebadf]) successfully deleted
I20240503 08:12:44.164901 31511 tablet_replica.cc:330] T afc92633cbb644be92ab6480743761c1 P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:12:44.165117 31511 raft_consensus.cc:2229] T afc92633cbb644be92ab6480743761c1 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:44.165400 31511 raft_consensus.cc:2258] T afc92633cbb644be92ab6480743761c1 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:44.167287 31511 ts_tablet_manager.cc:1875] T afc92633cbb644be92ab6480743761c1 P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:44.167929 31268 ts_tablet_manager.cc:1888] T afc92633cbb644be92ab6480743761c1 P d09a11a03989421b9f28019dc1c14cc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:44.168056 31268 log.cc:1196] T afc92633cbb644be92ab6480743761c1 P d09a11a03989421b9f28019dc1c14cc2: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/wals/afc92633cbb644be92ab6480743761c1
I20240503 08:12:44.168479 31267 ts_tablet_manager.cc:1888] T 52014c83a92b420a8651668b9d5df87f P 1b411c6bb25549368041773d7b710d48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:44.168500 31268 ts_tablet_manager.cc:1909] T afc92633cbb644be92ab6480743761c1 P d09a11a03989421b9f28019dc1c14cc2: Deleting consensus metadata
I20240503 08:12:44.168478 31007 tablet_service.cc:1467] Processing CreateTablet for tablet 91c573181144456aa7e27a3026a639c4 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=ae93b0e34a8d4e648e3405fef1bfcbf8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:44.168659 31267 log.cc:1196] T 52014c83a92b420a8651668b9d5df87f P 1b411c6bb25549368041773d7b710d48: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/52014c83a92b420a8651668b9d5df87f
I20240503 08:12:44.168886 31007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91c573181144456aa7e27a3026a639c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.169009 31267 ts_tablet_manager.cc:1909] T 52014c83a92b420a8651668b9d5df87f P 1b411c6bb25549368041773d7b710d48: Deleting consensus metadata
I20240503 08:12:44.169870 28373 tablet_service.cc:1467] Processing CreateTablet for tablet 8938f59aec8447a088108d3f17348436 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=ae93b0e34a8d4e648e3405fef1bfcbf8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:44.170228 28373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8938f59aec8447a088108d3f17348436. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.170116 30741 tablet_service.cc:1467] Processing CreateTablet for tablet 91c573181144456aa7e27a3026a639c4 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=ae93b0e34a8d4e648e3405fef1bfcbf8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:44.170575 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91c573181144456aa7e27a3026a639c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.173503 27849 catalog_manager.cc:4830] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): tablet afc92633cbb644be92ab6480743761c1 (table rebalance_test_table_14 [id=d2bc25b059a84de8a7559b7f9dcebadf]) successfully deleted
I20240503 08:12:44.173817 31268 tablet_replica.cc:330] T 53af98acddc84bd3b69e00db733379a8 P d09a11a03989421b9f28019dc1c14cc2: stopping tablet replica
I20240503 08:12:44.174645 31008 tablet_service.cc:1467] Processing CreateTablet for tablet 393abed63f334ac4b71986573c86c4d4 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=ae93b0e34a8d4e648e3405fef1bfcbf8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:44.175005 31008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 393abed63f334ac4b71986573c86c4d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.175598 28372 tablet_service.cc:1467] Processing CreateTablet for tablet 91c573181144456aa7e27a3026a639c4 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=ae93b0e34a8d4e648e3405fef1bfcbf8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:44.175974 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91c573181144456aa7e27a3026a639c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.177089 31268 raft_consensus.cc:2229] T 53af98acddc84bd3b69e00db733379a8 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:44.177162 27848 catalog_manager.cc:4830] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): tablet 52014c83a92b420a8651668b9d5df87f (table rebalancer_dynamic_table_new_name [id=bd215262ff114f8c9f8b510aea62bfcd]) successfully deleted
I20240503 08:12:44.177323 31268 raft_consensus.cc:2258] T 53af98acddc84bd3b69e00db733379a8 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:44.178028 31268 ts_tablet_manager.cc:1875] T 53af98acddc84bd3b69e00db733379a8 P d09a11a03989421b9f28019dc1c14cc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:44.173718 27982 tablet_service.cc:1467] Processing CreateTablet for tablet 8938f59aec8447a088108d3f17348436 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=ae93b0e34a8d4e648e3405fef1bfcbf8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:44.178484 27982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8938f59aec8447a088108d3f17348436. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.178833 30873 tablet_service.cc:1467] Processing CreateTablet for tablet 8938f59aec8447a088108d3f17348436 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=ae93b0e34a8d4e648e3405fef1bfcbf8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:44.179215 30873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8938f59aec8447a088108d3f17348436. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.182138 31140 tablet_service.cc:1467] Processing CreateTablet for tablet 393abed63f334ac4b71986573c86c4d4 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=ae93b0e34a8d4e648e3405fef1bfcbf8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:44.182606 31140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 393abed63f334ac4b71986573c86c4d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.182265 31587 ts_tablet_manager.cc:1888] T 52014c83a92b420a8651668b9d5df87f P b00f244c69df401997ea99d492218a98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:44.184576 31587 log.cc:1196] T 52014c83a92b420a8651668b9d5df87f P b00f244c69df401997ea99d492218a98: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal/wals/52014c83a92b420a8651668b9d5df87f
I20240503 08:12:44.186043 31587 ts_tablet_manager.cc:1909] T 52014c83a92b420a8651668b9d5df87f P b00f244c69df401997ea99d492218a98: Deleting consensus metadata
I20240503 08:12:44.187115 27846 catalog_manager.cc:4830] TS b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): tablet 52014c83a92b420a8651668b9d5df87f (table rebalancer_dynamic_table_new_name [id=bd215262ff114f8c9f8b510aea62bfcd]) successfully deleted
I20240503 08:12:44.187521 30740 tablet_service.cc:1467] Processing CreateTablet for tablet 393abed63f334ac4b71986573c86c4d4 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=ae93b0e34a8d4e648e3405fef1bfcbf8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:44.187932 30740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 393abed63f334ac4b71986573c86c4d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.189440 31268 ts_tablet_manager.cc:1888] T 53af98acddc84bd3b69e00db733379a8 P d09a11a03989421b9f28019dc1c14cc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:44.189571 31268 log.cc:1196] T 53af98acddc84bd3b69e00db733379a8 P d09a11a03989421b9f28019dc1c14cc2: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/wals/53af98acddc84bd3b69e00db733379a8
I20240503 08:12:44.189999 31268 ts_tablet_manager.cc:1909] T 53af98acddc84bd3b69e00db733379a8 P d09a11a03989421b9f28019dc1c14cc2: Deleting consensus metadata
I20240503 08:12:44.191977 27849 catalog_manager.cc:4830] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): tablet 53af98acddc84bd3b69e00db733379a8 (table rebalance_test_table_14 [id=d2bc25b059a84de8a7559b7f9dcebadf]) successfully deleted
I20240503 08:12:44.193006 31593 tablet_bootstrap.cc:492] T 8938f59aec8447a088108d3f17348436 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:44.194526 31593 tablet_bootstrap.cc:654] T 8938f59aec8447a088108d3f17348436 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.194774 31591 tablet_bootstrap.cc:492] T 8938f59aec8447a088108d3f17348436 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:44.201362 31511 ts_tablet_manager.cc:1888] T afc92633cbb644be92ab6480743761c1 P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:44.201483 31511 log.cc:1196] T afc92633cbb644be92ab6480743761c1 P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/afc92633cbb644be92ab6480743761c1
I20240503 08:12:44.201958 31511 ts_tablet_manager.cc:1909] T afc92633cbb644be92ab6480743761c1 P 833582ff68c94269be364ebd187a75db: Deleting consensus metadata
I20240503 08:12:44.208442 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet afc92633cbb644be92ab6480743761c1 (table rebalance_test_table_14 [id=d2bc25b059a84de8a7559b7f9dcebadf]) successfully deleted
I20240503 08:12:44.214051 31589 tablet_bootstrap.cc:492] T 8938f59aec8447a088108d3f17348436 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:44.214596 31592 tablet_bootstrap.cc:492] T 91c573181144456aa7e27a3026a639c4 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:44.214706 31591 tablet_bootstrap.cc:654] T 8938f59aec8447a088108d3f17348436 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.216295 31592 tablet_bootstrap.cc:654] T 91c573181144456aa7e27a3026a639c4 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.217614 31595 tablet_bootstrap.cc:492] T 393abed63f334ac4b71986573c86c4d4 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:44.219347 31595 tablet_bootstrap.cc:654] T 393abed63f334ac4b71986573c86c4d4 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.221680 31590 tablet_bootstrap.cc:492] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:44.222523 31589 tablet_bootstrap.cc:654] T 8938f59aec8447a088108d3f17348436 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.223398 31590 tablet_bootstrap.cc:654] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.223659 31591 tablet_bootstrap.cc:492] T 8938f59aec8447a088108d3f17348436 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:44.223801 31591 ts_tablet_manager.cc:1367] T 8938f59aec8447a088108d3f17348436 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.000s
I20240503 08:12:44.224212 31588 tablet_replica.cc:330] T 52014c83a92b420a8651668b9d5df87f P ce1496a47d084365ad786cd09e472a2c: stopping tablet replica
I20240503 08:12:44.224509 31588 raft_consensus.cc:2229] T 52014c83a92b420a8651668b9d5df87f P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:44.224596 31591 raft_consensus.cc:348] T 8938f59aec8447a088108d3f17348436 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:44.224884 31588 raft_consensus.cc:2258] T 52014c83a92b420a8651668b9d5df87f P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:44.224910 31591 raft_consensus.cc:374] T 8938f59aec8447a088108d3f17348436 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.224998 31591 raft_consensus.cc:729] T 8938f59aec8447a088108d3f17348436 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.225327 31591 consensus_queue.cc:260] T 8938f59aec8447a088108d3f17348436 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:44.226135 31588 ts_tablet_manager.cc:1875] T 52014c83a92b420a8651668b9d5df87f P ce1496a47d084365ad786cd09e472a2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:44.226248 31592 tablet_bootstrap.cc:492] T 91c573181144456aa7e27a3026a639c4 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:44.226370 31592 ts_tablet_manager.cc:1367] T 91c573181144456aa7e27a3026a639c4 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240503 08:12:44.227780 31595 tablet_bootstrap.cc:492] T 393abed63f334ac4b71986573c86c4d4 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:44.227973 31595 ts_tablet_manager.cc:1367] T 393abed63f334ac4b71986573c86c4d4 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240503 08:12:44.228514 31591 ts_tablet_manager.cc:1398] T 8938f59aec8447a088108d3f17348436 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:12:44.232105 31589 tablet_bootstrap.cc:492] T 8938f59aec8447a088108d3f17348436 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:44.232111 31592 raft_consensus.cc:348] T 91c573181144456aa7e27a3026a639c4 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:44.232344 31589 ts_tablet_manager.cc:1367] T 8938f59aec8447a088108d3f17348436 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.006s
I20240503 08:12:44.229209 31595 raft_consensus.cc:348] T 393abed63f334ac4b71986573c86c4d4 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:44.232648 31592 raft_consensus.cc:374] T 91c573181144456aa7e27a3026a639c4 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.232802 31595 raft_consensus.cc:374] T 393abed63f334ac4b71986573c86c4d4 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.232937 31595 raft_consensus.cc:729] T 393abed63f334ac4b71986573c86c4d4 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.233067 31592 raft_consensus.cc:729] T 91c573181144456aa7e27a3026a639c4 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.233119 31589 raft_consensus.cc:348] T 8938f59aec8447a088108d3f17348436 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:44.233345 31589 raft_consensus.cc:374] T 8938f59aec8447a088108d3f17348436 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.233459 31589 raft_consensus.cc:729] T 8938f59aec8447a088108d3f17348436 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.233487 31592 consensus_queue.cc:260] T 91c573181144456aa7e27a3026a639c4 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:44.233757 31589 consensus_queue.cc:260] T 8938f59aec8447a088108d3f17348436 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:44.233839 31595 consensus_queue.cc:260] T 393abed63f334ac4b71986573c86c4d4 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:44.238596 31588 ts_tablet_manager.cc:1888] T 52014c83a92b420a8651668b9d5df87f P ce1496a47d084365ad786cd09e472a2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:44.238729 31588 log.cc:1196] T 52014c83a92b420a8651668b9d5df87f P ce1496a47d084365ad786cd09e472a2c: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal/wals/52014c83a92b420a8651668b9d5df87f
I20240503 08:12:44.238746 31592 ts_tablet_manager.cc:1398] T 91c573181144456aa7e27a3026a639c4 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240503 08:12:44.238638 31595 ts_tablet_manager.cc:1398] T 393abed63f334ac4b71986573c86c4d4 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 08:12:44.239171 31588 ts_tablet_manager.cc:1909] T 52014c83a92b420a8651668b9d5df87f P ce1496a47d084365ad786cd09e472a2c: Deleting consensus metadata
I20240503 08:12:44.240084 31374 raft_consensus.cc:482] T 8938f59aec8447a088108d3f17348436 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:44.240201 27846 catalog_manager.cc:4830] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): tablet 52014c83a92b420a8651668b9d5df87f (table rebalancer_dynamic_table_new_name [id=bd215262ff114f8c9f8b510aea62bfcd]) successfully deleted
I20240503 08:12:44.240208 31374 raft_consensus.cc:504] T 8938f59aec8447a088108d3f17348436 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:44.240756 31374 leader_election.cc:288] T 8938f59aec8447a088108d3f17348436 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:12:44.241482 30894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8938f59aec8447a088108d3f17348436" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:12:44.241642 31592 tablet_bootstrap.cc:492] T 393abed63f334ac4b71986573c86c4d4 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
W20240503 08:12:44.242144 28326 leader_election.cc:341] T 8938f59aec8447a088108d3f17348436 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:44.243283 31592 tablet_bootstrap.cc:654] T 393abed63f334ac4b71986573c86c4d4 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.245972 31593 tablet_bootstrap.cc:492] T 8938f59aec8447a088108d3f17348436 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:44.246121 31590 tablet_bootstrap.cc:492] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:44.246177 31593 ts_tablet_manager.cc:1367] T 8938f59aec8447a088108d3f17348436 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.053s user 0.009s sys 0.030s
I20240503 08:12:44.246315 31590 ts_tablet_manager.cc:1367] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.000s
I20240503 08:12:44.246945 31593 raft_consensus.cc:348] T 8938f59aec8447a088108d3f17348436 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:44.247195 31593 raft_consensus.cc:374] T 8938f59aec8447a088108d3f17348436 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.247331 31593 raft_consensus.cc:729] T 8938f59aec8447a088108d3f17348436 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.247598 31593 consensus_queue.cc:260] T 8938f59aec8447a088108d3f17348436 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:44.248142 31593 ts_tablet_manager.cc:1398] T 8938f59aec8447a088108d3f17348436 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:44.247332 31590 raft_consensus.cc:348] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:44.249169 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8938f59aec8447a088108d3f17348436" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:12:44.249358 28003 raft_consensus.cc:2454] T 8938f59aec8447a088108d3f17348436 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:12:44.249814 28329 leader_election.cc:302] T 8938f59aec8447a088108d3f17348436 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 833582ff68c94269be364ebd187a75db, d09a11a03989421b9f28019dc1c14cc2; no voters: ce1496a47d084365ad786cd09e472a2c
I20240503 08:12:44.250342 31601 raft_consensus.cc:2781] T 8938f59aec8447a088108d3f17348436 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:44.249275 31590 raft_consensus.cc:374] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.250965 31590 raft_consensus.cc:729] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.250516 31601 raft_consensus.cc:482] T 8938f59aec8447a088108d3f17348436 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:44.251088 31601 raft_consensus.cc:3037] T 8938f59aec8447a088108d3f17348436 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.251298 31590 consensus_queue.cc:260] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:44.253893 31601 raft_consensus.cc:504] T 8938f59aec8447a088108d3f17348436 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:44.255059 31590 ts_tablet_manager.cc:1398] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 08:12:44.255698 31590 tablet_bootstrap.cc:492] T 91c573181144456aa7e27a3026a639c4 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:44.256028 31601 leader_election.cc:288] T 8938f59aec8447a088108d3f17348436 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Requested vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:12:44.256220 30894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8938f59aec8447a088108d3f17348436" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:12:44.256403 30894 raft_consensus.cc:3037] T 8938f59aec8447a088108d3f17348436 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.257056 27859 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:44.258204 31590 tablet_bootstrap.cc:654] T 91c573181144456aa7e27a3026a639c4 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.259828 31590 tablet_bootstrap.cc:492] T 91c573181144456aa7e27a3026a639c4 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:44.259964 31590 ts_tablet_manager.cc:1367] T 91c573181144456aa7e27a3026a639c4 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:44.260501 31590 raft_consensus.cc:348] T 91c573181144456aa7e27a3026a639c4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:44.260638 31590 raft_consensus.cc:374] T 91c573181144456aa7e27a3026a639c4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.260680 31590 raft_consensus.cc:729] T 91c573181144456aa7e27a3026a639c4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.260906 31590 consensus_queue.cc:260] T 91c573181144456aa7e27a3026a639c4 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:44.261303 31590 ts_tablet_manager.cc:1398] T 91c573181144456aa7e27a3026a639c4 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:12:44.262245 30894 raft_consensus.cc:2454] T 8938f59aec8447a088108d3f17348436 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:12:44.263792 28326 leader_election.cc:302] T 8938f59aec8447a088108d3f17348436 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:44.265239 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8938f59aec8447a088108d3f17348436" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:12:44.265461 28003 raft_consensus.cc:3037] T 8938f59aec8447a088108d3f17348436 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.266297 31592 tablet_bootstrap.cc:492] T 393abed63f334ac4b71986573c86c4d4 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:44.266470 31592 ts_tablet_manager.cc:1367] T 393abed63f334ac4b71986573c86c4d4 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.003s
I20240503 08:12:44.266929 31374 raft_consensus.cc:2781] T 8938f59aec8447a088108d3f17348436 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:44.267061 31374 raft_consensus.cc:686] T 8938f59aec8447a088108d3f17348436 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Becoming Leader. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Running, Role: LEADER
I20240503 08:12:44.267235 31592 raft_consensus.cc:348] T 393abed63f334ac4b71986573c86c4d4 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:44.267448 31592 raft_consensus.cc:374] T 393abed63f334ac4b71986573c86c4d4 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.267338 31374 consensus_queue.cc:237] T 8938f59aec8447a088108d3f17348436 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:44.267524 31592 raft_consensus.cc:729] T 393abed63f334ac4b71986573c86c4d4 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.267777 31592 consensus_queue.cc:260] T 393abed63f334ac4b71986573c86c4d4 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:44.268039 28003 raft_consensus.cc:2454] T 8938f59aec8447a088108d3f17348436 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:12:44.268705 30874 tablet_service.cc:1467] Processing CreateTablet for tablet fa55c1cc7d704670b7d39732a8c32997 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=543318f235e9403c9b540323ddf28945]), partition=
I20240503 08:12:44.268975 31008 tablet_service.cc:1467] Processing CreateTablet for tablet fa55c1cc7d704670b7d39732a8c32997 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=543318f235e9403c9b540323ddf28945]), partition=
I20240503 08:12:44.269035 30874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa55c1cc7d704670b7d39732a8c32997. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.269279 31008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa55c1cc7d704670b7d39732a8c32997. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.270021 31589 ts_tablet_manager.cc:1398] T 8938f59aec8447a088108d3f17348436 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.038s user 0.002s sys 0.000s
I20240503 08:12:44.270288 31589 tablet_bootstrap.cc:492] T 91c573181144456aa7e27a3026a639c4 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:44.271834 31589 tablet_bootstrap.cc:654] T 91c573181144456aa7e27a3026a639c4 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.274435 28373 tablet_service.cc:1467] Processing CreateTablet for tablet fa55c1cc7d704670b7d39732a8c32997 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=543318f235e9403c9b540323ddf28945]), partition=
I20240503 08:12:44.274806 28373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa55c1cc7d704670b7d39732a8c32997. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.279345 31593 tablet_bootstrap.cc:492] T fa55c1cc7d704670b7d39732a8c32997 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:44.279779 31590 tablet_bootstrap.cc:492] T fa55c1cc7d704670b7d39732a8c32997 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:44.280815 31593 tablet_bootstrap.cc:654] T fa55c1cc7d704670b7d39732a8c32997 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.281111 31590 tablet_bootstrap.cc:654] T fa55c1cc7d704670b7d39732a8c32997 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.281910 31592 ts_tablet_manager.cc:1398] T 393abed63f334ac4b71986573c86c4d4 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:44.286237 31593 tablet_bootstrap.cc:492] T fa55c1cc7d704670b7d39732a8c32997 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:44.286425 31593 ts_tablet_manager.cc:1367] T fa55c1cc7d704670b7d39732a8c32997 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20240503 08:12:44.287222 31593 raft_consensus.cc:348] T fa55c1cc7d704670b7d39732a8c32997 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:44.287503 31593 raft_consensus.cc:374] T fa55c1cc7d704670b7d39732a8c32997 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.287604 31593 raft_consensus.cc:729] T fa55c1cc7d704670b7d39732a8c32997 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.287897 31593 consensus_queue.cc:260] T fa55c1cc7d704670b7d39732a8c32997 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:44.289974 31593 ts_tablet_manager.cc:1398] T fa55c1cc7d704670b7d39732a8c32997 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 08:12:44.296118 27856 catalog_manager.cc:5484] T 8938f59aec8447a088108d3f17348436 P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68). New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:12:44.297299 31590 tablet_bootstrap.cc:492] T fa55c1cc7d704670b7d39732a8c32997 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:44.297492 31590 ts_tablet_manager.cc:1367] T fa55c1cc7d704670b7d39732a8c32997 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.009s
I20240503 08:12:44.298130 31589 tablet_bootstrap.cc:492] T 91c573181144456aa7e27a3026a639c4 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:44.298353 31589 ts_tablet_manager.cc:1367] T 91c573181144456aa7e27a3026a639c4 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.021s
I20240503 08:12:44.298267 31590 raft_consensus.cc:348] T fa55c1cc7d704670b7d39732a8c32997 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:44.298436 31590 raft_consensus.cc:374] T fa55c1cc7d704670b7d39732a8c32997 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.298576 31590 raft_consensus.cc:729] T fa55c1cc7d704670b7d39732a8c32997 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.298828 31590 consensus_queue.cc:260] T fa55c1cc7d704670b7d39732a8c32997 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:44.299075 31589 raft_consensus.cc:348] T 91c573181144456aa7e27a3026a639c4 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:44.299312 31589 raft_consensus.cc:374] T 91c573181144456aa7e27a3026a639c4 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.299348 31590 ts_tablet_manager.cc:1398] T fa55c1cc7d704670b7d39732a8c32997 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:44.299398 31589 raft_consensus.cc:729] T 91c573181144456aa7e27a3026a639c4 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.299628 31589 consensus_queue.cc:260] T 91c573181144456aa7e27a3026a639c4 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:44.300146 31589 ts_tablet_manager.cc:1398] T 91c573181144456aa7e27a3026a639c4 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:12:44.300443 31589 tablet_bootstrap.cc:492] T fa55c1cc7d704670b7d39732a8c32997 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:44.301915 31589 tablet_bootstrap.cc:654] T fa55c1cc7d704670b7d39732a8c32997 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.305181 31589 tablet_bootstrap.cc:492] T fa55c1cc7d704670b7d39732a8c32997 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:44.305305 31589 ts_tablet_manager.cc:1367] T fa55c1cc7d704670b7d39732a8c32997 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240503 08:12:44.305874 31589 raft_consensus.cc:348] T fa55c1cc7d704670b7d39732a8c32997 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:44.306034 31589 raft_consensus.cc:374] T fa55c1cc7d704670b7d39732a8c32997 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.306174 31589 raft_consensus.cc:729] T fa55c1cc7d704670b7d39732a8c32997 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.306414 31589 consensus_queue.cc:260] T fa55c1cc7d704670b7d39732a8c32997 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:44.306931 31589 ts_tablet_manager.cc:1398] T fa55c1cc7d704670b7d39732a8c32997 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:44.344374 31301 raft_consensus.cc:482] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:44.344534 31301 raft_consensus.cc:504] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:44.345278 31301 leader_election.cc:288] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:44.345700 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "393abed63f334ac4b71986573c86c4d4" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:12:44.345957 30762 raft_consensus.cc:2454] T 393abed63f334ac4b71986573c86c4d4 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 0.
I20240503 08:12:44.346489 30961 leader_election.cc:302] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:44.346803 31301 raft_consensus.cc:2781] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:44.346925 31301 raft_consensus.cc:482] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:44.347029 31301 raft_consensus.cc:3037] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.347501 31287 raft_consensus.cc:482] T 91c573181144456aa7e27a3026a639c4 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:44.347666 31287 raft_consensus.cc:504] T 91c573181144456aa7e27a3026a639c4 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:44.348358 31287 leader_election.cc:288] T 91c573181144456aa7e27a3026a639c4 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:44.349691 31301 raft_consensus.cc:504] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:44.350562 31301 leader_election.cc:288] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 election: Requested vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:44.351030 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "393abed63f334ac4b71986573c86c4d4" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:12:44.351220 30762 raft_consensus.cc:3037] T 393abed63f334ac4b71986573c86c4d4 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.353572 31582 raft_consensus.cc:482] T 393abed63f334ac4b71986573c86c4d4 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:44.353718 31582 raft_consensus.cc:504] T 393abed63f334ac4b71986573c86c4d4 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:44.354149 30762 raft_consensus.cc:2454] T 393abed63f334ac4b71986573c86c4d4 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 1.
I20240503 08:12:44.354501 31582 leader_election.cc:288] T 393abed63f334ac4b71986573c86c4d4 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:12:44.355089 30961 leader_election.cc:302] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:44.355434 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "393abed63f334ac4b71986573c86c4d4" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:44.355690 31027 raft_consensus.cc:2379] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b00f244c69df401997ea99d492218a98 in current term 1: Already voted for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in this term.
I20240503 08:12:44.357704 31161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "393abed63f334ac4b71986573c86c4d4" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:12:44.357823 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "393abed63f334ac4b71986573c86c4d4" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:12:44.357931 31161 raft_consensus.cc:2454] T 393abed63f334ac4b71986573c86c4d4 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 0.
I20240503 08:12:44.360112 31301 raft_consensus.cc:2781] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:44.360855 31301 raft_consensus.cc:686] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 LEADER]: Becoming Leader. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Running, Role: LEADER
I20240503 08:12:44.361616 31301 consensus_queue.cc:237] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:44.362053 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "393abed63f334ac4b71986573c86c4d4" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:12:44.362066 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91c573181144456aa7e27a3026a639c4" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:44.362260 30762 raft_consensus.cc:2379] T 393abed63f334ac4b71986573c86c4d4 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b00f244c69df401997ea99d492218a98 in current term 1: Already voted for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in this term.
I20240503 08:12:44.362241 31027 raft_consensus.cc:2454] T 91c573181144456aa7e27a3026a639c4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:12:44.362622 31096 leader_election.cc:302] T 393abed63f334ac4b71986573c86c4d4 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b00f244c69df401997ea99d492218a98; no voters: 6d1d7de19459435fbde1999e3dfd226b, ced21ca983b749d0b4d4f6a0e2426bf8
I20240503 08:12:44.362887 31582 raft_consensus.cc:3037] T 393abed63f334ac4b71986573c86c4d4 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.363196 30695 leader_election.cc:302] T 91c573181144456aa7e27a3026a639c4 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:44.364760 31582 raft_consensus.cc:2726] T 393abed63f334ac4b71986573c86c4d4 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:12:44.365568 31612 raft_consensus.cc:2781] T 91c573181144456aa7e27a3026a639c4 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:44.365672 31612 raft_consensus.cc:482] T 91c573181144456aa7e27a3026a639c4 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:44.365808 31612 raft_consensus.cc:3037] T 91c573181144456aa7e27a3026a639c4 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.369016 31612 raft_consensus.cc:504] T 91c573181144456aa7e27a3026a639c4 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:44.371095 28393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91c573181144456aa7e27a3026a639c4" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:44.371300 28393 raft_consensus.cc:2454] T 91c573181144456aa7e27a3026a639c4 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:12:44.371351 31612 leader_election.cc:288] T 91c573181144456aa7e27a3026a639c4 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Requested vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:44.371675 28393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91c573181144456aa7e27a3026a639c4" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:12:44.371874 28393 raft_consensus.cc:3037] T 91c573181144456aa7e27a3026a639c4 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.372169 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91c573181144456aa7e27a3026a639c4" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:12:44.372371 31027 raft_consensus.cc:3037] T 91c573181144456aa7e27a3026a639c4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.374027 28393 raft_consensus.cc:2454] T 91c573181144456aa7e27a3026a639c4 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:12:44.374575 30697 leader_election.cc:302] T 91c573181144456aa7e27a3026a639c4 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:44.374497 27859 catalog_manager.cc:5484] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71). New cstate: current_term: 1 leader_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:12:44.374838 31027 raft_consensus.cc:2454] T 91c573181144456aa7e27a3026a639c4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:12:44.374872 31612 raft_consensus.cc:2781] T 91c573181144456aa7e27a3026a639c4 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:44.375308 31612 raft_consensus.cc:686] T 91c573181144456aa7e27a3026a639c4 P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Becoming Leader. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Running, Role: LEADER
I20240503 08:12:44.375562 31612 consensus_queue.cc:237] T 91c573181144456aa7e27a3026a639c4 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:44.378454 27859 catalog_manager.cc:5484] T 91c573181144456aa7e27a3026a639c4 P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69). New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:12:44.480935 31374 raft_consensus.cc:482] T fa55c1cc7d704670b7d39732a8c32997 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:44.481093 31374 raft_consensus.cc:504] T fa55c1cc7d704670b7d39732a8c32997 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:44.481724 31374 leader_election.cc:288] T fa55c1cc7d704670b7d39732a8c32997 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:12:44.482167 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa55c1cc7d704670b7d39732a8c32997" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:44.482393 31027 raft_consensus.cc:2454] T fa55c1cc7d704670b7d39732a8c32997 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:12:44.482383 30894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa55c1cc7d704670b7d39732a8c32997" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:12:44.482646 30894 raft_consensus.cc:2454] T fa55c1cc7d704670b7d39732a8c32997 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:12:44.482805 28327 leader_election.cc:302] T fa55c1cc7d704670b7d39732a8c32997 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:44.483295 31374 raft_consensus.cc:2781] T fa55c1cc7d704670b7d39732a8c32997 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:44.483388 31374 raft_consensus.cc:482] T fa55c1cc7d704670b7d39732a8c32997 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:44.483470 31374 raft_consensus.cc:3037] T fa55c1cc7d704670b7d39732a8c32997 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.485338 31374 raft_consensus.cc:504] T fa55c1cc7d704670b7d39732a8c32997 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:44.486428 31374 leader_election.cc:288] T fa55c1cc7d704670b7d39732a8c32997 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Requested vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:12:44.486677 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa55c1cc7d704670b7d39732a8c32997" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:12:44.486864 31027 raft_consensus.cc:3037] T fa55c1cc7d704670b7d39732a8c32997 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.487051 30894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa55c1cc7d704670b7d39732a8c32997" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:12:44.487315 30894 raft_consensus.cc:3037] T fa55c1cc7d704670b7d39732a8c32997 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.489308 30894 raft_consensus.cc:2454] T fa55c1cc7d704670b7d39732a8c32997 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:12:44.489667 28326 leader_election.cc:302] T fa55c1cc7d704670b7d39732a8c32997 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:44.489750 31027 raft_consensus.cc:2454] T fa55c1cc7d704670b7d39732a8c32997 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:12:44.490005 31374 raft_consensus.cc:2781] T fa55c1cc7d704670b7d39732a8c32997 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:44.490484 31374 raft_consensus.cc:686] T fa55c1cc7d704670b7d39732a8c32997 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Becoming Leader. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Running, Role: LEADER
I20240503 08:12:44.490763 31374 consensus_queue.cc:237] T fa55c1cc7d704670b7d39732a8c32997 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:44.494485 27856 catalog_manager.cc:5484] T fa55c1cc7d704670b7d39732a8c32997 P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68). New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:12:44.617755 27856 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:44.626030 31027 raft_consensus.cc:1261] T fa55c1cc7d704670b7d39732a8c32997 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:44.626622 31602 consensus_queue.cc:1035] T fa55c1cc7d704670b7d39732a8c32997 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:44.627020 30894 raft_consensus.cc:1261] T fa55c1cc7d704670b7d39732a8c32997 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:44.628897 31374 consensus_queue.cc:1035] T fa55c1cc7d704670b7d39732a8c32997 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 08:12:44.634683 27856 catalog_manager.cc:5576] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915) does not have the latest schema for tablet fa55c1cc7d704670b7d39732a8c32997 (table rebalancer_dynamic_table [id=543318f235e9403c9b540323ddf28945]). Expected version 1 got 0
I20240503 08:12:44.636471 31623 tablet.cc:1722] T fa55c1cc7d704670b7d39732a8c32997 P d09a11a03989421b9f28019dc1c14cc2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:44.636801 27856 catalog_manager.cc:5576] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747) does not have the latest schema for tablet fa55c1cc7d704670b7d39732a8c32997 (table rebalancer_dynamic_table [id=543318f235e9403c9b540323ddf28945]). Expected version 1 got 0
I20240503 08:12:44.637890 31623 tablet.cc:1620] T fa55c1cc7d704670b7d39732a8c32997 P d09a11a03989421b9f28019dc1c14cc2: MemRowSet was empty: no flush needed.
I20240503 08:12:44.644515 27856 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240503 08:12:44.644716 27856 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240503 08:12:44.647382 31625 tablet.cc:1722] T fa55c1cc7d704670b7d39732a8c32997 P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:44.647914 31625 tablet.cc:1620] T fa55c1cc7d704670b7d39732a8c32997 P ce1496a47d084365ad786cd09e472a2c: MemRowSet was empty: no flush needed.
I20240503 08:12:44.648384 27855 catalog_manager.cc:5576] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915) does not have the latest schema for tablet fa55c1cc7d704670b7d39732a8c32997 (table rebalancer_dynamic_table [id=543318f235e9403c9b540323ddf28945]). Expected version 1 got 0
I20240503 08:12:44.648869 27856 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet d03de2915a144ce1b57f1765bc7ec6cf
I20240503 08:12:44.649360 27856 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet 366ac74fe5154b15af08bac60648b7b9
I20240503 08:12:44.649721 28112 tablet_service.cc:1514] Processing DeleteTablet for tablet d03de2915a144ce1b57f1765bc7ec6cf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:44 UTC) from {username='slave'} at 127.0.0.1:57912
I20240503 08:12:44.649861 27856 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet 3fd92f707605425c92677dfc5f8358d4
I20240503 08:12:44.650031 31267 tablet_replica.cc:330] T d03de2915a144ce1b57f1765bc7ec6cf P 1b411c6bb25549368041773d7b710d48: stopping tablet replica
I20240503 08:12:44.650230 31267 raft_consensus.cc:2229] T d03de2915a144ce1b57f1765bc7ec6cf P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:44.650570 31267 raft_consensus.cc:2258] T d03de2915a144ce1b57f1765bc7ec6cf P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:44.651274 31267 ts_tablet_manager.cc:1875] T d03de2915a144ce1b57f1765bc7ec6cf P 1b411c6bb25549368041773d7b710d48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:44.651356 28373 tablet_service.cc:1514] Processing DeleteTablet for tablet 366ac74fe5154b15af08bac60648b7b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:44 UTC) from {username='slave'} at 127.0.0.1:34786
I20240503 08:12:44.651647 31268 tablet_replica.cc:330] T 366ac74fe5154b15af08bac60648b7b9 P d09a11a03989421b9f28019dc1c14cc2: stopping tablet replica
I20240503 08:12:44.651818 31268 raft_consensus.cc:2229] T 366ac74fe5154b15af08bac60648b7b9 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:44.652041 31268 raft_consensus.cc:2258] T 366ac74fe5154b15af08bac60648b7b9 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:44.652060 27856 catalog_manager.cc:5576] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747) does not have the latest schema for tablet fa55c1cc7d704670b7d39732a8c32997 (table rebalancer_dynamic_table [id=543318f235e9403c9b540323ddf28945]). Expected version 1 got 0
I20240503 08:12:44.652369 31624 tablet.cc:1722] T fa55c1cc7d704670b7d39732a8c32997 P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:44.652670 31268 ts_tablet_manager.cc:1875] T 366ac74fe5154b15af08bac60648b7b9 P d09a11a03989421b9f28019dc1c14cc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:44.652868 31624 tablet.cc:1620] T fa55c1cc7d704670b7d39732a8c32997 P ced21ca983b749d0b4d4f6a0e2426bf8: MemRowSet was empty: no flush needed.
I20240503 08:12:44.653543 27982 tablet_service.cc:1514] Processing DeleteTablet for tablet d03de2915a144ce1b57f1765bc7ec6cf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:44 UTC) from {username='slave'} at 127.0.0.1:55610
I20240503 08:12:44.653740 27983 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fd92f707605425c92677dfc5f8358d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:44 UTC) from {username='slave'} at 127.0.0.1:55610
I20240503 08:12:44.653877 27981 tablet_service.cc:1514] Processing DeleteTablet for tablet 366ac74fe5154b15af08bac60648b7b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:44 UTC) from {username='slave'} at 127.0.0.1:55610
I20240503 08:12:44.651376 28372 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fd92f707605425c92677dfc5f8358d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:44 UTC) from {username='slave'} at 127.0.0.1:34786
I20240503 08:12:44.654054 28112 tablet_service.cc:1514] Processing DeleteTablet for tablet 366ac74fe5154b15af08bac60648b7b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:44 UTC) from {username='slave'} at 127.0.0.1:57912
I20240503 08:12:44.654106 31511 tablet_replica.cc:330] T d03de2915a144ce1b57f1765bc7ec6cf P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:12:44.654292 31511 raft_consensus.cc:2229] T d03de2915a144ce1b57f1765bc7ec6cf P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:44.654512 31511 raft_consensus.cc:2258] T d03de2915a144ce1b57f1765bc7ec6cf P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:44.655206 31511 ts_tablet_manager.cc:1875] T d03de2915a144ce1b57f1765bc7ec6cf P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:44.655596 27859 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=543318f235e9403c9b540323ddf28945] alter complete (version 1)
I20240503 08:12:44.659897 31268 ts_tablet_manager.cc:1888] T 366ac74fe5154b15af08bac60648b7b9 P d09a11a03989421b9f28019dc1c14cc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:44.660063 31268 log.cc:1196] T 366ac74fe5154b15af08bac60648b7b9 P d09a11a03989421b9f28019dc1c14cc2: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/wals/366ac74fe5154b15af08bac60648b7b9
I20240503 08:12:44.660377 28243 tablet_service.cc:1514] Processing DeleteTablet for tablet d03de2915a144ce1b57f1765bc7ec6cf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:44 UTC) from {username='slave'} at 127.0.0.1:41960
I20240503 08:12:44.660878 31268 ts_tablet_manager.cc:1909] T 366ac74fe5154b15af08bac60648b7b9 P d09a11a03989421b9f28019dc1c14cc2: Deleting consensus metadata
I20240503 08:12:44.660924 31511 ts_tablet_manager.cc:1888] T d03de2915a144ce1b57f1765bc7ec6cf P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:44.661054 31511 log.cc:1196] T d03de2915a144ce1b57f1765bc7ec6cf P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/d03de2915a144ce1b57f1765bc7ec6cf
I20240503 08:12:44.661453 31511 ts_tablet_manager.cc:1909] T d03de2915a144ce1b57f1765bc7ec6cf P 833582ff68c94269be364ebd187a75db: Deleting consensus metadata
I20240503 08:12:44.661798 31268 tablet_replica.cc:330] T 3fd92f707605425c92677dfc5f8358d4 P d09a11a03989421b9f28019dc1c14cc2: stopping tablet replica
I20240503 08:12:44.661998 31268 raft_consensus.cc:2229] T 3fd92f707605425c92677dfc5f8358d4 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:44.662240 31268 raft_consensus.cc:2258] T 3fd92f707605425c92677dfc5f8358d4 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:44.662567 27849 catalog_manager.cc:4830] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): tablet 366ac74fe5154b15af08bac60648b7b9 (table rebalance_test_table_16 [id=8910f6c8c8f340c4a138c3d7438bc73a]) successfully deleted
I20240503 08:12:44.663875 31268 ts_tablet_manager.cc:1875] T 3fd92f707605425c92677dfc5f8358d4 P d09a11a03989421b9f28019dc1c14cc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:44.664278 31511 tablet_replica.cc:330] T 3fd92f707605425c92677dfc5f8358d4 P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:12:44.664456 31511 raft_consensus.cc:2229] T 3fd92f707605425c92677dfc5f8358d4 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:44.664803 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet d03de2915a144ce1b57f1765bc7ec6cf (table rebalance_test_table_16 [id=8910f6c8c8f340c4a138c3d7438bc73a]) successfully deleted
I20240503 08:12:44.665021 31511 raft_consensus.cc:2258] T 3fd92f707605425c92677dfc5f8358d4 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:44.665709 31511 ts_tablet_manager.cc:1875] T 3fd92f707605425c92677dfc5f8358d4 P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:44.666968 28242 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fd92f707605425c92677dfc5f8358d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:44 UTC) from {username='slave'} at 127.0.0.1:41960
I20240503 08:12:44.668012 31267 ts_tablet_manager.cc:1888] T d03de2915a144ce1b57f1765bc7ec6cf P 1b411c6bb25549368041773d7b710d48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:44.668134 31267 log.cc:1196] T d03de2915a144ce1b57f1765bc7ec6cf P 1b411c6bb25549368041773d7b710d48: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/d03de2915a144ce1b57f1765bc7ec6cf
I20240503 08:12:44.668527 31267 ts_tablet_manager.cc:1909] T d03de2915a144ce1b57f1765bc7ec6cf P 1b411c6bb25549368041773d7b710d48: Deleting consensus metadata
I20240503 08:12:44.669369 31267 tablet_replica.cc:330] T 366ac74fe5154b15af08bac60648b7b9 P 1b411c6bb25549368041773d7b710d48: stopping tablet replica
I20240503 08:12:44.669495 27848 catalog_manager.cc:4830] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): tablet d03de2915a144ce1b57f1765bc7ec6cf (table rebalance_test_table_16 [id=8910f6c8c8f340c4a138c3d7438bc73a]) successfully deleted
I20240503 08:12:44.669570 31267 raft_consensus.cc:2229] T 366ac74fe5154b15af08bac60648b7b9 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:44.670977 31268 ts_tablet_manager.cc:1888] T 3fd92f707605425c92677dfc5f8358d4 P d09a11a03989421b9f28019dc1c14cc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:44.671099 31268 log.cc:1196] T 3fd92f707605425c92677dfc5f8358d4 P d09a11a03989421b9f28019dc1c14cc2: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/wals/3fd92f707605425c92677dfc5f8358d4
I20240503 08:12:44.671439 31268 ts_tablet_manager.cc:1909] T 3fd92f707605425c92677dfc5f8358d4 P d09a11a03989421b9f28019dc1c14cc2: Deleting consensus metadata
I20240503 08:12:44.672621 27849 catalog_manager.cc:4830] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): tablet 3fd92f707605425c92677dfc5f8358d4 (table rebalance_test_table_16 [id=8910f6c8c8f340c4a138c3d7438bc73a]) successfully deleted
W20240503 08:12:44.673173 27938 consensus_peers.cc:485] T 366ac74fe5154b15af08bac60648b7b9 P 833582ff68c94269be364ebd187a75db -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:44.673478 31267 raft_consensus.cc:2258] T 366ac74fe5154b15af08bac60648b7b9 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:44.673882 31626 tablet_replica.cc:330] T d03de2915a144ce1b57f1765bc7ec6cf P aae860d7ae2b4002bf7794c278f9cc56: stopping tablet replica
I20240503 08:12:44.674152 31267 ts_tablet_manager.cc:1875] T 366ac74fe5154b15af08bac60648b7b9 P 1b411c6bb25549368041773d7b710d48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:44.674151 31626 raft_consensus.cc:2229] T d03de2915a144ce1b57f1765bc7ec6cf P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:44.674484 31626 raft_consensus.cc:2258] T d03de2915a144ce1b57f1765bc7ec6cf P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:44.675184 31626 ts_tablet_manager.cc:1875] T d03de2915a144ce1b57f1765bc7ec6cf P aae860d7ae2b4002bf7794c278f9cc56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:44.675761 31511 ts_tablet_manager.cc:1888] T 3fd92f707605425c92677dfc5f8358d4 P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:44.675865 31511 log.cc:1196] T 3fd92f707605425c92677dfc5f8358d4 P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/3fd92f707605425c92677dfc5f8358d4
I20240503 08:12:44.676187 31511 ts_tablet_manager.cc:1909] T 3fd92f707605425c92677dfc5f8358d4 P 833582ff68c94269be364ebd187a75db: Deleting consensus metadata
I20240503 08:12:44.676879 31511 tablet_replica.cc:330] T 366ac74fe5154b15af08bac60648b7b9 P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:12:44.677006 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet 3fd92f707605425c92677dfc5f8358d4 (table rebalance_test_table_16 [id=8910f6c8c8f340c4a138c3d7438bc73a]) successfully deleted
I20240503 08:12:44.677063 31511 raft_consensus.cc:2229] T 366ac74fe5154b15af08bac60648b7b9 P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:44.677362 31511 raft_consensus.cc:2258] T 366ac74fe5154b15af08bac60648b7b9 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:44.677987 31511 ts_tablet_manager.cc:1875] T 366ac74fe5154b15af08bac60648b7b9 P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:44.681138 31626 ts_tablet_manager.cc:1888] T d03de2915a144ce1b57f1765bc7ec6cf P aae860d7ae2b4002bf7794c278f9cc56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:44.681260 31626 log.cc:1196] T d03de2915a144ce1b57f1765bc7ec6cf P aae860d7ae2b4002bf7794c278f9cc56: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal/wals/d03de2915a144ce1b57f1765bc7ec6cf
I20240503 08:12:44.681469 31267 ts_tablet_manager.cc:1888] T 366ac74fe5154b15af08bac60648b7b9 P 1b411c6bb25549368041773d7b710d48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:44.681603 31267 log.cc:1196] T 366ac74fe5154b15af08bac60648b7b9 P 1b411c6bb25549368041773d7b710d48: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/366ac74fe5154b15af08bac60648b7b9
I20240503 08:12:44.681716 31626 ts_tablet_manager.cc:1909] T d03de2915a144ce1b57f1765bc7ec6cf P aae860d7ae2b4002bf7794c278f9cc56: Deleting consensus metadata
I20240503 08:12:44.681978 31267 ts_tablet_manager.cc:1909] T 366ac74fe5154b15af08bac60648b7b9 P 1b411c6bb25549368041773d7b710d48: Deleting consensus metadata
I20240503 08:12:44.682526 31626 tablet_replica.cc:330] T 3fd92f707605425c92677dfc5f8358d4 P aae860d7ae2b4002bf7794c278f9cc56: stopping tablet replica
I20240503 08:12:44.682636 27847 catalog_manager.cc:4830] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): tablet d03de2915a144ce1b57f1765bc7ec6cf (table rebalance_test_table_16 [id=8910f6c8c8f340c4a138c3d7438bc73a]) successfully deleted
I20240503 08:12:44.682723 31626 raft_consensus.cc:2229] T 3fd92f707605425c92677dfc5f8358d4 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:44.682888 31511 ts_tablet_manager.cc:1888] T 366ac74fe5154b15af08bac60648b7b9 P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:44.683004 31511 log.cc:1196] T 366ac74fe5154b15af08bac60648b7b9 P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/366ac74fe5154b15af08bac60648b7b9
I20240503 08:12:44.682986 31626 raft_consensus.cc:2258] T 3fd92f707605425c92677dfc5f8358d4 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:44.683305 27848 catalog_manager.cc:4830] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): tablet 366ac74fe5154b15af08bac60648b7b9 (table rebalance_test_table_16 [id=8910f6c8c8f340c4a138c3d7438bc73a]) successfully deleted
I20240503 08:12:44.683419 31511 ts_tablet_manager.cc:1909] T 366ac74fe5154b15af08bac60648b7b9 P 833582ff68c94269be364ebd187a75db: Deleting consensus metadata
I20240503 08:12:44.683516 31626 ts_tablet_manager.cc:1875] T 3fd92f707605425c92677dfc5f8358d4 P aae860d7ae2b4002bf7794c278f9cc56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:44.684211 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet 366ac74fe5154b15af08bac60648b7b9 (table rebalance_test_table_16 [id=8910f6c8c8f340c4a138c3d7438bc73a]) successfully deleted
I20240503 08:12:44.687310 31626 ts_tablet_manager.cc:1888] T 3fd92f707605425c92677dfc5f8358d4 P aae860d7ae2b4002bf7794c278f9cc56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:44.687408 31626 log.cc:1196] T 3fd92f707605425c92677dfc5f8358d4 P aae860d7ae2b4002bf7794c278f9cc56: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal/wals/3fd92f707605425c92677dfc5f8358d4
I20240503 08:12:44.687768 31626 ts_tablet_manager.cc:1909] T 3fd92f707605425c92677dfc5f8358d4 P aae860d7ae2b4002bf7794c278f9cc56: Deleting consensus metadata
I20240503 08:12:44.688450 27847 catalog_manager.cc:4830] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): tablet 3fd92f707605425c92677dfc5f8358d4 (table rebalance_test_table_16 [id=8910f6c8c8f340c4a138c3d7438bc73a]) successfully deleted
I20240503 08:12:44.696871 31602 consensus_queue.cc:1035] T 8938f59aec8447a088108d3f17348436 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:44.700852 31374 consensus_queue.cc:1035] T 8938f59aec8447a088108d3f17348436 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:44.764504 27855 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:44.771286 31623 tablet.cc:1722] T fa55c1cc7d704670b7d39732a8c32997 P d09a11a03989421b9f28019dc1c14cc2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:44.771749 31623 tablet.cc:1620] T fa55c1cc7d704670b7d39732a8c32997 P d09a11a03989421b9f28019dc1c14cc2: MemRowSet was empty: no flush needed.
I20240503 08:12:44.771739 31624 tablet.cc:1722] T fa55c1cc7d704670b7d39732a8c32997 P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:44.772171 31624 tablet.cc:1620] T fa55c1cc7d704670b7d39732a8c32997 P ced21ca983b749d0b4d4f6a0e2426bf8: MemRowSet was empty: no flush needed.
I20240503 08:12:44.772269 31625 tablet.cc:1722] T fa55c1cc7d704670b7d39732a8c32997 P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:44.772691 31625 tablet.cc:1620] T fa55c1cc7d704670b7d39732a8c32997 P ce1496a47d084365ad786cd09e472a2c: MemRowSet was empty: no flush needed.
I20240503 08:12:44.779354 27855 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=543318f235e9403c9b540323ddf28945] alter complete (version 2)
I20240503 08:12:44.799203 31301 consensus_queue.cc:1035] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:44.802860 31632 consensus_queue.cc:1035] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:44.818250 31612 consensus_queue.cc:1035] T 91c573181144456aa7e27a3026a639c4 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:44.818504 31287 consensus_queue.cc:1035] T 91c573181144456aa7e27a3026a639c4 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:44.880869 27859 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:44.887599 31623 tablet.cc:1722] T fa55c1cc7d704670b7d39732a8c32997 P d09a11a03989421b9f28019dc1c14cc2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:44.888168 31625 tablet.cc:1722] T fa55c1cc7d704670b7d39732a8c32997 P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:44.888775 31624 tablet.cc:1722] T fa55c1cc7d704670b7d39732a8c32997 P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:44.895617 27859 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table_new_name [id=543318f235e9403c9b540323ddf28945] alter complete (version 3)
I20240503 08:12:44.916152 27856 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_extra_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:44.924096 31140 tablet_service.cc:1467] Processing CreateTablet for tablet 01c4229dfd25458296e85e412eae13bf (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=9ee47b3ce8574777976ff3cb1ea03e9c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:44.924598 31140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01c4229dfd25458296e85e412eae13bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.924805 30741 tablet_service.cc:1467] Processing CreateTablet for tablet 01c4229dfd25458296e85e412eae13bf (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=9ee47b3ce8574777976ff3cb1ea03e9c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:44.924913 31141 tablet_service.cc:1467] Processing CreateTablet for tablet 7df30eb244914dcabc6965324eced236 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=9ee47b3ce8574777976ff3cb1ea03e9c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:44.925226 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01c4229dfd25458296e85e412eae13bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.925235 31141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7df30eb244914dcabc6965324eced236. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.929369 27981 tablet_service.cc:1467] Processing CreateTablet for tablet 01c4229dfd25458296e85e412eae13bf (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=9ee47b3ce8574777976ff3cb1ea03e9c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:44.929684 27981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01c4229dfd25458296e85e412eae13bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.930159 30874 tablet_service.cc:1467] Processing CreateTablet for tablet 7df30eb244914dcabc6965324eced236 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=9ee47b3ce8574777976ff3cb1ea03e9c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:44.930569 30874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7df30eb244914dcabc6965324eced236. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.931932 31008 tablet_service.cc:1467] Processing CreateTablet for tablet 7df30eb244914dcabc6965324eced236 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=9ee47b3ce8574777976ff3cb1ea03e9c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:44.932282 31639 tablet_bootstrap.cc:492] T 01c4229dfd25458296e85e412eae13bf P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:44.932313 31008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7df30eb244914dcabc6965324eced236. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.933830 31639 tablet_bootstrap.cc:654] T 01c4229dfd25458296e85e412eae13bf P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.934208 31139 tablet_service.cc:1467] Processing CreateTablet for tablet 4a23c8186c4d4b23b62d7474088b45c1 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=9ee47b3ce8574777976ff3cb1ea03e9c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:44.934584 31139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a23c8186c4d4b23b62d7474088b45c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.938082 31007 tablet_service.cc:1467] Processing CreateTablet for tablet 4a23c8186c4d4b23b62d7474088b45c1 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=9ee47b3ce8574777976ff3cb1ea03e9c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:44.938457 31007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a23c8186c4d4b23b62d7474088b45c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.940979 31642 tablet_bootstrap.cc:492] T 01c4229dfd25458296e85e412eae13bf P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:44.942150 28243 tablet_service.cc:1467] Processing CreateTablet for tablet 4a23c8186c4d4b23b62d7474088b45c1 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=9ee47b3ce8574777976ff3cb1ea03e9c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:44.942580 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a23c8186c4d4b23b62d7474088b45c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.944038 31642 tablet_bootstrap.cc:654] T 01c4229dfd25458296e85e412eae13bf P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.946452 31641 tablet_bootstrap.cc:492] T 7df30eb244914dcabc6965324eced236 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:44.947429 31643 tablet_bootstrap.cc:492] T 01c4229dfd25458296e85e412eae13bf P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:44.948839 31641 tablet_bootstrap.cc:654] T 7df30eb244914dcabc6965324eced236 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.950013 31643 tablet_bootstrap.cc:654] T 01c4229dfd25458296e85e412eae13bf P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.950843 31642 tablet_bootstrap.cc:492] T 01c4229dfd25458296e85e412eae13bf P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:44.951433 31642 ts_tablet_manager.cc:1367] T 01c4229dfd25458296e85e412eae13bf P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.003s
I20240503 08:12:44.954209 31646 tablet_bootstrap.cc:492] T 4a23c8186c4d4b23b62d7474088b45c1 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:44.954823 31641 tablet_bootstrap.cc:492] T 7df30eb244914dcabc6965324eced236 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:44.954964 31641 ts_tablet_manager.cc:1367] T 7df30eb244914dcabc6965324eced236 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240503 08:12:44.955104 31642 raft_consensus.cc:348] T 01c4229dfd25458296e85e412eae13bf P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:44.955709 31641 raft_consensus.cc:348] T 7df30eb244914dcabc6965324eced236 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:44.955925 31641 raft_consensus.cc:374] T 7df30eb244914dcabc6965324eced236 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.956007 31641 raft_consensus.cc:729] T 7df30eb244914dcabc6965324eced236 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.956184 31642 raft_consensus.cc:374] T 01c4229dfd25458296e85e412eae13bf P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.956240 31641 consensus_queue.cc:260] T 7df30eb244914dcabc6965324eced236 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:44.956637 31642 raft_consensus.cc:729] T 01c4229dfd25458296e85e412eae13bf P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.957610 31648 tablet_bootstrap.cc:492] T 4a23c8186c4d4b23b62d7474088b45c1 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:44.957787 31641 ts_tablet_manager.cc:1398] T 7df30eb244914dcabc6965324eced236 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:44.959515 31648 tablet_bootstrap.cc:654] T 4a23c8186c4d4b23b62d7474088b45c1 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.959672 31642 consensus_queue.cc:260] T 01c4229dfd25458296e85e412eae13bf P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:44.959968 31643 tablet_bootstrap.cc:492] T 01c4229dfd25458296e85e412eae13bf P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:44.960150 31643 ts_tablet_manager.cc:1367] T 01c4229dfd25458296e85e412eae13bf P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20240503 08:12:44.960177 31642 ts_tablet_manager.cc:1398] T 01c4229dfd25458296e85e412eae13bf P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20240503 08:12:44.961463 31643 raft_consensus.cc:348] T 01c4229dfd25458296e85e412eae13bf P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:44.961750 31643 raft_consensus.cc:374] T 01c4229dfd25458296e85e412eae13bf P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.962272 31643 raft_consensus.cc:729] T 01c4229dfd25458296e85e412eae13bf P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.962549 31643 consensus_queue.cc:260] T 01c4229dfd25458296e85e412eae13bf P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:44.962976 31646 tablet_bootstrap.cc:654] T 4a23c8186c4d4b23b62d7474088b45c1 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.963156 31643 ts_tablet_manager.cc:1398] T 01c4229dfd25458296e85e412eae13bf P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:12:44.963961 31639 tablet_bootstrap.cc:492] T 01c4229dfd25458296e85e412eae13bf P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:44.964116 31639 ts_tablet_manager.cc:1367] T 01c4229dfd25458296e85e412eae13bf P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.032s user 0.001s sys 0.014s
I20240503 08:12:44.964880 31639 raft_consensus.cc:348] T 01c4229dfd25458296e85e412eae13bf P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:44.965071 31639 raft_consensus.cc:374] T 01c4229dfd25458296e85e412eae13bf P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.965165 31639 raft_consensus.cc:729] T 01c4229dfd25458296e85e412eae13bf P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.965365 31639 consensus_queue.cc:260] T 01c4229dfd25458296e85e412eae13bf P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:44.965936 31639 ts_tablet_manager.cc:1398] T 01c4229dfd25458296e85e412eae13bf P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:44.966570 31639 tablet_bootstrap.cc:492] T 7df30eb244914dcabc6965324eced236 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:44.968633 31639 tablet_bootstrap.cc:654] T 7df30eb244914dcabc6965324eced236 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.973223 31646 tablet_bootstrap.cc:492] T 4a23c8186c4d4b23b62d7474088b45c1 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:44.973423 31646 ts_tablet_manager.cc:1367] T 4a23c8186c4d4b23b62d7474088b45c1 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.006s
I20240503 08:12:44.973453 31648 tablet_bootstrap.cc:492] T 4a23c8186c4d4b23b62d7474088b45c1 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:44.973642 31648 ts_tablet_manager.cc:1367] T 4a23c8186c4d4b23b62d7474088b45c1 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.002s
I20240503 08:12:44.974234 31646 raft_consensus.cc:348] T 4a23c8186c4d4b23b62d7474088b45c1 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:44.974514 31646 raft_consensus.cc:374] T 4a23c8186c4d4b23b62d7474088b45c1 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.974537 31648 raft_consensus.cc:348] T 4a23c8186c4d4b23b62d7474088b45c1 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:44.974660 31646 raft_consensus.cc:729] T 4a23c8186c4d4b23b62d7474088b45c1 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.974787 31648 raft_consensus.cc:374] T 4a23c8186c4d4b23b62d7474088b45c1 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.974913 31648 raft_consensus.cc:729] T 4a23c8186c4d4b23b62d7474088b45c1 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.974969 31646 consensus_queue.cc:260] T 4a23c8186c4d4b23b62d7474088b45c1 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:44.975200 31648 consensus_queue.cc:260] T 4a23c8186c4d4b23b62d7474088b45c1 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:44.975731 31646 ts_tablet_manager.cc:1398] T 4a23c8186c4d4b23b62d7474088b45c1 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:44.975746 31648 ts_tablet_manager.cc:1398] T 4a23c8186c4d4b23b62d7474088b45c1 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:44.975994 31648 tablet_bootstrap.cc:492] T 7df30eb244914dcabc6965324eced236 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:44.976673 31639 tablet_bootstrap.cc:492] T 7df30eb244914dcabc6965324eced236 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:44.976872 31639 ts_tablet_manager.cc:1367] T 7df30eb244914dcabc6965324eced236 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.002s
I20240503 08:12:44.977501 31648 tablet_bootstrap.cc:654] T 7df30eb244914dcabc6965324eced236 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.977612 31639 raft_consensus.cc:348] T 7df30eb244914dcabc6965324eced236 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:44.977839 31639 raft_consensus.cc:374] T 7df30eb244914dcabc6965324eced236 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.977972 31639 raft_consensus.cc:729] T 7df30eb244914dcabc6965324eced236 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.978228 31639 consensus_queue.cc:260] T 7df30eb244914dcabc6965324eced236 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:44.978883 31639 ts_tablet_manager.cc:1398] T 7df30eb244914dcabc6965324eced236 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:12:44.979676 31639 tablet_bootstrap.cc:492] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:44.981137 31639 tablet_bootstrap.cc:654] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.990847 31648 tablet_bootstrap.cc:492] T 7df30eb244914dcabc6965324eced236 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:44.991004 31648 ts_tablet_manager.cc:1367] T 7df30eb244914dcabc6965324eced236 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.003s
I20240503 08:12:44.991616 31648 raft_consensus.cc:348] T 7df30eb244914dcabc6965324eced236 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:44.991842 31648 raft_consensus.cc:374] T 7df30eb244914dcabc6965324eced236 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.991966 31648 raft_consensus.cc:729] T 7df30eb244914dcabc6965324eced236 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.992208 31648 consensus_queue.cc:260] T 7df30eb244914dcabc6965324eced236 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:44.992697 31648 ts_tablet_manager.cc:1398] T 7df30eb244914dcabc6965324eced236 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:44.993307 31639 tablet_bootstrap.cc:492] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:44.993471 31639 ts_tablet_manager.cc:1367] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.011s
I20240503 08:12:44.994891 31639 raft_consensus.cc:348] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:44.995102 31639 raft_consensus.cc:374] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.995263 31639 raft_consensus.cc:729] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.995463 31639 consensus_queue.cc:260] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:44.995905 31639 ts_tablet_manager.cc:1398] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:12:44.998665 27859 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:44.998893 27859 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:45.001510 27859 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet fa55c1cc7d704670b7d39732a8c32997
I20240503 08:12:45.002174 31007 tablet_service.cc:1514] Processing DeleteTablet for tablet fa55c1cc7d704670b7d39732a8c32997 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:44 UTC) from {username='slave'} at 127.0.0.1:47020
I20240503 08:12:45.002297 30874 tablet_service.cc:1514] Processing DeleteTablet for tablet fa55c1cc7d704670b7d39732a8c32997 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:44 UTC) from {username='slave'} at 127.0.0.1:34426
I20240503 08:12:45.002889 31655 tablet_replica.cc:330] T fa55c1cc7d704670b7d39732a8c32997 P ce1496a47d084365ad786cd09e472a2c: stopping tablet replica
I20240503 08:12:45.003232 31655 raft_consensus.cc:2229] T fa55c1cc7d704670b7d39732a8c32997 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:45.003304 28372 tablet_service.cc:1514] Processing DeleteTablet for tablet fa55c1cc7d704670b7d39732a8c32997 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:44 UTC) from {username='slave'} at 127.0.0.1:34786
I20240503 08:12:45.003544 31655 raft_consensus.cc:2258] T fa55c1cc7d704670b7d39732a8c32997 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:45.003625 31268 tablet_replica.cc:330] T fa55c1cc7d704670b7d39732a8c32997 P d09a11a03989421b9f28019dc1c14cc2: stopping tablet replica
I20240503 08:12:45.003827 31268 raft_consensus.cc:2229] T fa55c1cc7d704670b7d39732a8c32997 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:45.004196 31268 raft_consensus.cc:2258] T fa55c1cc7d704670b7d39732a8c32997 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:45.004637 31655 ts_tablet_manager.cc:1875] T fa55c1cc7d704670b7d39732a8c32997 P ce1496a47d084365ad786cd09e472a2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:45.005213 31268 ts_tablet_manager.cc:1875] T fa55c1cc7d704670b7d39732a8c32997 P d09a11a03989421b9f28019dc1c14cc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:45.010167 31654 tablet_replica.cc:330] T fa55c1cc7d704670b7d39732a8c32997 P ced21ca983b749d0b4d4f6a0e2426bf8: stopping tablet replica
I20240503 08:12:45.010351 31654 raft_consensus.cc:2229] T fa55c1cc7d704670b7d39732a8c32997 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:45.010589 31654 raft_consensus.cc:2258] T fa55c1cc7d704670b7d39732a8c32997 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:45.010756 31655 ts_tablet_manager.cc:1888] T fa55c1cc7d704670b7d39732a8c32997 P ce1496a47d084365ad786cd09e472a2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:45.010901 31655 log.cc:1196] T fa55c1cc7d704670b7d39732a8c32997 P ce1496a47d084365ad786cd09e472a2c: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal/wals/fa55c1cc7d704670b7d39732a8c32997
I20240503 08:12:45.011394 31655 ts_tablet_manager.cc:1909] T fa55c1cc7d704670b7d39732a8c32997 P ce1496a47d084365ad786cd09e472a2c: Deleting consensus metadata
I20240503 08:12:45.013021 27846 catalog_manager.cc:4830] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): tablet fa55c1cc7d704670b7d39732a8c32997 (table rebalancer_dynamic_table_new_name [id=543318f235e9403c9b540323ddf28945]) successfully deleted
I20240503 08:12:45.013746 31268 ts_tablet_manager.cc:1888] T fa55c1cc7d704670b7d39732a8c32997 P d09a11a03989421b9f28019dc1c14cc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:45.013890 31268 log.cc:1196] T fa55c1cc7d704670b7d39732a8c32997 P d09a11a03989421b9f28019dc1c14cc2: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/wals/fa55c1cc7d704670b7d39732a8c32997
I20240503 08:12:45.014204 31268 ts_tablet_manager.cc:1909] T fa55c1cc7d704670b7d39732a8c32997 P d09a11a03989421b9f28019dc1c14cc2: Deleting consensus metadata
I20240503 08:12:45.014267 31654 ts_tablet_manager.cc:1875] T fa55c1cc7d704670b7d39732a8c32997 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:45.015142 27849 catalog_manager.cc:4830] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): tablet fa55c1cc7d704670b7d39732a8c32997 (table rebalancer_dynamic_table_new_name [id=543318f235e9403c9b540323ddf28945]) successfully deleted
I20240503 08:12:45.018604 31654 ts_tablet_manager.cc:1888] T fa55c1cc7d704670b7d39732a8c32997 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:45.018709 31654 log.cc:1196] T fa55c1cc7d704670b7d39732a8c32997 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal/wals/fa55c1cc7d704670b7d39732a8c32997
I20240503 08:12:45.019022 31654 ts_tablet_manager.cc:1909] T fa55c1cc7d704670b7d39732a8c32997 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting consensus metadata
I20240503 08:12:45.019801 27847 catalog_manager.cc:4830] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): tablet fa55c1cc7d704670b7d39732a8c32997 (table rebalancer_dynamic_table_new_name [id=543318f235e9403c9b540323ddf28945]) successfully deleted
I20240503 08:12:45.061604 31632 raft_consensus.cc:482] T 7df30eb244914dcabc6965324eced236 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:45.061792 31632 raft_consensus.cc:504] T 7df30eb244914dcabc6965324eced236 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:45.062458 31632 leader_election.cc:288] T 7df30eb244914dcabc6965324eced236 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:45.062930 30894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7df30eb244914dcabc6965324eced236" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:12:45.063048 31161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7df30eb244914dcabc6965324eced236" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:12:45.063145 30894 raft_consensus.cc:2454] T 7df30eb244914dcabc6965324eced236 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 0.
I20240503 08:12:45.063305 31161 raft_consensus.cc:2454] T 7df30eb244914dcabc6965324eced236 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 0.
I20240503 08:12:45.063490 30959 leader_election.cc:302] T 7df30eb244914dcabc6965324eced236 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:45.063804 31632 raft_consensus.cc:2781] T 7df30eb244914dcabc6965324eced236 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:45.063906 31632 raft_consensus.cc:482] T 7df30eb244914dcabc6965324eced236 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:45.063997 31632 raft_consensus.cc:3037] T 7df30eb244914dcabc6965324eced236 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:45.065727 31632 raft_consensus.cc:504] T 7df30eb244914dcabc6965324eced236 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:45.066254 31632 leader_election.cc:288] T 7df30eb244914dcabc6965324eced236 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 election: Requested vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:45.066619 30894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7df30eb244914dcabc6965324eced236" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:12:45.066715 31161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7df30eb244914dcabc6965324eced236" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:12:45.066809 30894 raft_consensus.cc:3037] T 7df30eb244914dcabc6965324eced236 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:45.066895 31161 raft_consensus.cc:3037] T 7df30eb244914dcabc6965324eced236 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:45.068697 31161 raft_consensus.cc:2454] T 7df30eb244914dcabc6965324eced236 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 1.
I20240503 08:12:45.068858 30894 raft_consensus.cc:2454] T 7df30eb244914dcabc6965324eced236 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 1.
I20240503 08:12:45.069056 30959 leader_election.cc:302] T 7df30eb244914dcabc6965324eced236 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:45.069329 31632 raft_consensus.cc:2781] T 7df30eb244914dcabc6965324eced236 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:45.069474 31632 raft_consensus.cc:686] T 7df30eb244914dcabc6965324eced236 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 LEADER]: Becoming Leader. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Running, Role: LEADER
I20240503 08:12:45.069703 31632 consensus_queue.cc:237] T 7df30eb244914dcabc6965324eced236 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:45.072696 27859 catalog_manager.cc:5484] T 7df30eb244914dcabc6965324eced236 P ced21ca983b749d0b4d4f6a0e2426bf8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71). New cstate: current_term: 1 leader_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:12:45.080315 31612 raft_consensus.cc:482] T 01c4229dfd25458296e85e412eae13bf P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:45.080478 31612 raft_consensus.cc:504] T 01c4229dfd25458296e85e412eae13bf P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:45.081130 31612 leader_election.cc:288] T 01c4229dfd25458296e85e412eae13bf P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:12:45.082036 31161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01c4229dfd25458296e85e412eae13bf" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:12:45.082257 31161 raft_consensus.cc:2454] T 01c4229dfd25458296e85e412eae13bf P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:12:45.082738 30694 leader_election.cc:302] T 01c4229dfd25458296e85e412eae13bf P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, b00f244c69df401997ea99d492218a98; no voters:
I20240503 08:12:45.083025 31612 raft_consensus.cc:2781] T 01c4229dfd25458296e85e412eae13bf P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:45.083128 31612 raft_consensus.cc:482] T 01c4229dfd25458296e85e412eae13bf P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:45.083232 31612 raft_consensus.cc:3037] T 01c4229dfd25458296e85e412eae13bf P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:45.085270 31612 raft_consensus.cc:504] T 01c4229dfd25458296e85e412eae13bf P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:45.085891 31612 leader_election.cc:288] T 01c4229dfd25458296e85e412eae13bf P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Requested vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:12:45.086164 31161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01c4229dfd25458296e85e412eae13bf" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:12:45.086329 31161 raft_consensus.cc:3037] T 01c4229dfd25458296e85e412eae13bf P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:45.088243 31161 raft_consensus.cc:2454] T 01c4229dfd25458296e85e412eae13bf P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:12:45.088559 30694 leader_election.cc:302] T 01c4229dfd25458296e85e412eae13bf P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, b00f244c69df401997ea99d492218a98; no voters:
I20240503 08:12:45.088811 31612 raft_consensus.cc:2781] T 01c4229dfd25458296e85e412eae13bf P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:45.088965 31612 raft_consensus.cc:686] T 01c4229dfd25458296e85e412eae13bf P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Becoming Leader. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Running, Role: LEADER
I20240503 08:12:45.089234 31612 consensus_queue.cc:237] T 01c4229dfd25458296e85e412eae13bf P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:45.100515 27859 catalog_manager.cc:5484] T 01c4229dfd25458296e85e412eae13bf P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69). New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:12:45.102458 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01c4229dfd25458296e85e412eae13bf" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:12:45.102639 28002 raft_consensus.cc:3037] T 01c4229dfd25458296e85e412eae13bf P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:45.104684 28002 raft_consensus.cc:2454] T 01c4229dfd25458296e85e412eae13bf P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:12:45.105285 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01c4229dfd25458296e85e412eae13bf" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:12:45.105508 28003 raft_consensus.cc:2362] T 01c4229dfd25458296e85e412eae13bf P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1. Re-sending same reply.
I20240503 08:12:45.105407 27859 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:45.111680 28112 tablet_service.cc:1467] Processing CreateTablet for tablet a0432ff96c5b499188edaf1c2ff33d40 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0910363578a64386a6c023a1146caceb]), partition=
I20240503 08:12:45.111858 31007 tablet_service.cc:1467] Processing CreateTablet for tablet a0432ff96c5b499188edaf1c2ff33d40 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0910363578a64386a6c023a1146caceb]), partition=
I20240503 08:12:45.112035 28112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0432ff96c5b499188edaf1c2ff33d40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:45.112095 30874 tablet_service.cc:1467] Processing CreateTablet for tablet a0432ff96c5b499188edaf1c2ff33d40 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0910363578a64386a6c023a1146caceb]), partition=
I20240503 08:12:45.112208 31007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0432ff96c5b499188edaf1c2ff33d40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:45.112429 30874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0432ff96c5b499188edaf1c2ff33d40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:45.116753 31660 tablet_bootstrap.cc:492] T a0432ff96c5b499188edaf1c2ff33d40 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:45.118036 31660 tablet_bootstrap.cc:654] T a0432ff96c5b499188edaf1c2ff33d40 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:45.118103 31641 tablet_bootstrap.cc:492] T a0432ff96c5b499188edaf1c2ff33d40 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:45.118634 31648 tablet_bootstrap.cc:492] T a0432ff96c5b499188edaf1c2ff33d40 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:45.119478 31641 tablet_bootstrap.cc:654] T a0432ff96c5b499188edaf1c2ff33d40 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:45.120015 31648 tablet_bootstrap.cc:654] T a0432ff96c5b499188edaf1c2ff33d40 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:45.120327 31660 tablet_bootstrap.cc:492] T a0432ff96c5b499188edaf1c2ff33d40 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:45.120440 31660 ts_tablet_manager.cc:1367] T a0432ff96c5b499188edaf1c2ff33d40 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:12:45.121109 31641 tablet_bootstrap.cc:492] T a0432ff96c5b499188edaf1c2ff33d40 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:45.121098 31660 raft_consensus.cc:348] T a0432ff96c5b499188edaf1c2ff33d40 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:45.121248 31641 ts_tablet_manager.cc:1367] T a0432ff96c5b499188edaf1c2ff33d40 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:12:45.121304 31660 raft_consensus.cc:374] T a0432ff96c5b499188edaf1c2ff33d40 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:45.121387 31660 raft_consensus.cc:729] T a0432ff96c5b499188edaf1c2ff33d40 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:45.121587 31660 consensus_queue.cc:260] T a0432ff96c5b499188edaf1c2ff33d40 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:45.122398 31641 raft_consensus.cc:348] T a0432ff96c5b499188edaf1c2ff33d40 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:45.122666 31641 raft_consensus.cc:374] T a0432ff96c5b499188edaf1c2ff33d40 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:45.122779 31641 raft_consensus.cc:729] T a0432ff96c5b499188edaf1c2ff33d40 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:45.123071 31641 consensus_queue.cc:260] T a0432ff96c5b499188edaf1c2ff33d40 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:45.124282 31641 ts_tablet_manager.cc:1398] T a0432ff96c5b499188edaf1c2ff33d40 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:45.125806 31660 ts_tablet_manager.cc:1398] T a0432ff96c5b499188edaf1c2ff33d40 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:12:45.129067 31648 tablet_bootstrap.cc:492] T a0432ff96c5b499188edaf1c2ff33d40 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:45.129186 31648 ts_tablet_manager.cc:1367] T a0432ff96c5b499188edaf1c2ff33d40 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240503 08:12:45.129750 31648 raft_consensus.cc:348] T a0432ff96c5b499188edaf1c2ff33d40 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:45.129930 31648 raft_consensus.cc:374] T a0432ff96c5b499188edaf1c2ff33d40 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:45.130034 31648 raft_consensus.cc:729] T a0432ff96c5b499188edaf1c2ff33d40 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:45.130218 31648 consensus_queue.cc:260] T a0432ff96c5b499188edaf1c2ff33d40 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:45.130609 31648 ts_tablet_manager.cc:1398] T a0432ff96c5b499188edaf1c2ff33d40 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:12:45.151479 27859 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240503 08:12:45.151729 27859 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240503 08:12:45.151932 30510 raft_consensus.cc:482] T a0432ff96c5b499188edaf1c2ff33d40 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:45.152087 30510 raft_consensus.cc:504] T a0432ff96c5b499188edaf1c2ff33d40 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:45.152864 30510 leader_election.cc:288] T a0432ff96c5b499188edaf1c2ff33d40 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:12:45.156414 27859 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet 479beba3dd3e42faa60c598ad74c7041
I20240503 08:12:45.157425 27981 tablet_service.cc:1514] Processing DeleteTablet for tablet 479beba3dd3e42faa60c598ad74c7041 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:45 UTC) from {username='slave'} at 127.0.0.1:55610
I20240503 08:12:45.157547 27859 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet 7c3e836df884424e99a892a2942faebe
I20240503 08:12:45.157526 28372 tablet_service.cc:1514] Processing DeleteTablet for tablet 479beba3dd3e42faa60c598ad74c7041 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:45 UTC) from {username='slave'} at 127.0.0.1:34786
I20240503 08:12:45.157845 31511 tablet_replica.cc:330] T 479beba3dd3e42faa60c598ad74c7041 P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:12:45.157883 31268 tablet_replica.cc:330] T 479beba3dd3e42faa60c598ad74c7041 P d09a11a03989421b9f28019dc1c14cc2: stopping tablet replica
I20240503 08:12:45.158073 31511 raft_consensus.cc:2229] T 479beba3dd3e42faa60c598ad74c7041 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:45.158102 31268 raft_consensus.cc:2229] T 479beba3dd3e42faa60c598ad74c7041 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:45.158303 27981 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c3e836df884424e99a892a2942faebe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:45 UTC) from {username='slave'} at 127.0.0.1:55610
I20240503 08:12:45.158345 31268 raft_consensus.cc:2258] T 479beba3dd3e42faa60c598ad74c7041 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:45.158354 31511 raft_consensus.cc:2258] T 479beba3dd3e42faa60c598ad74c7041 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:45.158452 27859 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet 66c9df9a0fa24db49a6120cc9fe1bc77
I20240503 08:12:45.158946 27981 tablet_service.cc:1514] Processing DeleteTablet for tablet 66c9df9a0fa24db49a6120cc9fe1bc77 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:45 UTC) from {username='slave'} at 127.0.0.1:55610
I20240503 08:12:45.159058 31268 ts_tablet_manager.cc:1875] T 479beba3dd3e42faa60c598ad74c7041 P d09a11a03989421b9f28019dc1c14cc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:45.159078 31511 ts_tablet_manager.cc:1875] T 479beba3dd3e42faa60c598ad74c7041 P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:45.159456 28373 tablet_service.cc:1514] Processing DeleteTablet for tablet 66c9df9a0fa24db49a6120cc9fe1bc77 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:45 UTC) from {username='slave'} at 127.0.0.1:34786
I20240503 08:12:45.161999 28372 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c3e836df884424e99a892a2942faebe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:45 UTC) from {username='slave'} at 127.0.0.1:34786
I20240503 08:12:45.162312 28243 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c3e836df884424e99a892a2942faebe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:45 UTC) from {username='slave'} at 127.0.0.1:41960
I20240503 08:12:45.162585 31626 tablet_replica.cc:330] T 7c3e836df884424e99a892a2942faebe P aae860d7ae2b4002bf7794c278f9cc56: stopping tablet replica
I20240503 08:12:45.162772 31626 raft_consensus.cc:2229] T 7c3e836df884424e99a892a2942faebe P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:45.163012 31626 raft_consensus.cc:2258] T 7c3e836df884424e99a892a2942faebe P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:45.163681 31626 ts_tablet_manager.cc:1875] T 7c3e836df884424e99a892a2942faebe P aae860d7ae2b4002bf7794c278f9cc56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:45.165666 31511 ts_tablet_manager.cc:1888] T 479beba3dd3e42faa60c598ad74c7041 P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:45.165788 31511 log.cc:1196] T 479beba3dd3e42faa60c598ad74c7041 P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/479beba3dd3e42faa60c598ad74c7041
I20240503 08:12:45.165735 31268 ts_tablet_manager.cc:1888] T 479beba3dd3e42faa60c598ad74c7041 P d09a11a03989421b9f28019dc1c14cc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:45.165979 31268 log.cc:1196] T 479beba3dd3e42faa60c598ad74c7041 P d09a11a03989421b9f28019dc1c14cc2: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/wals/479beba3dd3e42faa60c598ad74c7041
I20240503 08:12:45.166213 31511 ts_tablet_manager.cc:1909] T 479beba3dd3e42faa60c598ad74c7041 P 833582ff68c94269be364ebd187a75db: Deleting consensus metadata
I20240503 08:12:45.166349 31268 ts_tablet_manager.cc:1909] T 479beba3dd3e42faa60c598ad74c7041 P d09a11a03989421b9f28019dc1c14cc2: Deleting consensus metadata
I20240503 08:12:45.166937 31511 tablet_replica.cc:330] T 7c3e836df884424e99a892a2942faebe P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:12:45.167064 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet 479beba3dd3e42faa60c598ad74c7041 (table rebalance_test_table_18 [id=1cae04951c764544be14f32e3cbc0ed0]) successfully deleted
I20240503 08:12:45.167308 27849 catalog_manager.cc:4830] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): tablet 479beba3dd3e42faa60c598ad74c7041 (table rebalance_test_table_18 [id=1cae04951c764544be14f32e3cbc0ed0]) successfully deleted
I20240503 08:12:45.167392 31268 tablet_replica.cc:330] T 66c9df9a0fa24db49a6120cc9fe1bc77 P d09a11a03989421b9f28019dc1c14cc2: stopping tablet replica
I20240503 08:12:45.167577 31268 raft_consensus.cc:2229] T 66c9df9a0fa24db49a6120cc9fe1bc77 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:45.167788 31511 raft_consensus.cc:2229] T 7c3e836df884424e99a892a2942faebe P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:45.167879 31268 raft_consensus.cc:2258] T 66c9df9a0fa24db49a6120cc9fe1bc77 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:45.167995 31511 raft_consensus.cc:2258] T 7c3e836df884424e99a892a2942faebe P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:45.168506 31511 ts_tablet_manager.cc:1875] T 7c3e836df884424e99a892a2942faebe P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:45.168495 31268 ts_tablet_manager.cc:1875] T 66c9df9a0fa24db49a6120cc9fe1bc77 P d09a11a03989421b9f28019dc1c14cc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:45.172407 31626 ts_tablet_manager.cc:1888] T 7c3e836df884424e99a892a2942faebe P aae860d7ae2b4002bf7794c278f9cc56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:45.174513 31511 ts_tablet_manager.cc:1888] T 7c3e836df884424e99a892a2942faebe P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:45.174659 31511 log.cc:1196] T 7c3e836df884424e99a892a2942faebe P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/7c3e836df884424e99a892a2942faebe
I20240503 08:12:45.174837 31268 ts_tablet_manager.cc:1888] T 66c9df9a0fa24db49a6120cc9fe1bc77 P d09a11a03989421b9f28019dc1c14cc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:45.174922 31626 log.cc:1196] T 7c3e836df884424e99a892a2942faebe P aae860d7ae2b4002bf7794c278f9cc56: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal/wals/7c3e836df884424e99a892a2942faebe
I20240503 08:12:45.175098 31511 ts_tablet_manager.cc:1909] T 7c3e836df884424e99a892a2942faebe P 833582ff68c94269be364ebd187a75db: Deleting consensus metadata
I20240503 08:12:45.175129 31268 log.cc:1196] T 66c9df9a0fa24db49a6120cc9fe1bc77 P d09a11a03989421b9f28019dc1c14cc2: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/wals/66c9df9a0fa24db49a6120cc9fe1bc77
I20240503 08:12:45.175546 31268 ts_tablet_manager.cc:1909] T 66c9df9a0fa24db49a6120cc9fe1bc77 P d09a11a03989421b9f28019dc1c14cc2: Deleting consensus metadata
I20240503 08:12:45.175591 31626 ts_tablet_manager.cc:1909] T 7c3e836df884424e99a892a2942faebe P aae860d7ae2b4002bf7794c278f9cc56: Deleting consensus metadata
I20240503 08:12:45.175874 31511 tablet_replica.cc:330] T 66c9df9a0fa24db49a6120cc9fe1bc77 P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:12:45.175971 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet 7c3e836df884424e99a892a2942faebe (table rebalance_test_table_18 [id=1cae04951c764544be14f32e3cbc0ed0]) successfully deleted
I20240503 08:12:45.176057 31511 raft_consensus.cc:2229] T 66c9df9a0fa24db49a6120cc9fe1bc77 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:45.176273 31511 raft_consensus.cc:2258] T 66c9df9a0fa24db49a6120cc9fe1bc77 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:45.176366 31268 tablet_replica.cc:330] T 7c3e836df884424e99a892a2942faebe P d09a11a03989421b9f28019dc1c14cc2: stopping tablet replica
I20240503 08:12:45.176493 27847 catalog_manager.cc:4830] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): tablet 7c3e836df884424e99a892a2942faebe (table rebalance_test_table_18 [id=1cae04951c764544be14f32e3cbc0ed0]) successfully deleted
I20240503 08:12:45.176524 27849 catalog_manager.cc:4830] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): tablet 66c9df9a0fa24db49a6120cc9fe1bc77 (table rebalance_test_table_18 [id=1cae04951c764544be14f32e3cbc0ed0]) successfully deleted
I20240503 08:12:45.176615 31268 raft_consensus.cc:2229] T 7c3e836df884424e99a892a2942faebe P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:45.176885 31511 ts_tablet_manager.cc:1875] T 66c9df9a0fa24db49a6120cc9fe1bc77 P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:45.176895 31268 raft_consensus.cc:2258] T 7c3e836df884424e99a892a2942faebe P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:45.177532 31268 ts_tablet_manager.cc:1875] T 7c3e836df884424e99a892a2942faebe P d09a11a03989421b9f28019dc1c14cc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:45.180661 31511 ts_tablet_manager.cc:1888] T 66c9df9a0fa24db49a6120cc9fe1bc77 P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:45.180764 31511 log.cc:1196] T 66c9df9a0fa24db49a6120cc9fe1bc77 P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/66c9df9a0fa24db49a6120cc9fe1bc77
I20240503 08:12:45.181257 31511 ts_tablet_manager.cc:1909] T 66c9df9a0fa24db49a6120cc9fe1bc77 P 833582ff68c94269be364ebd187a75db: Deleting consensus metadata
I20240503 08:12:45.181476 28112 tablet_service.cc:1514] Processing DeleteTablet for tablet 479beba3dd3e42faa60c598ad74c7041 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:45 UTC) from {username='slave'} at 127.0.0.1:57912
I20240503 08:12:45.181718 28111 tablet_service.cc:1514] Processing DeleteTablet for tablet 66c9df9a0fa24db49a6120cc9fe1bc77 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:45 UTC) from {username='slave'} at 127.0.0.1:57912
I20240503 08:12:45.181941 31267 tablet_replica.cc:330] T 479beba3dd3e42faa60c598ad74c7041 P 1b411c6bb25549368041773d7b710d48: stopping tablet replica
I20240503 08:12:45.182158 31267 raft_consensus.cc:2229] T 479beba3dd3e42faa60c598ad74c7041 P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:45.182189 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet 66c9df9a0fa24db49a6120cc9fe1bc77 (table rebalance_test_table_18 [id=1cae04951c764544be14f32e3cbc0ed0]) successfully deleted
I20240503 08:12:45.182508 31267 raft_consensus.cc:2258] T 479beba3dd3e42faa60c598ad74c7041 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:45.183241 31267 ts_tablet_manager.cc:1875] T 479beba3dd3e42faa60c598ad74c7041 P 1b411c6bb25549368041773d7b710d48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:45.185540 31268 ts_tablet_manager.cc:1888] T 7c3e836df884424e99a892a2942faebe P d09a11a03989421b9f28019dc1c14cc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:45.185662 31268 log.cc:1196] T 7c3e836df884424e99a892a2942faebe P d09a11a03989421b9f28019dc1c14cc2: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/wals/7c3e836df884424e99a892a2942faebe
I20240503 08:12:45.186072 31268 ts_tablet_manager.cc:1909] T 7c3e836df884424e99a892a2942faebe P d09a11a03989421b9f28019dc1c14cc2: Deleting consensus metadata
I20240503 08:12:45.189015 27849 catalog_manager.cc:4830] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): tablet 7c3e836df884424e99a892a2942faebe (table rebalance_test_table_18 [id=1cae04951c764544be14f32e3cbc0ed0]) successfully deleted
I20240503 08:12:45.189184 31267 ts_tablet_manager.cc:1888] T 479beba3dd3e42faa60c598ad74c7041 P 1b411c6bb25549368041773d7b710d48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:45.189349 31267 log.cc:1196] T 479beba3dd3e42faa60c598ad74c7041 P 1b411c6bb25549368041773d7b710d48: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/479beba3dd3e42faa60c598ad74c7041
I20240503 08:12:45.189823 31267 ts_tablet_manager.cc:1909] T 479beba3dd3e42faa60c598ad74c7041 P 1b411c6bb25549368041773d7b710d48: Deleting consensus metadata
I20240503 08:12:45.190685 31267 tablet_replica.cc:330] T 66c9df9a0fa24db49a6120cc9fe1bc77 P 1b411c6bb25549368041773d7b710d48: stopping tablet replica
I20240503 08:12:45.190901 31267 raft_consensus.cc:2229] T 66c9df9a0fa24db49a6120cc9fe1bc77 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:45.190840 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0432ff96c5b499188edaf1c2ff33d40" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:45.191047 31027 raft_consensus.cc:2454] T a0432ff96c5b499188edaf1c2ff33d40 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:45.191118 31267 raft_consensus.cc:2258] T 66c9df9a0fa24db49a6120cc9fe1bc77 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:45.191327 27848 catalog_manager.cc:4830] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): tablet 479beba3dd3e42faa60c598ad74c7041 (table rebalance_test_table_18 [id=1cae04951c764544be14f32e3cbc0ed0]) successfully deleted
I20240503 08:12:45.191694 28067 leader_election.cc:302] T a0432ff96c5b499188edaf1c2ff33d40 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:45.191785 31267 ts_tablet_manager.cc:1875] T 66c9df9a0fa24db49a6120cc9fe1bc77 P 1b411c6bb25549368041773d7b710d48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:45.191979 30510 raft_consensus.cc:2781] T a0432ff96c5b499188edaf1c2ff33d40 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:45.192091 30510 raft_consensus.cc:482] T a0432ff96c5b499188edaf1c2ff33d40 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:45.192214 30510 raft_consensus.cc:3037] T a0432ff96c5b499188edaf1c2ff33d40 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:45.195176 30894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0432ff96c5b499188edaf1c2ff33d40" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:12:45.195387 30894 raft_consensus.cc:2454] T a0432ff96c5b499188edaf1c2ff33d40 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:45.196048 30510 raft_consensus.cc:504] T a0432ff96c5b499188edaf1c2ff33d40 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:45.198269 30510 leader_election.cc:288] T a0432ff96c5b499188edaf1c2ff33d40 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Requested vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:12:45.198405 31267 ts_tablet_manager.cc:1888] T 66c9df9a0fa24db49a6120cc9fe1bc77 P 1b411c6bb25549368041773d7b710d48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:45.198537 31267 log.cc:1196] T 66c9df9a0fa24db49a6120cc9fe1bc77 P 1b411c6bb25549368041773d7b710d48: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/66c9df9a0fa24db49a6120cc9fe1bc77
I20240503 08:12:45.198655 30894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0432ff96c5b499188edaf1c2ff33d40" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:12:45.198854 30894 raft_consensus.cc:3037] T a0432ff96c5b499188edaf1c2ff33d40 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:45.198904 31267 ts_tablet_manager.cc:1909] T 66c9df9a0fa24db49a6120cc9fe1bc77 P 1b411c6bb25549368041773d7b710d48: Deleting consensus metadata
I20240503 08:12:45.200580 30894 raft_consensus.cc:2454] T a0432ff96c5b499188edaf1c2ff33d40 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:12:45.200965 27848 catalog_manager.cc:4830] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): tablet 66c9df9a0fa24db49a6120cc9fe1bc77 (table rebalance_test_table_18 [id=1cae04951c764544be14f32e3cbc0ed0]) successfully deleted
I20240503 08:12:45.200973 28066 leader_election.cc:302] T a0432ff96c5b499188edaf1c2ff33d40 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:45.201218 30510 raft_consensus.cc:2781] T a0432ff96c5b499188edaf1c2ff33d40 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:45.201359 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0432ff96c5b499188edaf1c2ff33d40" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:12:45.201536 31027 raft_consensus.cc:3037] T a0432ff96c5b499188edaf1c2ff33d40 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:45.201630 30510 raft_consensus.cc:686] T a0432ff96c5b499188edaf1c2ff33d40 P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Becoming Leader. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Running, Role: LEADER
I20240503 08:12:45.201998 30510 consensus_queue.cc:237] T a0432ff96c5b499188edaf1c2ff33d40 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:45.206869 27859 catalog_manager.cc:5484] T a0432ff96c5b499188edaf1c2ff33d40 P 1b411c6bb25549368041773d7b710d48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b411c6bb25549368041773d7b710d48 (127.20.230.66). New cstate: current_term: 1 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:12:45.227541 31027 raft_consensus.cc:2454] T a0432ff96c5b499188edaf1c2ff33d40 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:12:45.325083 27859 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:45.331180 31027 raft_consensus.cc:1261] T a0432ff96c5b499188edaf1c2ff33d40 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:45.331183 30894 raft_consensus.cc:1261] T a0432ff96c5b499188edaf1c2ff33d40 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:45.331823 31668 consensus_queue.cc:1035] T a0432ff96c5b499188edaf1c2ff33d40 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:45.332129 30510 consensus_queue.cc:1035] T a0432ff96c5b499188edaf1c2ff33d40 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:45.335564 27859 catalog_manager.cc:5576] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747) does not have the latest schema for tablet a0432ff96c5b499188edaf1c2ff33d40 (table rebalancer_dynamic_table [id=0910363578a64386a6c023a1146caceb]). Expected version 1 got 0
I20240503 08:12:45.336498 31625 tablet.cc:1722] T a0432ff96c5b499188edaf1c2ff33d40 P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:45.337004 31625 tablet.cc:1620] T a0432ff96c5b499188edaf1c2ff33d40 P ce1496a47d084365ad786cd09e472a2c: MemRowSet was empty: no flush needed.
I20240503 08:12:45.342694 27859 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=0910363578a64386a6c023a1146caceb] alter complete (version 1)
I20240503 08:12:45.346714 31673 tablet.cc:1722] T a0432ff96c5b499188edaf1c2ff33d40 P 1b411c6bb25549368041773d7b710d48: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:45.347236 31673 tablet.cc:1620] T a0432ff96c5b499188edaf1c2ff33d40 P 1b411c6bb25549368041773d7b710d48: MemRowSet was empty: no flush needed.
I20240503 08:12:45.348598 27855 catalog_manager.cc:5576] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915) does not have the latest schema for tablet a0432ff96c5b499188edaf1c2ff33d40 (table rebalancer_dynamic_table [id=0910363578a64386a6c023a1146caceb]). Expected version 1 got 0
I20240503 08:12:45.350203 31624 tablet.cc:1722] T a0432ff96c5b499188edaf1c2ff33d40 P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:45.350641 31624 tablet.cc:1620] T a0432ff96c5b499188edaf1c2ff33d40 P ced21ca983b749d0b4d4f6a0e2426bf8: MemRowSet was empty: no flush needed.
I20240503 08:12:45.351630 27855 catalog_manager.cc:5576] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915) does not have the latest schema for tablet a0432ff96c5b499188edaf1c2ff33d40 (table rebalancer_dynamic_table [id=0910363578a64386a6c023a1146caceb]). Expected version 1 got 0
I20240503 08:12:45.444756 27855 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:45.452431 31625 tablet.cc:1722] T a0432ff96c5b499188edaf1c2ff33d40 P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:45.452416 31673 tablet.cc:1722] T a0432ff96c5b499188edaf1c2ff33d40 P 1b411c6bb25549368041773d7b710d48: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:45.452898 31625 tablet.cc:1620] T a0432ff96c5b499188edaf1c2ff33d40 P ce1496a47d084365ad786cd09e472a2c: MemRowSet was empty: no flush needed.
I20240503 08:12:45.452961 31673 tablet.cc:1620] T a0432ff96c5b499188edaf1c2ff33d40 P 1b411c6bb25549368041773d7b710d48: MemRowSet was empty: no flush needed.
I20240503 08:12:45.456302 31582 raft_consensus.cc:482] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:45.456523 31582 raft_consensus.cc:504] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:45.457192 31582 leader_election.cc:288] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:45.457688 30652 raft_consensus.cc:482] T 4a23c8186c4d4b23b62d7474088b45c1 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:45.457877 30652 raft_consensus.cc:504] T 4a23c8186c4d4b23b62d7474088b45c1 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:45.458552 30652 leader_election.cc:288] T 4a23c8186c4d4b23b62d7474088b45c1 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:45.459036 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a23c8186c4d4b23b62d7474088b45c1" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:45.459249 31027 raft_consensus.cc:2454] T 4a23c8186c4d4b23b62d7474088b45c1 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 0.
I20240503 08:12:45.459870 31095 leader_election.cc:302] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:45.460129 31582 raft_consensus.cc:2781] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:45.460222 31582 raft_consensus.cc:482] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:45.460325 31582 raft_consensus.cc:3037] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:45.460302 31624 tablet.cc:1722] T a0432ff96c5b499188edaf1c2ff33d40 P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:45.460227 31161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a23c8186c4d4b23b62d7474088b45c1" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:12:45.460776 31624 tablet.cc:1620] T a0432ff96c5b499188edaf1c2ff33d40 P ced21ca983b749d0b4d4f6a0e2426bf8: MemRowSet was empty: no flush needed.
I20240503 08:12:45.462893 31582 raft_consensus.cc:504] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:45.464133 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a23c8186c4d4b23b62d7474088b45c1" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:12:45.464313 31027 raft_consensus.cc:3037] T 4a23c8186c4d4b23b62d7474088b45c1 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:45.464531 31028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a23c8186c4d4b23b62d7474088b45c1" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:45.465600 31161 raft_consensus.cc:2379] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate aae860d7ae2b4002bf7794c278f9cc56 in current term 1: Already voted for candidate b00f244c69df401997ea99d492218a98 in this term.
I20240503 08:12:45.466488 28196 leader_election.cc:302] T 4a23c8186c4d4b23b62d7474088b45c1 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56; no voters: b00f244c69df401997ea99d492218a98, ced21ca983b749d0b4d4f6a0e2426bf8
I20240503 08:12:45.466763 30652 raft_consensus.cc:3037] T 4a23c8186c4d4b23b62d7474088b45c1 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:45.466862 31027 raft_consensus.cc:2454] T 4a23c8186c4d4b23b62d7474088b45c1 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 1.
I20240503 08:12:45.467243 31095 leader_election.cc:302] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:45.467772 31582 leader_election.cc:288] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:45.467931 31582 raft_consensus.cc:2781] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:45.468045 31582 raft_consensus.cc:686] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Becoming Leader. State: Replica: b00f244c69df401997ea99d492218a98, State: Running, Role: LEADER
I20240503 08:12:45.468268 31582 consensus_queue.cc:237] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:45.469194 30652 raft_consensus.cc:2726] T 4a23c8186c4d4b23b62d7474088b45c1 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:12:45.474479 28261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a23c8186c4d4b23b62d7474088b45c1" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:12:45.474612 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a23c8186c4d4b23b62d7474088b45c1" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:12:45.474715 28261 raft_consensus.cc:2454] T 4a23c8186c4d4b23b62d7474088b45c1 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 1.
I20240503 08:12:45.475062 27856 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=0910363578a64386a6c023a1146caceb] alter complete (version 2)
I20240503 08:12:45.477342 27859 catalog_manager.cc:5484] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98 reported cstate change: term changed from 0 to 1, leader changed from <none> to b00f244c69df401997ea99d492218a98 (127.20.230.72). New cstate: current_term: 1 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:12:45.578495 27859 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:45.585665 31673 tablet.cc:1722] T a0432ff96c5b499188edaf1c2ff33d40 P 1b411c6bb25549368041773d7b710d48: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:45.586161 31624 tablet.cc:1722] T a0432ff96c5b499188edaf1c2ff33d40 P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:45.590442 31625 tablet.cc:1722] T a0432ff96c5b499188edaf1c2ff33d40 P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:45.594273 31632 consensus_queue.cc:1035] T 7df30eb244914dcabc6965324eced236 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:45.595779 27859 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table_new_name [id=0910363578a64386a6c023a1146caceb] alter complete (version 3)
I20240503 08:12:45.597741 31612 consensus_queue.cc:1035] T 01c4229dfd25458296e85e412eae13bf P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:45.602258 31680 consensus_queue.cc:1035] T 7df30eb244914dcabc6965324eced236 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:45.618584 31688 consensus_queue.cc:1035] T 01c4229dfd25458296e85e412eae13bf P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:45.697808 27859 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:45.698061 27859 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:45.701094 27859 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet a0432ff96c5b499188edaf1c2ff33d40
I20240503 08:12:45.701891 28111 tablet_service.cc:1514] Processing DeleteTablet for tablet a0432ff96c5b499188edaf1c2ff33d40 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:45 UTC) from {username='slave'} at 127.0.0.1:57912
I20240503 08:12:45.702082 31007 tablet_service.cc:1514] Processing DeleteTablet for tablet a0432ff96c5b499188edaf1c2ff33d40 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:45 UTC) from {username='slave'} at 127.0.0.1:47020
I20240503 08:12:45.702570 30874 tablet_service.cc:1514] Processing DeleteTablet for tablet a0432ff96c5b499188edaf1c2ff33d40 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:45 UTC) from {username='slave'} at 127.0.0.1:34426
I20240503 08:12:45.702621 31691 tablet_replica.cc:330] T a0432ff96c5b499188edaf1c2ff33d40 P ced21ca983b749d0b4d4f6a0e2426bf8: stopping tablet replica
I20240503 08:12:45.702877 31691 raft_consensus.cc:2229] T a0432ff96c5b499188edaf1c2ff33d40 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:45.703104 31691 raft_consensus.cc:2258] T a0432ff96c5b499188edaf1c2ff33d40 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:45.703859 31691 ts_tablet_manager.cc:1875] T a0432ff96c5b499188edaf1c2ff33d40 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:45.706079 31692 tablet_replica.cc:330] T a0432ff96c5b499188edaf1c2ff33d40 P ce1496a47d084365ad786cd09e472a2c: stopping tablet replica
I20240503 08:12:45.706326 31692 raft_consensus.cc:2229] T a0432ff96c5b499188edaf1c2ff33d40 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:45.706596 31692 raft_consensus.cc:2258] T a0432ff96c5b499188edaf1c2ff33d40 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:45.707460 31692 ts_tablet_manager.cc:1875] T a0432ff96c5b499188edaf1c2ff33d40 P ce1496a47d084365ad786cd09e472a2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:45.708978 31691 ts_tablet_manager.cc:1888] T a0432ff96c5b499188edaf1c2ff33d40 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:45.709151 31691 log.cc:1196] T a0432ff96c5b499188edaf1c2ff33d40 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal/wals/a0432ff96c5b499188edaf1c2ff33d40
I20240503 08:12:45.710925 31691 ts_tablet_manager.cc:1909] T a0432ff96c5b499188edaf1c2ff33d40 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting consensus metadata
I20240503 08:12:45.711971 27847 catalog_manager.cc:4830] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): tablet a0432ff96c5b499188edaf1c2ff33d40 (table rebalancer_dynamic_table_new_name [id=0910363578a64386a6c023a1146caceb]) successfully deleted
I20240503 08:12:45.713122 31692 ts_tablet_manager.cc:1888] T a0432ff96c5b499188edaf1c2ff33d40 P ce1496a47d084365ad786cd09e472a2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:45.713248 31692 log.cc:1196] T a0432ff96c5b499188edaf1c2ff33d40 P ce1496a47d084365ad786cd09e472a2c: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal/wals/a0432ff96c5b499188edaf1c2ff33d40
I20240503 08:12:45.713640 31692 ts_tablet_manager.cc:1909] T a0432ff96c5b499188edaf1c2ff33d40 P ce1496a47d084365ad786cd09e472a2c: Deleting consensus metadata
I20240503 08:12:45.714500 27846 catalog_manager.cc:4830] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): tablet a0432ff96c5b499188edaf1c2ff33d40 (table rebalancer_dynamic_table_new_name [id=0910363578a64386a6c023a1146caceb]) successfully deleted
I20240503 08:12:45.716405 31690 tablet_replica.cc:330] T a0432ff96c5b499188edaf1c2ff33d40 P 1b411c6bb25549368041773d7b710d48: stopping tablet replica
I20240503 08:12:45.716614 31690 raft_consensus.cc:2229] T a0432ff96c5b499188edaf1c2ff33d40 P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:45.716951 31690 raft_consensus.cc:2258] T a0432ff96c5b499188edaf1c2ff33d40 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:45.717609 31690 ts_tablet_manager.cc:1875] T a0432ff96c5b499188edaf1c2ff33d40 P 1b411c6bb25549368041773d7b710d48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:45.723549 31690 ts_tablet_manager.cc:1888] T a0432ff96c5b499188edaf1c2ff33d40 P 1b411c6bb25549368041773d7b710d48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:45.723670 31690 log.cc:1196] T a0432ff96c5b499188edaf1c2ff33d40 P 1b411c6bb25549368041773d7b710d48: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/a0432ff96c5b499188edaf1c2ff33d40
I20240503 08:12:45.724033 31690 ts_tablet_manager.cc:1909] T a0432ff96c5b499188edaf1c2ff33d40 P 1b411c6bb25549368041773d7b710d48: Deleting consensus metadata
I20240503 08:12:45.724928 27848 catalog_manager.cc:4830] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): tablet a0432ff96c5b499188edaf1c2ff33d40 (table rebalancer_dynamic_table_new_name [id=0910363578a64386a6c023a1146caceb]) successfully deleted
I20240503 08:12:45.804060 27859 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:45.810518 30874 tablet_service.cc:1467] Processing CreateTablet for tablet 98207a56344d46209b670dba4523212c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=54c2d37c1d4b49f89a8439e270fa56b5]), partition=
I20240503 08:12:45.810894 30874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98207a56344d46209b670dba4523212c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:45.810976 30741 tablet_service.cc:1467] Processing CreateTablet for tablet 98207a56344d46209b670dba4523212c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=54c2d37c1d4b49f89a8439e270fa56b5]), partition=
I20240503 08:12:45.811326 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98207a56344d46209b670dba4523212c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:45.811338 31139 tablet_service.cc:1467] Processing CreateTablet for tablet 98207a56344d46209b670dba4523212c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=54c2d37c1d4b49f89a8439e270fa56b5]), partition=
I20240503 08:12:45.811667 31139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98207a56344d46209b670dba4523212c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:45.815760 31693 tablet_bootstrap.cc:492] T 98207a56344d46209b670dba4523212c P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:45.819013 31693 tablet_bootstrap.cc:654] T 98207a56344d46209b670dba4523212c P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:45.819355 31695 tablet_bootstrap.cc:492] T 98207a56344d46209b670dba4523212c P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:45.819586 31694 tablet_bootstrap.cc:492] T 98207a56344d46209b670dba4523212c P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:45.820879 31694 tablet_bootstrap.cc:654] T 98207a56344d46209b670dba4523212c P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:45.821273 31695 tablet_bootstrap.cc:654] T 98207a56344d46209b670dba4523212c P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:45.821427 31693 tablet_bootstrap.cc:492] T 98207a56344d46209b670dba4523212c P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:45.821561 31693 ts_tablet_manager.cc:1367] T 98207a56344d46209b670dba4523212c P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 08:12:45.822281 31693 raft_consensus.cc:348] T 98207a56344d46209b670dba4523212c P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:45.822499 31693 raft_consensus.cc:374] T 98207a56344d46209b670dba4523212c P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:45.822592 31693 raft_consensus.cc:729] T 98207a56344d46209b670dba4523212c P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:45.822913 31694 tablet_bootstrap.cc:492] T 98207a56344d46209b670dba4523212c P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:45.822808 31693 consensus_queue.cc:260] T 98207a56344d46209b670dba4523212c P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:45.823040 31694 ts_tablet_manager.cc:1367] T 98207a56344d46209b670dba4523212c P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:12:45.823733 31694 raft_consensus.cc:348] T 98207a56344d46209b670dba4523212c P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:45.823921 31694 raft_consensus.cc:374] T 98207a56344d46209b670dba4523212c P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:45.824007 31694 raft_consensus.cc:729] T 98207a56344d46209b670dba4523212c P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:45.824215 31694 consensus_queue.cc:260] T 98207a56344d46209b670dba4523212c P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:45.824752 31694 ts_tablet_manager.cc:1398] T 98207a56344d46209b670dba4523212c P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:12:45.825963 31693 ts_tablet_manager.cc:1398] T 98207a56344d46209b670dba4523212c P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:12:45.846247 31677 consensus_queue.cc:1035] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:45.850121 31677 consensus_queue.cc:1035] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:45.850386 31695 tablet_bootstrap.cc:492] T 98207a56344d46209b670dba4523212c P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:45.850534 31695 ts_tablet_manager.cc:1367] T 98207a56344d46209b670dba4523212c P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.031s user 0.000s sys 0.003s
I20240503 08:12:45.851923 31695 raft_consensus.cc:348] T 98207a56344d46209b670dba4523212c P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:45.852126 31695 raft_consensus.cc:374] T 98207a56344d46209b670dba4523212c P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:45.852211 31695 raft_consensus.cc:729] T 98207a56344d46209b670dba4523212c P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:45.852468 31695 consensus_queue.cc:260] T 98207a56344d46209b670dba4523212c P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:45.852972 31695 ts_tablet_manager.cc:1398] T 98207a56344d46209b670dba4523212c P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:45.905576 31582 raft_consensus.cc:482] T 98207a56344d46209b670dba4523212c P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:45.905790 31582 raft_consensus.cc:504] T 98207a56344d46209b670dba4523212c P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:45.906482 31582 leader_election.cc:288] T 98207a56344d46209b670dba4523212c P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:12:45.906915 30894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98207a56344d46209b670dba4523212c" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:12:45.906961 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98207a56344d46209b670dba4523212c" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:12:45.907126 30894 raft_consensus.cc:2454] T 98207a56344d46209b670dba4523212c P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 0.
I20240503 08:12:45.907189 30762 raft_consensus.cc:2454] T 98207a56344d46209b670dba4523212c P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 0.
I20240503 08:12:45.907538 31094 leader_election.cc:302] T 98207a56344d46209b670dba4523212c P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:45.907812 31582 raft_consensus.cc:2781] T 98207a56344d46209b670dba4523212c P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:45.907907 31582 raft_consensus.cc:482] T 98207a56344d46209b670dba4523212c P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:45.907985 31582 raft_consensus.cc:3037] T 98207a56344d46209b670dba4523212c P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:45.909807 31582 raft_consensus.cc:504] T 98207a56344d46209b670dba4523212c P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:45.910334 31582 leader_election.cc:288] T 98207a56344d46209b670dba4523212c P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 election: Requested vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:12:45.910806 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98207a56344d46209b670dba4523212c" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:12:45.910823 30894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98207a56344d46209b670dba4523212c" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:12:45.910992 30894 raft_consensus.cc:3037] T 98207a56344d46209b670dba4523212c P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:45.911007 30762 raft_consensus.cc:3037] T 98207a56344d46209b670dba4523212c P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:45.913689 30894 raft_consensus.cc:2454] T 98207a56344d46209b670dba4523212c P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 1.
I20240503 08:12:45.913677 30762 raft_consensus.cc:2454] T 98207a56344d46209b670dba4523212c P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 1.
I20240503 08:12:45.914072 31096 leader_election.cc:302] T 98207a56344d46209b670dba4523212c P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, b00f244c69df401997ea99d492218a98; no voters:
I20240503 08:12:45.914423 31582 raft_consensus.cc:2781] T 98207a56344d46209b670dba4523212c P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:45.914557 31582 raft_consensus.cc:686] T 98207a56344d46209b670dba4523212c P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Becoming Leader. State: Replica: b00f244c69df401997ea99d492218a98, State: Running, Role: LEADER
I20240503 08:12:45.914809 31582 consensus_queue.cc:237] T 98207a56344d46209b670dba4523212c P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:45.917282 27855 catalog_manager.cc:5484] T 98207a56344d46209b670dba4523212c P b00f244c69df401997ea99d492218a98 reported cstate change: term changed from 0 to 1, leader changed from <none> to b00f244c69df401997ea99d492218a98 (127.20.230.72). New cstate: current_term: 1 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:12:46.014943 27855 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_extra_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:46.022982 30741 tablet_service.cc:1467] Processing CreateTablet for tablet 658865ee3f8d424ca5cd35e12e6d7c77 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=7f3bcd2f061e424892943977bcde8c80]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:46.023422 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 658865ee3f8d424ca5cd35e12e6d7c77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:46.023697 31008 tablet_service.cc:1467] Processing CreateTablet for tablet 7e0350bcc5fa4051bd10cc3de09639aa (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=7f3bcd2f061e424892943977bcde8c80]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:46.024114 31008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e0350bcc5fa4051bd10cc3de09639aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:46.023720 31007 tablet_service.cc:1467] Processing CreateTablet for tablet 658865ee3f8d424ca5cd35e12e6d7c77 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=7f3bcd2f061e424892943977bcde8c80]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:46.024516 31007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 658865ee3f8d424ca5cd35e12e6d7c77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:46.029407 31706 tablet_bootstrap.cc:492] T 658865ee3f8d424ca5cd35e12e6d7c77 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:46.029384 28111 tablet_service.cc:1467] Processing CreateTablet for tablet 658865ee3f8d424ca5cd35e12e6d7c77 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=7f3bcd2f061e424892943977bcde8c80]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:46.029832 28111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 658865ee3f8d424ca5cd35e12e6d7c77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:46.029784 31139 tablet_service.cc:1467] Processing CreateTablet for tablet 7e0350bcc5fa4051bd10cc3de09639aa (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=7f3bcd2f061e424892943977bcde8c80]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:46.030164 31139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e0350bcc5fa4051bd10cc3de09639aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:46.030969 31706 tablet_bootstrap.cc:654] T 658865ee3f8d424ca5cd35e12e6d7c77 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:46.032927 30740 tablet_service.cc:1467] Processing CreateTablet for tablet df31bd9e544a44f280db2b0d021000c2 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=7f3bcd2f061e424892943977bcde8c80]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:46.033258 30740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df31bd9e544a44f280db2b0d021000c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:46.033967 31706 tablet_bootstrap.cc:492] T 658865ee3f8d424ca5cd35e12e6d7c77 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:46.034092 31706 ts_tablet_manager.cc:1367] T 658865ee3f8d424ca5cd35e12e6d7c77 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 08:12:46.034165 30874 tablet_service.cc:1467] Processing CreateTablet for tablet df31bd9e544a44f280db2b0d021000c2 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=7f3bcd2f061e424892943977bcde8c80]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:46.034580 30874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df31bd9e544a44f280db2b0d021000c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:46.034794 31706 raft_consensus.cc:348] T 658865ee3f8d424ca5cd35e12e6d7c77 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:46.034987 31706 raft_consensus.cc:374] T 658865ee3f8d424ca5cd35e12e6d7c77 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:46.035074 31706 raft_consensus.cc:729] T 658865ee3f8d424ca5cd35e12e6d7c77 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:46.035276 31706 consensus_queue.cc:260] T 658865ee3f8d424ca5cd35e12e6d7c77 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:46.035817 31706 ts_tablet_manager.cc:1398] T 658865ee3f8d424ca5cd35e12e6d7c77 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:46.036072 31706 tablet_bootstrap.cc:492] T 7e0350bcc5fa4051bd10cc3de09639aa P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:46.037524 31706 tablet_bootstrap.cc:654] T 7e0350bcc5fa4051bd10cc3de09639aa P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:46.043648 27981 tablet_service.cc:1467] Processing CreateTablet for tablet 7e0350bcc5fa4051bd10cc3de09639aa (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=7f3bcd2f061e424892943977bcde8c80]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:46.044081 27981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e0350bcc5fa4051bd10cc3de09639aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:46.043648 27983 tablet_service.cc:1467] Processing CreateTablet for tablet df31bd9e544a44f280db2b0d021000c2 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=7f3bcd2f061e424892943977bcde8c80]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:46.046631 27983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df31bd9e544a44f280db2b0d021000c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:46.054287 27855 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:46.057713 31709 tablet_bootstrap.cc:492] T 658865ee3f8d424ca5cd35e12e6d7c77 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:46.059484 31709 tablet_bootstrap.cc:654] T 658865ee3f8d424ca5cd35e12e6d7c77 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:46.060376 30762 raft_consensus.cc:1261] T 98207a56344d46209b670dba4523212c P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:46.060973 31701 consensus_queue.cc:1035] T 98207a56344d46209b670dba4523212c P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:46.066259 30894 raft_consensus.cc:1261] T 98207a56344d46209b670dba4523212c P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:46.066856 31701 consensus_queue.cc:1035] T 98207a56344d46209b670dba4523212c P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:46.070206 31706 tablet_bootstrap.cc:492] T 7e0350bcc5fa4051bd10cc3de09639aa P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:46.070803 31706 ts_tablet_manager.cc:1367] T 7e0350bcc5fa4051bd10cc3de09639aa P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.035s user 0.006s sys 0.021s
I20240503 08:12:46.073199 31706 raft_consensus.cc:348] T 7e0350bcc5fa4051bd10cc3de09639aa P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:46.073549 31695 tablet_bootstrap.cc:492] T 7e0350bcc5fa4051bd10cc3de09639aa P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:46.074344 31706 raft_consensus.cc:374] T 7e0350bcc5fa4051bd10cc3de09639aa P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:46.074419 31706 raft_consensus.cc:729] T 7e0350bcc5fa4051bd10cc3de09639aa P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:46.074874 31706 consensus_queue.cc:260] T 7e0350bcc5fa4051bd10cc3de09639aa P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:46.075021 31695 tablet_bootstrap.cc:654] T 7e0350bcc5fa4051bd10cc3de09639aa P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:46.077970 31706 ts_tablet_manager.cc:1398] T 7e0350bcc5fa4051bd10cc3de09639aa P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:12:46.081452 31693 tablet_bootstrap.cc:492] T df31bd9e544a44f280db2b0d021000c2 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:46.082680 31709 tablet_bootstrap.cc:492] T 658865ee3f8d424ca5cd35e12e6d7c77 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:46.082834 31709 ts_tablet_manager.cc:1367] T 658865ee3f8d424ca5cd35e12e6d7c77 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.002s
I20240503 08:12:46.082999 31693 tablet_bootstrap.cc:654] T df31bd9e544a44f280db2b0d021000c2 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:46.083667 31709 raft_consensus.cc:348] T 658865ee3f8d424ca5cd35e12e6d7c77 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:46.083864 31709 raft_consensus.cc:374] T 658865ee3f8d424ca5cd35e12e6d7c77 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:46.083972 31709 raft_consensus.cc:729] T 658865ee3f8d424ca5cd35e12e6d7c77 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:46.084192 31709 consensus_queue.cc:260] T 658865ee3f8d424ca5cd35e12e6d7c77 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:46.084764 31709 ts_tablet_manager.cc:1398] T 658865ee3f8d424ca5cd35e12e6d7c77 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:46.094852 31694 tablet_bootstrap.cc:492] T 658865ee3f8d424ca5cd35e12e6d7c77 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:46.096396 31694 tablet_bootstrap.cc:654] T 658865ee3f8d424ca5cd35e12e6d7c77 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:46.096827 27855 catalog_manager.cc:5576] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747) does not have the latest schema for tablet 98207a56344d46209b670dba4523212c (table rebalancer_dynamic_table [id=54c2d37c1d4b49f89a8439e270fa56b5]). Expected version 1 got 0
I20240503 08:12:46.098804 31694 tablet_bootstrap.cc:492] T 658865ee3f8d424ca5cd35e12e6d7c77 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:46.099063 31694 ts_tablet_manager.cc:1367] T 658865ee3f8d424ca5cd35e12e6d7c77 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:12:46.099900 31694 raft_consensus.cc:348] T 658865ee3f8d424ca5cd35e12e6d7c77 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:46.100173 31694 raft_consensus.cc:374] T 658865ee3f8d424ca5cd35e12e6d7c77 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:46.100286 31694 raft_consensus.cc:729] T 658865ee3f8d424ca5cd35e12e6d7c77 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:46.100742 31695 tablet_bootstrap.cc:492] T 7e0350bcc5fa4051bd10cc3de09639aa P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:46.100813 31694 consensus_queue.cc:260] T 658865ee3f8d424ca5cd35e12e6d7c77 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:46.101037 31695 ts_tablet_manager.cc:1367] T 7e0350bcc5fa4051bd10cc3de09639aa P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.004s
I20240503 08:12:46.101522 31694 ts_tablet_manager.cc:1398] T 658865ee3f8d424ca5cd35e12e6d7c77 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:46.101907 31694 tablet_bootstrap.cc:492] T df31bd9e544a44f280db2b0d021000c2 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:46.102269 31695 raft_consensus.cc:348] T 7e0350bcc5fa4051bd10cc3de09639aa P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:46.102975 31693 tablet_bootstrap.cc:492] T df31bd9e544a44f280db2b0d021000c2 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:46.103070 31695 raft_consensus.cc:374] T 7e0350bcc5fa4051bd10cc3de09639aa P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:46.103150 31695 raft_consensus.cc:729] T 7e0350bcc5fa4051bd10cc3de09639aa P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:46.103240 31693 ts_tablet_manager.cc:1367] T df31bd9e544a44f280db2b0d021000c2 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.004s
I20240503 08:12:46.103382 31695 consensus_queue.cc:260] T 7e0350bcc5fa4051bd10cc3de09639aa P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:46.103551 31694 tablet_bootstrap.cc:654] T df31bd9e544a44f280db2b0d021000c2 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:46.104154 31693 raft_consensus.cc:348] T df31bd9e544a44f280db2b0d021000c2 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:46.104452 31693 raft_consensus.cc:374] T df31bd9e544a44f280db2b0d021000c2 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:46.104588 31693 raft_consensus.cc:729] T df31bd9e544a44f280db2b0d021000c2 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:46.105041 31695 ts_tablet_manager.cc:1398] T 7e0350bcc5fa4051bd10cc3de09639aa P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:46.105734 31693 consensus_queue.cc:260] T df31bd9e544a44f280db2b0d021000c2 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:46.110148 31718 tablet.cc:1722] T 98207a56344d46209b670dba4523212c P b00f244c69df401997ea99d492218a98: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:46.110811 31718 tablet.cc:1620] T 98207a56344d46209b670dba4523212c P b00f244c69df401997ea99d492218a98: MemRowSet was empty: no flush needed.
I20240503 08:12:46.118891 31699 raft_consensus.cc:482] T df31bd9e544a44f280db2b0d021000c2 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:46.119060 31699 raft_consensus.cc:504] T df31bd9e544a44f280db2b0d021000c2 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:46.119746 31699 leader_election.cc:288] T df31bd9e544a44f280db2b0d021000c2 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:12:46.121479 27855 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=54c2d37c1d4b49f89a8439e270fa56b5] alter complete (version 1)
I20240503 08:12:46.121680 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df31bd9e544a44f280db2b0d021000c2" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:12:46.122450 31693 ts_tablet_manager.cc:1398] T df31bd9e544a44f280db2b0d021000c2 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.019s user 0.000s sys 0.002s
I20240503 08:12:46.122444 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df31bd9e544a44f280db2b0d021000c2" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
W20240503 08:12:46.123474 30829 leader_election.cc:341] T df31bd9e544a44f280db2b0d021000c2 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): Illegal state: must be running to vote when last-logged opid is not known
W20240503 08:12:46.123898 30830 leader_election.cc:341] T df31bd9e544a44f280db2b0d021000c2 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:46.124035 30830 leader_election.cc:302] T df31bd9e544a44f280db2b0d021000c2 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c; no voters: 6d1d7de19459435fbde1999e3dfd226b, 833582ff68c94269be364ebd187a75db
I20240503 08:12:46.124557 31699 raft_consensus.cc:2726] T df31bd9e544a44f280db2b0d021000c2 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:12:46.127153 31719 tablet.cc:1722] T 98207a56344d46209b670dba4523212c P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:46.127602 31719 tablet.cc:1620] T 98207a56344d46209b670dba4523212c P ce1496a47d084365ad786cd09e472a2c: MemRowSet was empty: no flush needed.
I20240503 08:12:46.126391 31712 tablet_bootstrap.cc:492] T 7e0350bcc5fa4051bd10cc3de09639aa P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:46.130348 27855 catalog_manager.cc:5576] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747) does not have the latest schema for tablet 98207a56344d46209b670dba4523212c (table rebalancer_dynamic_table [id=54c2d37c1d4b49f89a8439e270fa56b5]). Expected version 1 got 0
I20240503 08:12:46.130896 31712 tablet_bootstrap.cc:654] T 7e0350bcc5fa4051bd10cc3de09639aa P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:46.133420 31712 tablet_bootstrap.cc:492] T 7e0350bcc5fa4051bd10cc3de09639aa P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:46.133610 31712 ts_tablet_manager.cc:1367] T 7e0350bcc5fa4051bd10cc3de09639aa P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240503 08:12:46.134294 31694 tablet_bootstrap.cc:492] T df31bd9e544a44f280db2b0d021000c2 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:46.134346 31712 raft_consensus.cc:348] T 7e0350bcc5fa4051bd10cc3de09639aa P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:46.134457 31694 ts_tablet_manager.cc:1367] T df31bd9e544a44f280db2b0d021000c2 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.005s
I20240503 08:12:46.134532 31712 raft_consensus.cc:374] T 7e0350bcc5fa4051bd10cc3de09639aa P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:46.134629 31712 raft_consensus.cc:729] T 7e0350bcc5fa4051bd10cc3de09639aa P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:46.134846 31712 consensus_queue.cc:260] T 7e0350bcc5fa4051bd10cc3de09639aa P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:46.135179 31694 raft_consensus.cc:348] T df31bd9e544a44f280db2b0d021000c2 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:46.135408 31694 raft_consensus.cc:374] T df31bd9e544a44f280db2b0d021000c2 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:46.135502 31694 raft_consensus.cc:729] T df31bd9e544a44f280db2b0d021000c2 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:46.135691 31694 consensus_queue.cc:260] T df31bd9e544a44f280db2b0d021000c2 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:46.135893 27856 catalog_manager.cc:5576] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725) does not have the latest schema for tablet 98207a56344d46209b670dba4523212c (table rebalancer_dynamic_table [id=54c2d37c1d4b49f89a8439e270fa56b5]). Expected version 1 got 0
I20240503 08:12:46.135982 31712 ts_tablet_manager.cc:1398] T 7e0350bcc5fa4051bd10cc3de09639aa P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:46.136113 31694 ts_tablet_manager.cc:1398] T df31bd9e544a44f280db2b0d021000c2 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:12:46.136229 31712 tablet_bootstrap.cc:492] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:46.137805 31712 tablet_bootstrap.cc:654] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:46.143230 31712 tablet_bootstrap.cc:492] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:46.143432 31712 ts_tablet_manager.cc:1367] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240503 08:12:46.144189 31712 raft_consensus.cc:348] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:46.143527 31722 tablet.cc:1722] T 98207a56344d46209b670dba4523212c P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:46.144479 31712 raft_consensus.cc:374] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:46.144620 31712 raft_consensus.cc:729] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:46.144757 31722 tablet.cc:1620] T 98207a56344d46209b670dba4523212c P 6d1d7de19459435fbde1999e3dfd226b: MemRowSet was empty: no flush needed.
I20240503 08:12:46.144974 31712 consensus_queue.cc:260] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:46.145426 27856 catalog_manager.cc:5576] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725) does not have the latest schema for tablet 98207a56344d46209b670dba4523212c (table rebalancer_dynamic_table [id=54c2d37c1d4b49f89a8439e270fa56b5]). Expected version 1 got 0
I20240503 08:12:46.145617 31712 ts_tablet_manager.cc:1398] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:46.151121 30686 raft_consensus.cc:482] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:46.151232 30686 raft_consensus.cc:504] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:46.151794 30686 leader_election.cc:288] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:12:46.152062 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df31bd9e544a44f280db2b0d021000c2" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:12:46.152269 30762 raft_consensus.cc:2454] T df31bd9e544a44f280db2b0d021000c2 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:46.152259 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df31bd9e544a44f280db2b0d021000c2" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:12:46.152438 30895 raft_consensus.cc:2454] T df31bd9e544a44f280db2b0d021000c2 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:46.152602 27938 leader_election.cc:302] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, 833582ff68c94269be364ebd187a75db; no voters:
I20240503 08:12:46.152839 30686 raft_consensus.cc:2781] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:46.152944 30686 raft_consensus.cc:482] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:46.153028 30686 raft_consensus.cc:3037] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:46.154887 30686 raft_consensus.cc:504] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:46.155424 30686 leader_election.cc:288] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Requested vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:12:46.155774 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df31bd9e544a44f280db2b0d021000c2" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:12:46.155956 30762 raft_consensus.cc:3037] T df31bd9e544a44f280db2b0d021000c2 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:46.155968 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df31bd9e544a44f280db2b0d021000c2" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:12:46.156127 30895 raft_consensus.cc:3037] T df31bd9e544a44f280db2b0d021000c2 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:46.158684 30895 raft_consensus.cc:2454] T df31bd9e544a44f280db2b0d021000c2 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:46.158672 30762 raft_consensus.cc:2454] T df31bd9e544a44f280db2b0d021000c2 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:46.159044 27936 leader_election.cc:302] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:46.159312 30686 raft_consensus.cc:2781] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:46.159444 30686 raft_consensus.cc:686] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Becoming Leader. State: Replica: 833582ff68c94269be364ebd187a75db, State: Running, Role: LEADER
I20240503 08:12:46.159675 30686 consensus_queue.cc:237] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:46.162289 27855 catalog_manager.cc:5484] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db reported cstate change: term changed from 0 to 1, leader changed from <none> to 833582ff68c94269be364ebd187a75db (127.20.230.65). New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:12:46.227520 27855 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:46.235389 31718 tablet.cc:1722] T 98207a56344d46209b670dba4523212c P b00f244c69df401997ea99d492218a98: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:46.235917 31718 tablet.cc:1620] T 98207a56344d46209b670dba4523212c P b00f244c69df401997ea99d492218a98: MemRowSet was empty: no flush needed.
I20240503 08:12:46.236487 31719 tablet.cc:1722] T 98207a56344d46209b670dba4523212c P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:46.236858 31719 tablet.cc:1620] T 98207a56344d46209b670dba4523212c P ce1496a47d084365ad786cd09e472a2c: MemRowSet was empty: no flush needed.
I20240503 08:12:46.244612 27856 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=54c2d37c1d4b49f89a8439e270fa56b5] alter complete (version 2)
I20240503 08:12:46.245632 31722 tablet.cc:1722] T 98207a56344d46209b670dba4523212c P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:46.246057 31722 tablet.cc:1620] T 98207a56344d46209b670dba4523212c P 6d1d7de19459435fbde1999e3dfd226b: MemRowSet was empty: no flush needed.
I20240503 08:12:46.260907 31668 raft_consensus.cc:482] T 658865ee3f8d424ca5cd35e12e6d7c77 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:46.261050 31668 raft_consensus.cc:504] T 658865ee3f8d424ca5cd35e12e6d7c77 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:46.262292 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "658865ee3f8d424ca5cd35e12e6d7c77" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:46.262549 31027 raft_consensus.cc:2454] T 658865ee3f8d424ca5cd35e12e6d7c77 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:46.262990 28067 leader_election.cc:302] T 658865ee3f8d424ca5cd35e12e6d7c77 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:46.261732 31668 leader_election.cc:288] T 658865ee3f8d424ca5cd35e12e6d7c77 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:46.263465 31668 raft_consensus.cc:2781] T 658865ee3f8d424ca5cd35e12e6d7c77 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:46.263561 31668 raft_consensus.cc:482] T 658865ee3f8d424ca5cd35e12e6d7c77 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:46.263661 31668 raft_consensus.cc:3037] T 658865ee3f8d424ca5cd35e12e6d7c77 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:46.269220 31668 raft_consensus.cc:504] T 658865ee3f8d424ca5cd35e12e6d7c77 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:46.271955 31668 leader_election.cc:288] T 658865ee3f8d424ca5cd35e12e6d7c77 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Requested vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:46.274114 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "658865ee3f8d424ca5cd35e12e6d7c77" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:12:46.274348 31027 raft_consensus.cc:3037] T 658865ee3f8d424ca5cd35e12e6d7c77 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:46.277098 31027 raft_consensus.cc:2454] T 658865ee3f8d424ca5cd35e12e6d7c77 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:12:46.277982 28067 leader_election.cc:302] T 658865ee3f8d424ca5cd35e12e6d7c77 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:46.278759 31726 raft_consensus.cc:2781] T 658865ee3f8d424ca5cd35e12e6d7c77 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:46.278909 31726 raft_consensus.cc:686] T 658865ee3f8d424ca5cd35e12e6d7c77 P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Becoming Leader. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Running, Role: LEADER
I20240503 08:12:46.279209 31726 consensus_queue.cc:237] T 658865ee3f8d424ca5cd35e12e6d7c77 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:46.283016 27856 catalog_manager.cc:5484] T 658865ee3f8d424ca5cd35e12e6d7c77 P 1b411c6bb25549368041773d7b710d48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b411c6bb25549368041773d7b710d48 (127.20.230.66). New cstate: current_term: 1 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:12:46.286197 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "658865ee3f8d424ca5cd35e12e6d7c77" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:12:46.286314 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "658865ee3f8d424ca5cd35e12e6d7c77" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:12:46.286433 30762 raft_consensus.cc:2454] T 658865ee3f8d424ca5cd35e12e6d7c77 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:46.341575 30686 raft_consensus.cc:482] T 7e0350bcc5fa4051bd10cc3de09639aa P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:46.341742 30686 raft_consensus.cc:504] T 7e0350bcc5fa4051bd10cc3de09639aa P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:46.342473 30686 leader_election.cc:288] T 7e0350bcc5fa4051bd10cc3de09639aa P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:46.342813 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e0350bcc5fa4051bd10cc3de09639aa" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:46.343005 31027 raft_consensus.cc:2454] T 7e0350bcc5fa4051bd10cc3de09639aa P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:46.343436 27937 leader_election.cc:302] T 7e0350bcc5fa4051bd10cc3de09639aa P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:46.343780 30686 raft_consensus.cc:2781] T 7e0350bcc5fa4051bd10cc3de09639aa P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:46.343888 30686 raft_consensus.cc:482] T 7e0350bcc5fa4051bd10cc3de09639aa P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:46.343995 30686 raft_consensus.cc:3037] T 7e0350bcc5fa4051bd10cc3de09639aa P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:46.347658 30686 raft_consensus.cc:504] T 7e0350bcc5fa4051bd10cc3de09639aa P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:46.348277 30686 leader_election.cc:288] T 7e0350bcc5fa4051bd10cc3de09639aa P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Requested vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:46.348570 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e0350bcc5fa4051bd10cc3de09639aa" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:12:46.348732 31027 raft_consensus.cc:3037] T 7e0350bcc5fa4051bd10cc3de09639aa P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:46.350788 31027 raft_consensus.cc:2454] T 7e0350bcc5fa4051bd10cc3de09639aa P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:46.351248 27937 leader_election.cc:302] T 7e0350bcc5fa4051bd10cc3de09639aa P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:46.351804 30686 raft_consensus.cc:2781] T 7e0350bcc5fa4051bd10cc3de09639aa P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:46.351998 30686 raft_consensus.cc:686] T 7e0350bcc5fa4051bd10cc3de09639aa P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Becoming Leader. State: Replica: 833582ff68c94269be364ebd187a75db, State: Running, Role: LEADER
I20240503 08:12:46.351961 27856 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:46.352290 30686 consensus_queue.cc:237] T 7e0350bcc5fa4051bd10cc3de09639aa P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:46.353122 31161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e0350bcc5fa4051bd10cc3de09639aa" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:12:46.353363 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e0350bcc5fa4051bd10cc3de09639aa" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:12:46.353365 31161 raft_consensus.cc:2454] T 7e0350bcc5fa4051bd10cc3de09639aa P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:46.359423 27856 catalog_manager.cc:5484] T 7e0350bcc5fa4051bd10cc3de09639aa P 833582ff68c94269be364ebd187a75db reported cstate change: term changed from 0 to 1, leader changed from <none> to 833582ff68c94269be364ebd187a75db (127.20.230.65). New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:12:46.361199 31718 tablet.cc:1722] T 98207a56344d46209b670dba4523212c P b00f244c69df401997ea99d492218a98: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:46.362236 31719 tablet.cc:1722] T 98207a56344d46209b670dba4523212c P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:46.365548 31722 tablet.cc:1722] T 98207a56344d46209b670dba4523212c P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:46.371644 27855 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table_new_name [id=54c2d37c1d4b49f89a8439e270fa56b5] alter complete (version 3)
I20240503 08:12:46.476317 27856 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:46.476495 27856 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:46.479292 27856 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet 98207a56344d46209b670dba4523212c
I20240503 08:12:46.480049 30874 tablet_service.cc:1514] Processing DeleteTablet for tablet 98207a56344d46209b670dba4523212c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:46 UTC) from {username='slave'} at 127.0.0.1:34426
I20240503 08:12:46.480103 31139 tablet_service.cc:1514] Processing DeleteTablet for tablet 98207a56344d46209b670dba4523212c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:46 UTC) from {username='slave'} at 127.0.0.1:45814
I20240503 08:12:46.480211 30741 tablet_service.cc:1514] Processing DeleteTablet for tablet 98207a56344d46209b670dba4523212c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:46 UTC) from {username='slave'} at 127.0.0.1:46794
I20240503 08:12:46.480875 31733 tablet_replica.cc:330] T 98207a56344d46209b670dba4523212c P b00f244c69df401997ea99d492218a98: stopping tablet replica
I20240503 08:12:46.481015 31734 tablet_replica.cc:330] T 98207a56344d46209b670dba4523212c P 6d1d7de19459435fbde1999e3dfd226b: stopping tablet replica
I20240503 08:12:46.481092 31732 tablet_replica.cc:330] T 98207a56344d46209b670dba4523212c P ce1496a47d084365ad786cd09e472a2c: stopping tablet replica
I20240503 08:12:46.481108 31733 raft_consensus.cc:2229] T 98207a56344d46209b670dba4523212c P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:46.481261 31734 raft_consensus.cc:2229] T 98207a56344d46209b670dba4523212c P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:46.481372 31732 raft_consensus.cc:2229] T 98207a56344d46209b670dba4523212c P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:46.481532 31733 raft_consensus.cc:2258] T 98207a56344d46209b670dba4523212c P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:46.481626 31734 raft_consensus.cc:2258] T 98207a56344d46209b670dba4523212c P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:46.481657 31732 raft_consensus.cc:2258] T 98207a56344d46209b670dba4523212c P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:46.482407 31733 ts_tablet_manager.cc:1875] T 98207a56344d46209b670dba4523212c P b00f244c69df401997ea99d492218a98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:46.482622 31734 ts_tablet_manager.cc:1875] T 98207a56344d46209b670dba4523212c P 6d1d7de19459435fbde1999e3dfd226b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:46.482685 31732 ts_tablet_manager.cc:1875] T 98207a56344d46209b670dba4523212c P ce1496a47d084365ad786cd09e472a2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:46.492759 31733 ts_tablet_manager.cc:1888] T 98207a56344d46209b670dba4523212c P b00f244c69df401997ea99d492218a98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:46.492941 31733 log.cc:1196] T 98207a56344d46209b670dba4523212c P b00f244c69df401997ea99d492218a98: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal/wals/98207a56344d46209b670dba4523212c
I20240503 08:12:46.493430 31733 ts_tablet_manager.cc:1909] T 98207a56344d46209b670dba4523212c P b00f244c69df401997ea99d492218a98: Deleting consensus metadata
I20240503 08:12:46.494899 27846 catalog_manager.cc:4830] TS b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): tablet 98207a56344d46209b670dba4523212c (table rebalancer_dynamic_table_new_name [id=54c2d37c1d4b49f89a8439e270fa56b5]) successfully deleted
I20240503 08:12:46.497272 31734 ts_tablet_manager.cc:1888] T 98207a56344d46209b670dba4523212c P 6d1d7de19459435fbde1999e3dfd226b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:46.497390 31734 log.cc:1196] T 98207a56344d46209b670dba4523212c P 6d1d7de19459435fbde1999e3dfd226b: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal/wals/98207a56344d46209b670dba4523212c
I20240503 08:12:46.497586 31732 ts_tablet_manager.cc:1888] T 98207a56344d46209b670dba4523212c P ce1496a47d084365ad786cd09e472a2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:46.497696 31732 log.cc:1196] T 98207a56344d46209b670dba4523212c P ce1496a47d084365ad786cd09e472a2c: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal/wals/98207a56344d46209b670dba4523212c
I20240503 08:12:46.497813 31734 ts_tablet_manager.cc:1909] T 98207a56344d46209b670dba4523212c P 6d1d7de19459435fbde1999e3dfd226b: Deleting consensus metadata
I20240503 08:12:46.498167 31732 ts_tablet_manager.cc:1909] T 98207a56344d46209b670dba4523212c P ce1496a47d084365ad786cd09e472a2c: Deleting consensus metadata
I20240503 08:12:46.498767 27848 catalog_manager.cc:4830] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): tablet 98207a56344d46209b670dba4523212c (table rebalancer_dynamic_table_new_name [id=54c2d37c1d4b49f89a8439e270fa56b5]) successfully deleted
I20240503 08:12:46.498993 27846 catalog_manager.cc:4830] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): tablet 98207a56344d46209b670dba4523212c (table rebalancer_dynamic_table_new_name [id=54c2d37c1d4b49f89a8439e270fa56b5]) successfully deleted
I20240503 08:12:46.581845 27856 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:46.589543 28372 tablet_service.cc:1467] Processing CreateTablet for tablet 4eeaef7b0e3e489ea92bdde6a2b24835 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a860bd4dbc4546d683170d4cb58947a8]), partition=
I20240503 08:12:46.589752 31007 tablet_service.cc:1467] Processing CreateTablet for tablet 4eeaef7b0e3e489ea92bdde6a2b24835 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a860bd4dbc4546d683170d4cb58947a8]), partition=
I20240503 08:12:46.589967 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4eeaef7b0e3e489ea92bdde6a2b24835. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:46.590355 30741 tablet_service.cc:1467] Processing CreateTablet for tablet 4eeaef7b0e3e489ea92bdde6a2b24835 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a860bd4dbc4546d683170d4cb58947a8]), partition=
I20240503 08:12:46.590477 31007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4eeaef7b0e3e489ea92bdde6a2b24835. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:46.590734 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4eeaef7b0e3e489ea92bdde6a2b24835. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:46.596624 31694 tablet_bootstrap.cc:492] T 4eeaef7b0e3e489ea92bdde6a2b24835 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:46.598206 31694 tablet_bootstrap.cc:654] T 4eeaef7b0e3e489ea92bdde6a2b24835 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:46.603646 31735 tablet_bootstrap.cc:492] T 4eeaef7b0e3e489ea92bdde6a2b24835 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:46.604748 31735 tablet_bootstrap.cc:654] T 4eeaef7b0e3e489ea92bdde6a2b24835 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:46.606827 31736 tablet_bootstrap.cc:492] T 4eeaef7b0e3e489ea92bdde6a2b24835 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:46.607158 31735 tablet_bootstrap.cc:492] T 4eeaef7b0e3e489ea92bdde6a2b24835 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:46.607335 31735 ts_tablet_manager.cc:1367] T 4eeaef7b0e3e489ea92bdde6a2b24835 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240503 08:12:46.607985 31735 raft_consensus.cc:348] T 4eeaef7b0e3e489ea92bdde6a2b24835 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:46.608139 31735 raft_consensus.cc:374] T 4eeaef7b0e3e489ea92bdde6a2b24835 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:46.608237 31735 raft_consensus.cc:729] T 4eeaef7b0e3e489ea92bdde6a2b24835 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:46.608431 31735 consensus_queue.cc:260] T 4eeaef7b0e3e489ea92bdde6a2b24835 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:46.608711 31736 tablet_bootstrap.cc:654] T 4eeaef7b0e3e489ea92bdde6a2b24835 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:46.608928 31735 ts_tablet_manager.cc:1398] T 4eeaef7b0e3e489ea92bdde6a2b24835 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:12:46.609179 31694 tablet_bootstrap.cc:492] T 4eeaef7b0e3e489ea92bdde6a2b24835 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:46.609345 31694 ts_tablet_manager.cc:1367] T 4eeaef7b0e3e489ea92bdde6a2b24835 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20240503 08:12:46.610059 31694 raft_consensus.cc:348] T 4eeaef7b0e3e489ea92bdde6a2b24835 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:46.610517 31694 raft_consensus.cc:374] T 4eeaef7b0e3e489ea92bdde6a2b24835 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:46.610891 31694 raft_consensus.cc:729] T 4eeaef7b0e3e489ea92bdde6a2b24835 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:46.611116 31694 consensus_queue.cc:260] T 4eeaef7b0e3e489ea92bdde6a2b24835 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:46.611598 31694 ts_tablet_manager.cc:1398] T 4eeaef7b0e3e489ea92bdde6a2b24835 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:46.614486 31736 tablet_bootstrap.cc:492] T 4eeaef7b0e3e489ea92bdde6a2b24835 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:46.614761 31736 ts_tablet_manager.cc:1367] T 4eeaef7b0e3e489ea92bdde6a2b24835 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240503 08:12:46.615489 31736 raft_consensus.cc:348] T 4eeaef7b0e3e489ea92bdde6a2b24835 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:46.615727 31736 raft_consensus.cc:374] T 4eeaef7b0e3e489ea92bdde6a2b24835 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:46.615825 31736 raft_consensus.cc:729] T 4eeaef7b0e3e489ea92bdde6a2b24835 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:46.616034 31736 consensus_queue.cc:260] T 4eeaef7b0e3e489ea92bdde6a2b24835 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:46.616640 31736 ts_tablet_manager.cc:1398] T 4eeaef7b0e3e489ea92bdde6a2b24835 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:46.620935 30686 consensus_queue.cc:1035] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:46.622143 31601 raft_consensus.cc:482] T 4eeaef7b0e3e489ea92bdde6a2b24835 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:46.622298 31601 raft_consensus.cc:504] T 4eeaef7b0e3e489ea92bdde6a2b24835 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:46.623044 31601 leader_election.cc:288] T 4eeaef7b0e3e489ea92bdde6a2b24835 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:46.624480 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eeaef7b0e3e489ea92bdde6a2b24835" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:46.624714 31027 raft_consensus.cc:2454] T 4eeaef7b0e3e489ea92bdde6a2b24835 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:12:46.625149 28327 leader_election.cc:302] T 4eeaef7b0e3e489ea92bdde6a2b24835 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:46.625728 31601 raft_consensus.cc:2781] T 4eeaef7b0e3e489ea92bdde6a2b24835 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:46.625860 31601 raft_consensus.cc:482] T 4eeaef7b0e3e489ea92bdde6a2b24835 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:46.625967 31601 raft_consensus.cc:3037] T 4eeaef7b0e3e489ea92bdde6a2b24835 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:46.628610 31601 raft_consensus.cc:504] T 4eeaef7b0e3e489ea92bdde6a2b24835 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:46.629135 31601 leader_election.cc:288] T 4eeaef7b0e3e489ea92bdde6a2b24835 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Requested vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:46.630106 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eeaef7b0e3e489ea92bdde6a2b24835" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:12:46.630297 31027 raft_consensus.cc:3037] T 4eeaef7b0e3e489ea92bdde6a2b24835 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:46.632876 31027 raft_consensus.cc:2454] T 4eeaef7b0e3e489ea92bdde6a2b24835 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:12:46.634327 28327 leader_election.cc:302] T 4eeaef7b0e3e489ea92bdde6a2b24835 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:46.634593 31601 raft_consensus.cc:2781] T 4eeaef7b0e3e489ea92bdde6a2b24835 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:46.635015 31601 raft_consensus.cc:686] T 4eeaef7b0e3e489ea92bdde6a2b24835 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Becoming Leader. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Running, Role: LEADER
I20240503 08:12:46.635325 31601 consensus_queue.cc:237] T 4eeaef7b0e3e489ea92bdde6a2b24835 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:46.635555 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eeaef7b0e3e489ea92bdde6a2b24835" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:12:46.635735 30762 raft_consensus.cc:3037] T 4eeaef7b0e3e489ea92bdde6a2b24835 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:46.638021 30762 raft_consensus.cc:2454] T 4eeaef7b0e3e489ea92bdde6a2b24835 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:12:46.635504 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eeaef7b0e3e489ea92bdde6a2b24835" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:12:46.638340 30761 raft_consensus.cc:2362] T 4eeaef7b0e3e489ea92bdde6a2b24835 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1. Re-sending same reply.
I20240503 08:12:46.638635 27859 catalog_manager.cc:5484] T 4eeaef7b0e3e489ea92bdde6a2b24835 P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68). New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:12:46.645190 31705 consensus_queue.cc:1035] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:46.706535 30761 raft_consensus.cc:3037] T 658865ee3f8d424ca5cd35e12e6d7c77 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:46.709031 31731 consensus_queue.cc:1035] T 658865ee3f8d424ca5cd35e12e6d7c77 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:46.713593 31668 consensus_queue.cc:1035] T 658865ee3f8d424ca5cd35e12e6d7c77 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:46.753376 27859 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:46.759335 30761 raft_consensus.cc:1261] T 4eeaef7b0e3e489ea92bdde6a2b24835 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:46.759387 31027 raft_consensus.cc:1261] T 4eeaef7b0e3e489ea92bdde6a2b24835 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:46.759953 31601 consensus_queue.cc:1035] T 4eeaef7b0e3e489ea92bdde6a2b24835 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:46.760216 31743 consensus_queue.cc:1035] T 4eeaef7b0e3e489ea92bdde6a2b24835 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:46.763931 27856 catalog_manager.cc:5576] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725) does not have the latest schema for tablet 4eeaef7b0e3e489ea92bdde6a2b24835 (table rebalancer_dynamic_table [id=a860bd4dbc4546d683170d4cb58947a8]). Expected version 1 got 0
I20240503 08:12:46.764734 31722 tablet.cc:1722] T 4eeaef7b0e3e489ea92bdde6a2b24835 P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:46.765218 31722 tablet.cc:1620] T 4eeaef7b0e3e489ea92bdde6a2b24835 P 6d1d7de19459435fbde1999e3dfd226b: MemRowSet was empty: no flush needed.
I20240503 08:12:46.765507 31755 tablet.cc:1722] T 4eeaef7b0e3e489ea92bdde6a2b24835 P d09a11a03989421b9f28019dc1c14cc2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:46.765942 31159 raft_consensus.cc:3037] T 7e0350bcc5fa4051bd10cc3de09639aa P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:46.769609 30686 consensus_queue.cc:1035] T 7e0350bcc5fa4051bd10cc3de09639aa P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:46.769788 31755 tablet.cc:1620] T 4eeaef7b0e3e489ea92bdde6a2b24835 P d09a11a03989421b9f28019dc1c14cc2: MemRowSet was empty: no flush needed.
I20240503 08:12:46.769946 27856 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=a860bd4dbc4546d683170d4cb58947a8] alter complete (version 1)
I20240503 08:12:46.774152 31745 consensus_queue.cc:1035] T 7e0350bcc5fa4051bd10cc3de09639aa P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:46.784044 27856 catalog_manager.cc:5576] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915) does not have the latest schema for tablet 4eeaef7b0e3e489ea92bdde6a2b24835 (table rebalancer_dynamic_table [id=a860bd4dbc4546d683170d4cb58947a8]). Expected version 1 got 0
I20240503 08:12:46.784229 31758 tablet.cc:1722] T 4eeaef7b0e3e489ea92bdde6a2b24835 P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:46.784883 31758 tablet.cc:1620] T 4eeaef7b0e3e489ea92bdde6a2b24835 P ced21ca983b749d0b4d4f6a0e2426bf8: MemRowSet was empty: no flush needed.
I20240503 08:12:46.875864 27856 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:46.883661 31755 tablet.cc:1722] T 4eeaef7b0e3e489ea92bdde6a2b24835 P d09a11a03989421b9f28019dc1c14cc2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:46.884174 31755 tablet.cc:1620] T 4eeaef7b0e3e489ea92bdde6a2b24835 P d09a11a03989421b9f28019dc1c14cc2: MemRowSet was empty: no flush needed.
I20240503 08:12:46.884183 31758 tablet.cc:1722] T 4eeaef7b0e3e489ea92bdde6a2b24835 P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:46.884173 31722 tablet.cc:1722] T 4eeaef7b0e3e489ea92bdde6a2b24835 P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:46.884639 31722 tablet.cc:1620] T 4eeaef7b0e3e489ea92bdde6a2b24835 P 6d1d7de19459435fbde1999e3dfd226b: MemRowSet was empty: no flush needed.
I20240503 08:12:46.884632 31758 tablet.cc:1620] T 4eeaef7b0e3e489ea92bdde6a2b24835 P ced21ca983b749d0b4d4f6a0e2426bf8: MemRowSet was empty: no flush needed.
I20240503 08:12:46.892650 27856 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=a860bd4dbc4546d683170d4cb58947a8] alter complete (version 2)
I20240503 08:12:46.929342 27856 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_extra_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:46.937186 28372 tablet_service.cc:1467] Processing CreateTablet for tablet 8425cfe5c6134a60852e12f9af55227b (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=6db67fd7274948c0858a325398f6acbc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:46.937639 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8425cfe5c6134a60852e12f9af55227b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:46.937938 30741 tablet_service.cc:1467] Processing CreateTablet for tablet 71fa62206f744de0b97e3aa9ea753e37 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=6db67fd7274948c0858a325398f6acbc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:46.938369 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71fa62206f744de0b97e3aa9ea753e37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:46.938688 27981 tablet_service.cc:1467] Processing CreateTablet for tablet 8425cfe5c6134a60852e12f9af55227b (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=6db67fd7274948c0858a325398f6acbc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:46.939065 27981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8425cfe5c6134a60852e12f9af55227b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:46.942651 31139 tablet_service.cc:1467] Processing CreateTablet for tablet 8425cfe5c6134a60852e12f9af55227b (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=6db67fd7274948c0858a325398f6acbc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:46.942790 31141 tablet_service.cc:1467] Processing CreateTablet for tablet 720f36100d794158be13ff02595e6ddf (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=6db67fd7274948c0858a325398f6acbc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:46.942759 31140 tablet_service.cc:1467] Processing CreateTablet for tablet 71fa62206f744de0b97e3aa9ea753e37 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=6db67fd7274948c0858a325398f6acbc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:46.943071 31139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8425cfe5c6134a60852e12f9af55227b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:46.943291 31141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 720f36100d794158be13ff02595e6ddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:46.944065 30740 tablet_service.cc:1467] Processing CreateTablet for tablet 720f36100d794158be13ff02595e6ddf (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=6db67fd7274948c0858a325398f6acbc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:46.944424 30740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 720f36100d794158be13ff02595e6ddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:46.945135 28373 tablet_service.cc:1467] Processing CreateTablet for tablet 71fa62206f744de0b97e3aa9ea753e37 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=6db67fd7274948c0858a325398f6acbc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:46.945533 28373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71fa62206f744de0b97e3aa9ea753e37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:46.945641 31760 tablet_bootstrap.cc:492] T 8425cfe5c6134a60852e12f9af55227b P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:46.947229 31760 tablet_bootstrap.cc:654] T 8425cfe5c6134a60852e12f9af55227b P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:46.949079 31760 tablet_bootstrap.cc:492] T 8425cfe5c6134a60852e12f9af55227b P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:46.949198 31760 ts_tablet_manager.cc:1367] T 8425cfe5c6134a60852e12f9af55227b P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 08:12:46.949626 31736 tablet_bootstrap.cc:492] T 71fa62206f744de0b97e3aa9ea753e37 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:46.949926 31760 raft_consensus.cc:348] T 8425cfe5c6134a60852e12f9af55227b P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:46.950105 31760 raft_consensus.cc:374] T 8425cfe5c6134a60852e12f9af55227b P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:46.950204 31760 raft_consensus.cc:729] T 8425cfe5c6134a60852e12f9af55227b P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:46.950410 31760 consensus_queue.cc:260] T 8425cfe5c6134a60852e12f9af55227b P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:46.950867 31694 tablet_bootstrap.cc:492] T 720f36100d794158be13ff02595e6ddf P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:46.951110 31736 tablet_bootstrap.cc:654] T 71fa62206f744de0b97e3aa9ea753e37 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:46.951781 31140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71fa62206f744de0b97e3aa9ea753e37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:46.951978 31760 ts_tablet_manager.cc:1398] T 8425cfe5c6134a60852e12f9af55227b P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:46.952770 31694 tablet_bootstrap.cc:654] T 720f36100d794158be13ff02595e6ddf P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:46.961608 28372 tablet_service.cc:1467] Processing CreateTablet for tablet 720f36100d794158be13ff02595e6ddf (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=6db67fd7274948c0858a325398f6acbc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:46.962054 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 720f36100d794158be13ff02595e6ddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:46.969204 31736 tablet_bootstrap.cc:492] T 71fa62206f744de0b97e3aa9ea753e37 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:46.969388 31736 ts_tablet_manager.cc:1367] T 71fa62206f744de0b97e3aa9ea753e37 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.015s
I20240503 08:12:46.969431 31764 tablet_bootstrap.cc:492] T 71fa62206f744de0b97e3aa9ea753e37 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:46.970168 31736 raft_consensus.cc:348] T 71fa62206f744de0b97e3aa9ea753e37 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:46.970470 31736 raft_consensus.cc:374] T 71fa62206f744de0b97e3aa9ea753e37 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:46.970594 31736 raft_consensus.cc:729] T 71fa62206f744de0b97e3aa9ea753e37 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:46.970861 31736 consensus_queue.cc:260] T 71fa62206f744de0b97e3aa9ea753e37 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:46.971519 31736 ts_tablet_manager.cc:1398] T 71fa62206f744de0b97e3aa9ea753e37 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240503 08:12:46.971971 31736 tablet_bootstrap.cc:492] T 8425cfe5c6134a60852e12f9af55227b P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:46.972193 31764 tablet_bootstrap.cc:654] T 71fa62206f744de0b97e3aa9ea753e37 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:46.973577 31736 tablet_bootstrap.cc:654] T 8425cfe5c6134a60852e12f9af55227b P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:46.979271 31736 tablet_bootstrap.cc:492] T 8425cfe5c6134a60852e12f9af55227b P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:46.979461 31736 ts_tablet_manager.cc:1367] T 8425cfe5c6134a60852e12f9af55227b P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20240503 08:12:46.980456 31736 raft_consensus.cc:348] T 8425cfe5c6134a60852e12f9af55227b P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:46.980762 31736 raft_consensus.cc:374] T 8425cfe5c6134a60852e12f9af55227b P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:46.980980 31736 raft_consensus.cc:729] T 8425cfe5c6134a60852e12f9af55227b P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:46.981247 31736 consensus_queue.cc:260] T 8425cfe5c6134a60852e12f9af55227b P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:46.982041 31736 ts_tablet_manager.cc:1398] T 8425cfe5c6134a60852e12f9af55227b P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:46.983207 31764 tablet_bootstrap.cc:492] T 71fa62206f744de0b97e3aa9ea753e37 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:46.983348 31764 ts_tablet_manager.cc:1367] T 71fa62206f744de0b97e3aa9ea753e37 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.000s
I20240503 08:12:46.984009 31764 raft_consensus.cc:348] T 71fa62206f744de0b97e3aa9ea753e37 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:46.984158 31764 raft_consensus.cc:374] T 71fa62206f744de0b97e3aa9ea753e37 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:46.984275 31764 raft_consensus.cc:729] T 71fa62206f744de0b97e3aa9ea753e37 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:46.984469 31764 consensus_queue.cc:260] T 71fa62206f744de0b97e3aa9ea753e37 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:46.986467 31764 ts_tablet_manager.cc:1398] T 71fa62206f744de0b97e3aa9ea753e37 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:46.986781 31764 tablet_bootstrap.cc:492] T 720f36100d794158be13ff02595e6ddf P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:46.988493 31764 tablet_bootstrap.cc:654] T 720f36100d794158be13ff02595e6ddf P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:46.988953 31736 tablet_bootstrap.cc:492] T 720f36100d794158be13ff02595e6ddf P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:46.990479 31736 tablet_bootstrap.cc:654] T 720f36100d794158be13ff02595e6ddf P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:46.991951 31694 tablet_bootstrap.cc:492] T 720f36100d794158be13ff02595e6ddf P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:46.992089 31694 ts_tablet_manager.cc:1367] T 720f36100d794158be13ff02595e6ddf P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.005s
I20240503 08:12:46.992954 31694 raft_consensus.cc:348] T 720f36100d794158be13ff02595e6ddf P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:46.993162 31694 raft_consensus.cc:374] T 720f36100d794158be13ff02595e6ddf P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:46.993249 31694 raft_consensus.cc:729] T 720f36100d794158be13ff02595e6ddf P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:46.993460 31694 consensus_queue.cc:260] T 720f36100d794158be13ff02595e6ddf P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:46.994009 31694 ts_tablet_manager.cc:1398] T 720f36100d794158be13ff02595e6ddf P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:46.994881 31694 tablet_bootstrap.cc:492] T 71fa62206f744de0b97e3aa9ea753e37 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:46.995231 31736 tablet_bootstrap.cc:492] T 720f36100d794158be13ff02595e6ddf P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:46.995378 31736 ts_tablet_manager.cc:1367] T 720f36100d794158be13ff02595e6ddf P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 08:12:46.996279 31736 raft_consensus.cc:348] T 720f36100d794158be13ff02595e6ddf P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:46.996469 31736 raft_consensus.cc:374] T 720f36100d794158be13ff02595e6ddf P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:46.996547 31736 raft_consensus.cc:729] T 720f36100d794158be13ff02595e6ddf P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:46.996752 31736 consensus_queue.cc:260] T 720f36100d794158be13ff02595e6ddf P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:46.997532 31694 tablet_bootstrap.cc:654] T 71fa62206f744de0b97e3aa9ea753e37 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:46.998076 31736 ts_tablet_manager.cc:1398] T 720f36100d794158be13ff02595e6ddf P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:46.996882 27859 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:47.005101 31764 tablet_bootstrap.cc:492] T 720f36100d794158be13ff02595e6ddf P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:47.005292 31764 ts_tablet_manager.cc:1367] T 720f36100d794158be13ff02595e6ddf P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.004s
I20240503 08:12:47.006269 31694 tablet_bootstrap.cc:492] T 71fa62206f744de0b97e3aa9ea753e37 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:47.006215 31764 raft_consensus.cc:348] T 720f36100d794158be13ff02595e6ddf P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:47.006461 31764 raft_consensus.cc:374] T 720f36100d794158be13ff02595e6ddf P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:47.006580 31764 raft_consensus.cc:729] T 720f36100d794158be13ff02595e6ddf P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:47.006507 31694 ts_tablet_manager.cc:1367] T 71fa62206f744de0b97e3aa9ea753e37 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.001s
I20240503 08:12:47.006839 31764 consensus_queue.cc:260] T 720f36100d794158be13ff02595e6ddf P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:47.007369 31764 ts_tablet_manager.cc:1398] T 720f36100d794158be13ff02595e6ddf P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:47.007730 31764 tablet_bootstrap.cc:492] T 8425cfe5c6134a60852e12f9af55227b P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:47.007880 31755 tablet.cc:1722] T 4eeaef7b0e3e489ea92bdde6a2b24835 P d09a11a03989421b9f28019dc1c14cc2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:47.009240 31764 tablet_bootstrap.cc:654] T 8425cfe5c6134a60852e12f9af55227b P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:47.009251 31758 tablet.cc:1722] T 4eeaef7b0e3e489ea92bdde6a2b24835 P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:47.010334 31694 raft_consensus.cc:348] T 71fa62206f744de0b97e3aa9ea753e37 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:47.010545 31694 raft_consensus.cc:374] T 71fa62206f744de0b97e3aa9ea753e37 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:47.010614 31694 raft_consensus.cc:729] T 71fa62206f744de0b97e3aa9ea753e37 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:47.010932 31694 consensus_queue.cc:260] T 71fa62206f744de0b97e3aa9ea753e37 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:47.011438 31694 ts_tablet_manager.cc:1398] T 71fa62206f744de0b97e3aa9ea753e37 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240503 08:12:47.014421 31722 tablet.cc:1722] T 4eeaef7b0e3e489ea92bdde6a2b24835 P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:47.016671 27859 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table_new_name [id=a860bd4dbc4546d683170d4cb58947a8] alter complete (version 3)
I20240503 08:12:47.021119 31764 tablet_bootstrap.cc:492] T 8425cfe5c6134a60852e12f9af55227b P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:47.021282 31764 ts_tablet_manager.cc:1367] T 8425cfe5c6134a60852e12f9af55227b P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240503 08:12:47.021970 31764 raft_consensus.cc:348] T 8425cfe5c6134a60852e12f9af55227b P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:47.022157 31764 raft_consensus.cc:374] T 8425cfe5c6134a60852e12f9af55227b P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:47.022264 31764 raft_consensus.cc:729] T 8425cfe5c6134a60852e12f9af55227b P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:47.022490 31764 consensus_queue.cc:260] T 8425cfe5c6134a60852e12f9af55227b P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:47.022974 31764 ts_tablet_manager.cc:1398] T 8425cfe5c6134a60852e12f9af55227b P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:47.027289 31688 raft_consensus.cc:482] T 71fa62206f744de0b97e3aa9ea753e37 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:47.027459 31688 raft_consensus.cc:504] T 71fa62206f744de0b97e3aa9ea753e37 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:47.028023 31688 leader_election.cc:288] T 71fa62206f744de0b97e3aa9ea753e37 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:47.028407 28393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71fa62206f744de0b97e3aa9ea753e37" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:47.028436 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71fa62206f744de0b97e3aa9ea753e37" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:12:47.028569 28393 raft_consensus.cc:2454] T 71fa62206f744de0b97e3aa9ea753e37 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:12:47.028599 31159 raft_consensus.cc:2454] T 71fa62206f744de0b97e3aa9ea753e37 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:12:47.028925 30697 leader_election.cc:302] T 71fa62206f744de0b97e3aa9ea753e37 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:47.029174 31688 raft_consensus.cc:2781] T 71fa62206f744de0b97e3aa9ea753e37 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:47.029264 31688 raft_consensus.cc:482] T 71fa62206f744de0b97e3aa9ea753e37 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:47.029342 31688 raft_consensus.cc:3037] T 71fa62206f744de0b97e3aa9ea753e37 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:47.031338 31688 raft_consensus.cc:504] T 71fa62206f744de0b97e3aa9ea753e37 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:47.031872 31688 leader_election.cc:288] T 71fa62206f744de0b97e3aa9ea753e37 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Requested vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:47.032178 28393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71fa62206f744de0b97e3aa9ea753e37" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:12:47.032286 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71fa62206f744de0b97e3aa9ea753e37" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:12:47.032392 28393 raft_consensus.cc:3037] T 71fa62206f744de0b97e3aa9ea753e37 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:47.032474 31159 raft_consensus.cc:3037] T 71fa62206f744de0b97e3aa9ea753e37 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:47.035040 28393 raft_consensus.cc:2454] T 71fa62206f744de0b97e3aa9ea753e37 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:12:47.035039 31159 raft_consensus.cc:2454] T 71fa62206f744de0b97e3aa9ea753e37 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:12:47.035459 30697 leader_election.cc:302] T 71fa62206f744de0b97e3aa9ea753e37 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:47.035701 31688 raft_consensus.cc:2781] T 71fa62206f744de0b97e3aa9ea753e37 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:47.036485 31688 raft_consensus.cc:686] T 71fa62206f744de0b97e3aa9ea753e37 P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Becoming Leader. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Running, Role: LEADER
I20240503 08:12:47.036810 31688 consensus_queue.cc:237] T 71fa62206f744de0b97e3aa9ea753e37 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:47.039693 27859 catalog_manager.cc:5484] T 71fa62206f744de0b97e3aa9ea753e37 P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69). New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:12:47.095046 31601 raft_consensus.cc:482] T 8425cfe5c6134a60852e12f9af55227b P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:47.095247 31601 raft_consensus.cc:504] T 8425cfe5c6134a60852e12f9af55227b P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:47.096048 31601 leader_election.cc:288] T 8425cfe5c6134a60852e12f9af55227b P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:12:47.096422 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8425cfe5c6134a60852e12f9af55227b" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:12:47.096503 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8425cfe5c6134a60852e12f9af55227b" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:12:47.096697 31159 raft_consensus.cc:2454] T 8425cfe5c6134a60852e12f9af55227b P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:12:47.096748 28003 raft_consensus.cc:2454] T 8425cfe5c6134a60852e12f9af55227b P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:12:47.097071 28326 leader_election.cc:302] T 8425cfe5c6134a60852e12f9af55227b P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:47.097353 31601 raft_consensus.cc:2781] T 8425cfe5c6134a60852e12f9af55227b P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:47.097434 31601 raft_consensus.cc:482] T 8425cfe5c6134a60852e12f9af55227b P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:47.097537 31601 raft_consensus.cc:3037] T 8425cfe5c6134a60852e12f9af55227b P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:47.099658 31601 raft_consensus.cc:504] T 8425cfe5c6134a60852e12f9af55227b P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:47.100179 31601 leader_election.cc:288] T 8425cfe5c6134a60852e12f9af55227b P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Requested vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:12:47.100481 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8425cfe5c6134a60852e12f9af55227b" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:12:47.100615 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8425cfe5c6134a60852e12f9af55227b" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:12:47.100652 31159 raft_consensus.cc:3037] T 8425cfe5c6134a60852e12f9af55227b P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:47.100765 28003 raft_consensus.cc:3037] T 8425cfe5c6134a60852e12f9af55227b P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:47.102536 31159 raft_consensus.cc:2454] T 8425cfe5c6134a60852e12f9af55227b P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:12:47.102689 28003 raft_consensus.cc:2454] T 8425cfe5c6134a60852e12f9af55227b P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:12:47.102882 28326 leader_election.cc:302] T 8425cfe5c6134a60852e12f9af55227b P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:47.103137 31601 raft_consensus.cc:2781] T 8425cfe5c6134a60852e12f9af55227b P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:47.103276 31601 raft_consensus.cc:686] T 8425cfe5c6134a60852e12f9af55227b P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Becoming Leader. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Running, Role: LEADER
I20240503 08:12:47.103495 31601 consensus_queue.cc:237] T 8425cfe5c6134a60852e12f9af55227b P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:47.106292 27859 catalog_manager.cc:5484] T 8425cfe5c6134a60852e12f9af55227b P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68). New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:12:47.121297 27859 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:47.121503 27859 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:47.124162 27859 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet 4eeaef7b0e3e489ea92bdde6a2b24835
I20240503 08:12:47.124917 30741 tablet_service.cc:1514] Processing DeleteTablet for tablet 4eeaef7b0e3e489ea92bdde6a2b24835 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:47 UTC) from {username='slave'} at 127.0.0.1:46794
I20240503 08:12:47.125283 28372 tablet_service.cc:1514] Processing DeleteTablet for tablet 4eeaef7b0e3e489ea92bdde6a2b24835 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:47 UTC) from {username='slave'} at 127.0.0.1:34786
I20240503 08:12:47.125278 31007 tablet_service.cc:1514] Processing DeleteTablet for tablet 4eeaef7b0e3e489ea92bdde6a2b24835 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:47 UTC) from {username='slave'} at 127.0.0.1:47020
I20240503 08:12:47.125591 31774 tablet_replica.cc:330] T 4eeaef7b0e3e489ea92bdde6a2b24835 P 6d1d7de19459435fbde1999e3dfd226b: stopping tablet replica
I20240503 08:12:47.125793 31775 tablet_replica.cc:330] T 4eeaef7b0e3e489ea92bdde6a2b24835 P ced21ca983b749d0b4d4f6a0e2426bf8: stopping tablet replica
I20240503 08:12:47.125947 31774 raft_consensus.cc:2229] T 4eeaef7b0e3e489ea92bdde6a2b24835 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:47.126039 31775 raft_consensus.cc:2229] T 4eeaef7b0e3e489ea92bdde6a2b24835 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:47.126191 31776 tablet_replica.cc:330] T 4eeaef7b0e3e489ea92bdde6a2b24835 P d09a11a03989421b9f28019dc1c14cc2: stopping tablet replica
I20240503 08:12:47.126202 31774 raft_consensus.cc:2258] T 4eeaef7b0e3e489ea92bdde6a2b24835 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:47.126273 31775 raft_consensus.cc:2258] T 4eeaef7b0e3e489ea92bdde6a2b24835 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:47.126402 31776 raft_consensus.cc:2229] T 4eeaef7b0e3e489ea92bdde6a2b24835 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:47.126754 31776 raft_consensus.cc:2258] T 4eeaef7b0e3e489ea92bdde6a2b24835 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:47.127671 31776 ts_tablet_manager.cc:1875] T 4eeaef7b0e3e489ea92bdde6a2b24835 P d09a11a03989421b9f28019dc1c14cc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:47.127851 31774 ts_tablet_manager.cc:1875] T 4eeaef7b0e3e489ea92bdde6a2b24835 P 6d1d7de19459435fbde1999e3dfd226b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:47.127967 31775 ts_tablet_manager.cc:1875] T 4eeaef7b0e3e489ea92bdde6a2b24835 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:47.133968 31775 ts_tablet_manager.cc:1888] T 4eeaef7b0e3e489ea92bdde6a2b24835 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:47.133961 31776 ts_tablet_manager.cc:1888] T 4eeaef7b0e3e489ea92bdde6a2b24835 P d09a11a03989421b9f28019dc1c14cc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:47.133970 31774 ts_tablet_manager.cc:1888] T 4eeaef7b0e3e489ea92bdde6a2b24835 P 6d1d7de19459435fbde1999e3dfd226b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:47.134086 31776 log.cc:1196] T 4eeaef7b0e3e489ea92bdde6a2b24835 P d09a11a03989421b9f28019dc1c14cc2: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/wals/4eeaef7b0e3e489ea92bdde6a2b24835
I20240503 08:12:47.134084 31774 log.cc:1196] T 4eeaef7b0e3e489ea92bdde6a2b24835 P 6d1d7de19459435fbde1999e3dfd226b: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal/wals/4eeaef7b0e3e489ea92bdde6a2b24835
I20240503 08:12:47.134231 31775 log.cc:1196] T 4eeaef7b0e3e489ea92bdde6a2b24835 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal/wals/4eeaef7b0e3e489ea92bdde6a2b24835
I20240503 08:12:47.134552 31774 ts_tablet_manager.cc:1909] T 4eeaef7b0e3e489ea92bdde6a2b24835 P 6d1d7de19459435fbde1999e3dfd226b: Deleting consensus metadata
I20240503 08:12:47.134588 31776 ts_tablet_manager.cc:1909] T 4eeaef7b0e3e489ea92bdde6a2b24835 P d09a11a03989421b9f28019dc1c14cc2: Deleting consensus metadata
I20240503 08:12:47.135378 31775 ts_tablet_manager.cc:1909] T 4eeaef7b0e3e489ea92bdde6a2b24835 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting consensus metadata
I20240503 08:12:47.135476 27848 catalog_manager.cc:4830] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): tablet 4eeaef7b0e3e489ea92bdde6a2b24835 (table rebalancer_dynamic_table_new_name [id=a860bd4dbc4546d683170d4cb58947a8]) successfully deleted
I20240503 08:12:47.135684 27849 catalog_manager.cc:4830] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): tablet 4eeaef7b0e3e489ea92bdde6a2b24835 (table rebalancer_dynamic_table_new_name [id=a860bd4dbc4546d683170d4cb58947a8]) successfully deleted
I20240503 08:12:47.136336 27847 catalog_manager.cc:4830] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): tablet 4eeaef7b0e3e489ea92bdde6a2b24835 (table rebalancer_dynamic_table_new_name [id=a860bd4dbc4546d683170d4cb58947a8]) successfully deleted
I20240503 08:12:47.225914 31601 raft_consensus.cc:482] T 720f36100d794158be13ff02595e6ddf P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:47.226104 31601 raft_consensus.cc:504] T 720f36100d794158be13ff02595e6ddf P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:47.226761 31601 leader_election.cc:288] T 720f36100d794158be13ff02595e6ddf P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:12:47.226950 27859 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:47.227216 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "720f36100d794158be13ff02595e6ddf" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:12:47.227245 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "720f36100d794158be13ff02595e6ddf" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:12:47.227435 31159 raft_consensus.cc:2454] T 720f36100d794158be13ff02595e6ddf P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:12:47.227480 30761 raft_consensus.cc:2454] T 720f36100d794158be13ff02595e6ddf P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:12:47.227896 28328 leader_election.cc:302] T 720f36100d794158be13ff02595e6ddf P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:47.228161 31601 raft_consensus.cc:2781] T 720f36100d794158be13ff02595e6ddf P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:47.228257 31601 raft_consensus.cc:482] T 720f36100d794158be13ff02595e6ddf P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:47.228351 31601 raft_consensus.cc:3037] T 720f36100d794158be13ff02595e6ddf P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:47.230525 31601 raft_consensus.cc:504] T 720f36100d794158be13ff02595e6ddf P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:47.231184 31601 leader_election.cc:288] T 720f36100d794158be13ff02595e6ddf P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Requested vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:12:47.231601 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "720f36100d794158be13ff02595e6ddf" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:12:47.231815 31159 raft_consensus.cc:3037] T 720f36100d794158be13ff02595e6ddf P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:47.232074 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "720f36100d794158be13ff02595e6ddf" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:12:47.232261 30761 raft_consensus.cc:3037] T 720f36100d794158be13ff02595e6ddf P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:47.234596 30874 tablet_service.cc:1467] Processing CreateTablet for tablet 4b1e7176e2674b738c355a8d22fe33bd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=44aa40e4e68b4ea7bed7151e0ae50a58]), partition=
I20240503 08:12:47.234959 30874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b1e7176e2674b738c355a8d22fe33bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:47.235018 28111 tablet_service.cc:1467] Processing CreateTablet for tablet 4b1e7176e2674b738c355a8d22fe33bd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=44aa40e4e68b4ea7bed7151e0ae50a58]), partition=
I20240503 08:12:47.235394 28111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b1e7176e2674b738c355a8d22fe33bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:47.235757 31140 tablet_service.cc:1467] Processing CreateTablet for tablet 4b1e7176e2674b738c355a8d22fe33bd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=44aa40e4e68b4ea7bed7151e0ae50a58]), partition=
I20240503 08:12:47.236069 31140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b1e7176e2674b738c355a8d22fe33bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:47.236042 30761 raft_consensus.cc:2454] T 720f36100d794158be13ff02595e6ddf P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:12:47.236380 31159 raft_consensus.cc:2454] T 720f36100d794158be13ff02595e6ddf P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:12:47.237913 28326 leader_election.cc:302] T 720f36100d794158be13ff02595e6ddf P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:47.238294 31601 raft_consensus.cc:2781] T 720f36100d794158be13ff02595e6ddf P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:47.238468 31601 raft_consensus.cc:686] T 720f36100d794158be13ff02595e6ddf P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Becoming Leader. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Running, Role: LEADER
I20240503 08:12:47.238799 31601 consensus_queue.cc:237] T 720f36100d794158be13ff02595e6ddf P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:47.242370 31764 tablet_bootstrap.cc:492] T 4b1e7176e2674b738c355a8d22fe33bd P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:47.243952 31764 tablet_bootstrap.cc:654] T 4b1e7176e2674b738c355a8d22fe33bd P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:47.245337 31779 tablet_bootstrap.cc:492] T 4b1e7176e2674b738c355a8d22fe33bd P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:47.248627 31779 tablet_bootstrap.cc:654] T 4b1e7176e2674b738c355a8d22fe33bd P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:47.254050 31764 tablet_bootstrap.cc:492] T 4b1e7176e2674b738c355a8d22fe33bd P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:47.254267 31764 ts_tablet_manager.cc:1367] T 4b1e7176e2674b738c355a8d22fe33bd P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20240503 08:12:47.255074 31764 raft_consensus.cc:348] T 4b1e7176e2674b738c355a8d22fe33bd P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:47.255429 31764 raft_consensus.cc:374] T 4b1e7176e2674b738c355a8d22fe33bd P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:47.255546 31764 raft_consensus.cc:729] T 4b1e7176e2674b738c355a8d22fe33bd P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:47.255892 31764 consensus_queue.cc:260] T 4b1e7176e2674b738c355a8d22fe33bd P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:47.255954 27859 catalog_manager.cc:5484] T 720f36100d794158be13ff02595e6ddf P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68). New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:12:47.256601 31764 ts_tablet_manager.cc:1398] T 4b1e7176e2674b738c355a8d22fe33bd P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:47.257232 31778 tablet_bootstrap.cc:492] T 4b1e7176e2674b738c355a8d22fe33bd P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:47.258924 31778 tablet_bootstrap.cc:654] T 4b1e7176e2674b738c355a8d22fe33bd P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:47.261147 31779 tablet_bootstrap.cc:492] T 4b1e7176e2674b738c355a8d22fe33bd P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:47.261297 31779 ts_tablet_manager.cc:1367] T 4b1e7176e2674b738c355a8d22fe33bd P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.000s
I20240503 08:12:47.262128 31779 raft_consensus.cc:348] T 4b1e7176e2674b738c355a8d22fe33bd P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:47.262351 31779 raft_consensus.cc:374] T 4b1e7176e2674b738c355a8d22fe33bd P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:47.262445 31779 raft_consensus.cc:729] T 4b1e7176e2674b738c355a8d22fe33bd P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:47.262509 31778 tablet_bootstrap.cc:492] T 4b1e7176e2674b738c355a8d22fe33bd P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:47.262639 31778 ts_tablet_manager.cc:1367] T 4b1e7176e2674b738c355a8d22fe33bd P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 08:12:47.262730 31779 consensus_queue.cc:260] T 4b1e7176e2674b738c355a8d22fe33bd P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:47.263260 31779 ts_tablet_manager.cc:1398] T 4b1e7176e2674b738c355a8d22fe33bd P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:47.263335 31778 raft_consensus.cc:348] T 4b1e7176e2674b738c355a8d22fe33bd P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:47.263530 31778 raft_consensus.cc:374] T 4b1e7176e2674b738c355a8d22fe33bd P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:47.263643 31778 raft_consensus.cc:729] T 4b1e7176e2674b738c355a8d22fe33bd P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:47.263862 31778 consensus_queue.cc:260] T 4b1e7176e2674b738c355a8d22fe33bd P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:47.264360 31778 ts_tablet_manager.cc:1398] T 4b1e7176e2674b738c355a8d22fe33bd P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:47.359959 31731 raft_consensus.cc:482] T 4b1e7176e2674b738c355a8d22fe33bd P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:47.360127 31731 raft_consensus.cc:504] T 4b1e7176e2674b738c355a8d22fe33bd P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:47.360821 31731 leader_election.cc:288] T 4b1e7176e2674b738c355a8d22fe33bd P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:47.361224 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b1e7176e2674b738c355a8d22fe33bd" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:12:47.361496 30895 raft_consensus.cc:2454] T 4b1e7176e2674b738c355a8d22fe33bd P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:47.362138 28066 leader_election.cc:302] T 4b1e7176e2674b738c355a8d22fe33bd P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:47.362418 31731 raft_consensus.cc:2781] T 4b1e7176e2674b738c355a8d22fe33bd P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:47.362499 31731 raft_consensus.cc:482] T 4b1e7176e2674b738c355a8d22fe33bd P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:47.362597 31731 raft_consensus.cc:3037] T 4b1e7176e2674b738c355a8d22fe33bd P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:47.365582 31731 raft_consensus.cc:504] T 4b1e7176e2674b738c355a8d22fe33bd P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:47.366498 31731 leader_election.cc:288] T 4b1e7176e2674b738c355a8d22fe33bd P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Requested vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:47.366694 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b1e7176e2674b738c355a8d22fe33bd" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:12:47.366921 30895 raft_consensus.cc:3037] T 4b1e7176e2674b738c355a8d22fe33bd P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:47.368533 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b1e7176e2674b738c355a8d22fe33bd" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:12:47.368748 31159 raft_consensus.cc:2454] T 4b1e7176e2674b738c355a8d22fe33bd P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:47.368686 31161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b1e7176e2674b738c355a8d22fe33bd" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:12:47.368903 31161 raft_consensus.cc:3037] T 4b1e7176e2674b738c355a8d22fe33bd P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:47.369603 30895 raft_consensus.cc:2454] T 4b1e7176e2674b738c355a8d22fe33bd P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:12:47.370002 28066 leader_election.cc:302] T 4b1e7176e2674b738c355a8d22fe33bd P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:47.370218 31731 raft_consensus.cc:2781] T 4b1e7176e2674b738c355a8d22fe33bd P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:47.370577 31731 raft_consensus.cc:686] T 4b1e7176e2674b738c355a8d22fe33bd P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Becoming Leader. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Running, Role: LEADER
I20240503 08:12:47.370858 31731 consensus_queue.cc:237] T 4b1e7176e2674b738c355a8d22fe33bd P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:47.371266 31161 raft_consensus.cc:2454] T 4b1e7176e2674b738c355a8d22fe33bd P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:12:47.373643 27856 catalog_manager.cc:5484] T 4b1e7176e2674b738c355a8d22fe33bd P 1b411c6bb25549368041773d7b710d48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b411c6bb25549368041773d7b710d48 (127.20.230.66). New cstate: current_term: 1 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:12:47.431146 31688 consensus_queue.cc:1035] T 71fa62206f744de0b97e3aa9ea753e37 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:47.435637 31688 consensus_queue.cc:1035] T 71fa62206f744de0b97e3aa9ea753e37 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:47.495437 27856 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:47.501367 31161 raft_consensus.cc:1261] T 4b1e7176e2674b738c355a8d22fe33bd P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:47.501626 30894 raft_consensus.cc:1261] T 4b1e7176e2674b738c355a8d22fe33bd P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:47.502147 31786 consensus_queue.cc:1035] T 4b1e7176e2674b738c355a8d22fe33bd P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:47.502400 31731 consensus_queue.cc:1035] T 4b1e7176e2674b738c355a8d22fe33bd P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:47.513727 27856 catalog_manager.cc:5576] TS b00f244c69df401997ea99d492218a98 (127.20.230.72:40893) does not have the latest schema for tablet 4b1e7176e2674b738c355a8d22fe33bd (table rebalancer_dynamic_table [id=44aa40e4e68b4ea7bed7151e0ae50a58]). Expected version 1 got 0
I20240503 08:12:47.513809 31795 tablet.cc:1722] T 4b1e7176e2674b738c355a8d22fe33bd P 1b411c6bb25549368041773d7b710d48: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:47.514385 31795 tablet.cc:1620] T 4b1e7176e2674b738c355a8d22fe33bd P 1b411c6bb25549368041773d7b710d48: MemRowSet was empty: no flush needed.
I20240503 08:12:47.523316 27855 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=44aa40e4e68b4ea7bed7151e0ae50a58] alter complete (version 1)
I20240503 08:12:47.527128 27856 catalog_manager.cc:5576] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747) does not have the latest schema for tablet 4b1e7176e2674b738c355a8d22fe33bd (table rebalancer_dynamic_table [id=44aa40e4e68b4ea7bed7151e0ae50a58]). Expected version 1 got 0
I20240503 08:12:47.530227 31796 tablet.cc:1722] T 4b1e7176e2674b738c355a8d22fe33bd P b00f244c69df401997ea99d492218a98: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:47.530748 31796 tablet.cc:1620] T 4b1e7176e2674b738c355a8d22fe33bd P b00f244c69df401997ea99d492218a98: MemRowSet was empty: no flush needed.
I20240503 08:12:47.532829 31798 tablet.cc:1722] T 4b1e7176e2674b738c355a8d22fe33bd P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:47.533321 31798 tablet.cc:1620] T 4b1e7176e2674b738c355a8d22fe33bd P ce1496a47d084365ad786cd09e472a2c: MemRowSet was empty: no flush needed.
I20240503 08:12:47.534471 27856 catalog_manager.cc:5576] TS b00f244c69df401997ea99d492218a98 (127.20.230.72:40893) does not have the latest schema for tablet 4b1e7176e2674b738c355a8d22fe33bd (table rebalancer_dynamic_table [id=44aa40e4e68b4ea7bed7151e0ae50a58]). Expected version 1 got 0
I20240503 08:12:47.534976 27855 catalog_manager.cc:5576] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747) does not have the latest schema for tablet 4b1e7176e2674b738c355a8d22fe33bd (table rebalancer_dynamic_table [id=44aa40e4e68b4ea7bed7151e0ae50a58]). Expected version 1 got 0
I20240503 08:12:47.606874 31759 consensus_queue.cc:1035] T 8425cfe5c6134a60852e12f9af55227b P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:47.611470 31800 consensus_queue.cc:1035] T 8425cfe5c6134a60852e12f9af55227b P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:47.625433 27855 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:47.634606 31795 tablet.cc:1722] T 4b1e7176e2674b738c355a8d22fe33bd P 1b411c6bb25549368041773d7b710d48: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:47.635108 31795 tablet.cc:1620] T 4b1e7176e2674b738c355a8d22fe33bd P 1b411c6bb25549368041773d7b710d48: MemRowSet was empty: no flush needed.
I20240503 08:12:47.635902 31798 tablet.cc:1722] T 4b1e7176e2674b738c355a8d22fe33bd P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:47.636186 31796 tablet.cc:1722] T 4b1e7176e2674b738c355a8d22fe33bd P b00f244c69df401997ea99d492218a98: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:47.636370 31798 tablet.cc:1620] T 4b1e7176e2674b738c355a8d22fe33bd P ce1496a47d084365ad786cd09e472a2c: MemRowSet was empty: no flush needed.
I20240503 08:12:47.636580 31796 tablet.cc:1620] T 4b1e7176e2674b738c355a8d22fe33bd P b00f244c69df401997ea99d492218a98: MemRowSet was empty: no flush needed.
I20240503 08:12:47.644845 27855 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=44aa40e4e68b4ea7bed7151e0ae50a58] alter complete (version 2)
I20240503 08:12:47.688503 31601 consensus_queue.cc:1035] T 720f36100d794158be13ff02595e6ddf P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:47.693113 31800 consensus_queue.cc:1035] T 720f36100d794158be13ff02595e6ddf P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:47.747095 27859 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:47.754055 31795 tablet.cc:1722] T 4b1e7176e2674b738c355a8d22fe33bd P 1b411c6bb25549368041773d7b710d48: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:47.755211 31796 tablet.cc:1722] T 4b1e7176e2674b738c355a8d22fe33bd P b00f244c69df401997ea99d492218a98: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:47.758493 31798 tablet.cc:1722] T 4b1e7176e2674b738c355a8d22fe33bd P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:47.763119 27859 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table_new_name [id=44aa40e4e68b4ea7bed7151e0ae50a58] alter complete (version 3)
I20240503 08:12:47.831154 27862 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_extra_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:47.838539 28243 tablet_service.cc:1467] Processing CreateTablet for tablet b12ceaf7f8d04fad8b07b36459da7116 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=c740702474bd4c328043a94e39c2db1b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:47.838699 27981 tablet_service.cc:1467] Processing CreateTablet for tablet b12ceaf7f8d04fad8b07b36459da7116 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=c740702474bd4c328043a94e39c2db1b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:47.838985 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b12ceaf7f8d04fad8b07b36459da7116. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:47.839040 27981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b12ceaf7f8d04fad8b07b36459da7116. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:47.839257 28111 tablet_service.cc:1467] Processing CreateTablet for tablet 45f47478f49149b895060791655503ae (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=c740702474bd4c328043a94e39c2db1b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:47.839665 28111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45f47478f49149b895060791655503ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:47.843961 30874 tablet_service.cc:1467] Processing CreateTablet for tablet 549f64ad9ef64a3da619c5df5a411e1c (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=c740702474bd4c328043a94e39c2db1b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:47.844178 27983 tablet_service.cc:1467] Processing CreateTablet for tablet 45f47478f49149b895060791655503ae (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=c740702474bd4c328043a94e39c2db1b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:47.844348 30874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 549f64ad9ef64a3da619c5df5a411e1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:47.844519 27983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45f47478f49149b895060791655503ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:47.844503 30741 tablet_service.cc:1467] Processing CreateTablet for tablet 549f64ad9ef64a3da619c5df5a411e1c (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=c740702474bd4c328043a94e39c2db1b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:47.844897 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 549f64ad9ef64a3da619c5df5a411e1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:47.850864 31140 tablet_service.cc:1467] Processing CreateTablet for tablet 549f64ad9ef64a3da619c5df5a411e1c (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=c740702474bd4c328043a94e39c2db1b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:47.851307 31140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 549f64ad9ef64a3da619c5df5a411e1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:47.856016 31807 tablet_bootstrap.cc:492] T 45f47478f49149b895060791655503ae P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:47.858651 31807 tablet_bootstrap.cc:654] T 45f47478f49149b895060791655503ae P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:47.861068 31007 tablet_service.cc:1467] Processing CreateTablet for tablet b12ceaf7f8d04fad8b07b36459da7116 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=c740702474bd4c328043a94e39c2db1b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:47.861474 31007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b12ceaf7f8d04fad8b07b36459da7116. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:47.861667 31008 tablet_service.cc:1467] Processing CreateTablet for tablet 45f47478f49149b895060791655503ae (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=c740702474bd4c328043a94e39c2db1b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:47.862003 31008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45f47478f49149b895060791655503ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:47.864840 31807 tablet_bootstrap.cc:492] T 45f47478f49149b895060791655503ae P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:47.864980 31807 ts_tablet_manager.cc:1367] T 45f47478f49149b895060791655503ae P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20240503 08:12:47.865747 31807 raft_consensus.cc:348] T 45f47478f49149b895060791655503ae P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:47.867553 31807 raft_consensus.cc:374] T 45f47478f49149b895060791655503ae P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:47.867806 31807 raft_consensus.cc:729] T 45f47478f49149b895060791655503ae P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:47.868217 31807 consensus_queue.cc:260] T 45f47478f49149b895060791655503ae P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:47.869537 31807 ts_tablet_manager.cc:1398] T 45f47478f49149b895060791655503ae P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 08:12:47.878966 27859 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:47.879154 31814 tablet_bootstrap.cc:492] T 549f64ad9ef64a3da619c5df5a411e1c P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:47.879168 27859 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:47.882275 31814 tablet_bootstrap.cc:654] T 549f64ad9ef64a3da619c5df5a411e1c P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:47.883528 27859 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet 4b1e7176e2674b738c355a8d22fe33bd
I20240503 08:12:47.884218 30874 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b1e7176e2674b738c355a8d22fe33bd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:47 UTC) from {username='slave'} at 127.0.0.1:34426
I20240503 08:12:47.885433 31139 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b1e7176e2674b738c355a8d22fe33bd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:47 UTC) from {username='slave'} at 127.0.0.1:45814
I20240503 08:12:47.885994 31817 tablet_replica.cc:330] T 4b1e7176e2674b738c355a8d22fe33bd P b00f244c69df401997ea99d492218a98: stopping tablet replica
I20240503 08:12:47.886271 31817 raft_consensus.cc:2229] T 4b1e7176e2674b738c355a8d22fe33bd P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:47.886554 31817 raft_consensus.cc:2258] T 4b1e7176e2674b738c355a8d22fe33bd P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:47.886862 28111 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b1e7176e2674b738c355a8d22fe33bd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:47 UTC) from {username='slave'} at 127.0.0.1:57912
I20240503 08:12:47.887495 31818 tablet_replica.cc:330] T 4b1e7176e2674b738c355a8d22fe33bd P 1b411c6bb25549368041773d7b710d48: stopping tablet replica
I20240503 08:12:47.887794 31818 raft_consensus.cc:2229] T 4b1e7176e2674b738c355a8d22fe33bd P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:47.888466 31817 ts_tablet_manager.cc:1875] T 4b1e7176e2674b738c355a8d22fe33bd P b00f244c69df401997ea99d492218a98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:47.889341 31818 raft_consensus.cc:2258] T 4b1e7176e2674b738c355a8d22fe33bd P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:47.889730 31813 tablet_bootstrap.cc:492] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:47.891498 31818 ts_tablet_manager.cc:1875] T 4b1e7176e2674b738c355a8d22fe33bd P 1b411c6bb25549368041773d7b710d48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:47.891805 31815 tablet_replica.cc:330] T 4b1e7176e2674b738c355a8d22fe33bd P ce1496a47d084365ad786cd09e472a2c: stopping tablet replica
I20240503 08:12:47.891855 31811 tablet_bootstrap.cc:492] T 549f64ad9ef64a3da619c5df5a411e1c P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:47.892031 31815 raft_consensus.cc:2229] T 4b1e7176e2674b738c355a8d22fe33bd P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:47.892266 31815 raft_consensus.cc:2258] T 4b1e7176e2674b738c355a8d22fe33bd P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:47.893877 31815 ts_tablet_manager.cc:1875] T 4b1e7176e2674b738c355a8d22fe33bd P ce1496a47d084365ad786cd09e472a2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:47.894399 31810 tablet_bootstrap.cc:492] T 549f64ad9ef64a3da619c5df5a411e1c P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:47.894734 31808 tablet_bootstrap.cc:492] T b12ceaf7f8d04fad8b07b36459da7116 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:47.895011 31813 tablet_bootstrap.cc:654] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:47.894579 31814 tablet_bootstrap.cc:492] T 549f64ad9ef64a3da619c5df5a411e1c P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:47.895371 31814 ts_tablet_manager.cc:1367] T 549f64ad9ef64a3da619c5df5a411e1c P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.005s
I20240503 08:12:47.896248 31814 raft_consensus.cc:348] T 549f64ad9ef64a3da619c5df5a411e1c P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:47.896518 31814 raft_consensus.cc:374] T 549f64ad9ef64a3da619c5df5a411e1c P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:47.896653 31814 raft_consensus.cc:729] T 549f64ad9ef64a3da619c5df5a411e1c P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:47.896797 31811 tablet_bootstrap.cc:654] T 549f64ad9ef64a3da619c5df5a411e1c P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:47.897033 31814 consensus_queue.cc:260] T 549f64ad9ef64a3da619c5df5a411e1c P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:47.897740 31814 ts_tablet_manager.cc:1398] T 549f64ad9ef64a3da619c5df5a411e1c P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:12:47.898293 31808 tablet_bootstrap.cc:654] T b12ceaf7f8d04fad8b07b36459da7116 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:47.900640 31809 tablet_bootstrap.cc:492] T 45f47478f49149b895060791655503ae P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:47.902958 31810 tablet_bootstrap.cc:654] T 549f64ad9ef64a3da619c5df5a411e1c P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:47.903955 31818 ts_tablet_manager.cc:1888] T 4b1e7176e2674b738c355a8d22fe33bd P 1b411c6bb25549368041773d7b710d48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:47.904465 31817 ts_tablet_manager.cc:1888] T 4b1e7176e2674b738c355a8d22fe33bd P b00f244c69df401997ea99d492218a98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:47.904574 31817 log.cc:1196] T 4b1e7176e2674b738c355a8d22fe33bd P b00f244c69df401997ea99d492218a98: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal/wals/4b1e7176e2674b738c355a8d22fe33bd
I20240503 08:12:47.904664 31813 tablet_bootstrap.cc:492] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:47.904891 31813 ts_tablet_manager.cc:1367] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.003s
I20240503 08:12:47.905025 31817 ts_tablet_manager.cc:1909] T 4b1e7176e2674b738c355a8d22fe33bd P b00f244c69df401997ea99d492218a98: Deleting consensus metadata
I20240503 08:12:47.905194 31818 log.cc:1196] T 4b1e7176e2674b738c355a8d22fe33bd P 1b411c6bb25549368041773d7b710d48: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/4b1e7176e2674b738c355a8d22fe33bd
I20240503 08:12:47.905534 31818 ts_tablet_manager.cc:1909] T 4b1e7176e2674b738c355a8d22fe33bd P 1b411c6bb25549368041773d7b710d48: Deleting consensus metadata
I20240503 08:12:47.906405 27848 catalog_manager.cc:4830] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): tablet 4b1e7176e2674b738c355a8d22fe33bd (table rebalancer_dynamic_table_new_name [id=44aa40e4e68b4ea7bed7151e0ae50a58]) successfully deleted
I20240503 08:12:47.906931 31809 tablet_bootstrap.cc:654] T 45f47478f49149b895060791655503ae P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:47.905668 31813 raft_consensus.cc:348] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:47.907148 31813 raft_consensus.cc:374] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:47.907188 27846 catalog_manager.cc:4830] TS b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): tablet 4b1e7176e2674b738c355a8d22fe33bd (table rebalancer_dynamic_table_new_name [id=44aa40e4e68b4ea7bed7151e0ae50a58]) successfully deleted
I20240503 08:12:47.907248 31813 raft_consensus.cc:729] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:47.906925 31810 tablet_bootstrap.cc:492] T 549f64ad9ef64a3da619c5df5a411e1c P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:47.907431 31810 ts_tablet_manager.cc:1367] T 549f64ad9ef64a3da619c5df5a411e1c P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240503 08:12:47.907513 31813 consensus_queue.cc:260] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:47.909241 31813 ts_tablet_manager.cc:1398] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 08:12:47.909135 31810 raft_consensus.cc:348] T 549f64ad9ef64a3da619c5df5a411e1c P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:47.909338 31810 raft_consensus.cc:374] T 549f64ad9ef64a3da619c5df5a411e1c P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:47.909426 31810 raft_consensus.cc:729] T 549f64ad9ef64a3da619c5df5a411e1c P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:47.909592 31813 tablet_bootstrap.cc:492] T 45f47478f49149b895060791655503ae P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:47.909655 31810 consensus_queue.cc:260] T 549f64ad9ef64a3da619c5df5a411e1c P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:47.910423 31810 ts_tablet_manager.cc:1398] T 549f64ad9ef64a3da619c5df5a411e1c P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:47.910993 31815 ts_tablet_manager.cc:1888] T 4b1e7176e2674b738c355a8d22fe33bd P ce1496a47d084365ad786cd09e472a2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:47.911432 31815 log.cc:1196] T 4b1e7176e2674b738c355a8d22fe33bd P ce1496a47d084365ad786cd09e472a2c: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal/wals/4b1e7176e2674b738c355a8d22fe33bd
I20240503 08:12:47.912200 31815 ts_tablet_manager.cc:1909] T 4b1e7176e2674b738c355a8d22fe33bd P ce1496a47d084365ad786cd09e472a2c: Deleting consensus metadata
I20240503 08:12:47.912576 31811 tablet_bootstrap.cc:492] T 549f64ad9ef64a3da619c5df5a411e1c P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:47.912753 31811 ts_tablet_manager.cc:1367] T 549f64ad9ef64a3da619c5df5a411e1c P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20240503 08:12:47.913209 27846 catalog_manager.cc:4830] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): tablet 4b1e7176e2674b738c355a8d22fe33bd (table rebalancer_dynamic_table_new_name [id=44aa40e4e68b4ea7bed7151e0ae50a58]) successfully deleted
I20240503 08:12:47.913753 31811 raft_consensus.cc:348] T 549f64ad9ef64a3da619c5df5a411e1c P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:47.914031 31811 raft_consensus.cc:374] T 549f64ad9ef64a3da619c5df5a411e1c P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:47.914160 31811 raft_consensus.cc:729] T 549f64ad9ef64a3da619c5df5a411e1c P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:47.914547 31809 tablet_bootstrap.cc:492] T 45f47478f49149b895060791655503ae P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:47.914456 31811 consensus_queue.cc:260] T 549f64ad9ef64a3da619c5df5a411e1c P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:47.914697 31809 ts_tablet_manager.cc:1367] T 45f47478f49149b895060791655503ae P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20240503 08:12:47.914988 31808 tablet_bootstrap.cc:492] T b12ceaf7f8d04fad8b07b36459da7116 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:47.915273 31808 ts_tablet_manager.cc:1367] T b12ceaf7f8d04fad8b07b36459da7116 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20240503 08:12:47.915367 31809 raft_consensus.cc:348] T 45f47478f49149b895060791655503ae P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:47.915598 31809 raft_consensus.cc:374] T 45f47478f49149b895060791655503ae P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:47.915697 31809 raft_consensus.cc:729] T 45f47478f49149b895060791655503ae P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:47.915951 31811 ts_tablet_manager.cc:1398] T 549f64ad9ef64a3da619c5df5a411e1c P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:47.916072 31808 raft_consensus.cc:348] T b12ceaf7f8d04fad8b07b36459da7116 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:47.916273 31808 raft_consensus.cc:374] T b12ceaf7f8d04fad8b07b36459da7116 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:47.916378 31808 raft_consensus.cc:729] T b12ceaf7f8d04fad8b07b36459da7116 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:47.916610 31813 tablet_bootstrap.cc:654] T 45f47478f49149b895060791655503ae P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:47.916638 31808 consensus_queue.cc:260] T b12ceaf7f8d04fad8b07b36459da7116 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:47.917198 31808 ts_tablet_manager.cc:1398] T b12ceaf7f8d04fad8b07b36459da7116 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:47.917493 31809 consensus_queue.cc:260] T 45f47478f49149b895060791655503ae P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:47.918066 31809 ts_tablet_manager.cc:1398] T 45f47478f49149b895060791655503ae P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:12:47.918331 31809 tablet_bootstrap.cc:492] T b12ceaf7f8d04fad8b07b36459da7116 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:47.919823 31809 tablet_bootstrap.cc:654] T b12ceaf7f8d04fad8b07b36459da7116 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:47.921662 31813 tablet_bootstrap.cc:492] T 45f47478f49149b895060791655503ae P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:47.921903 31813 ts_tablet_manager.cc:1367] T 45f47478f49149b895060791655503ae P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240503 08:12:47.922633 31813 raft_consensus.cc:348] T 45f47478f49149b895060791655503ae P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:47.922798 31813 raft_consensus.cc:374] T 45f47478f49149b895060791655503ae P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:47.922995 31813 raft_consensus.cc:729] T 45f47478f49149b895060791655503ae P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:47.923216 31813 consensus_queue.cc:260] T 45f47478f49149b895060791655503ae P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:47.923812 31813 ts_tablet_manager.cc:1398] T 45f47478f49149b895060791655503ae P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:47.925066 31809 tablet_bootstrap.cc:492] T b12ceaf7f8d04fad8b07b36459da7116 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:47.925264 31809 ts_tablet_manager.cc:1367] T b12ceaf7f8d04fad8b07b36459da7116 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20240503 08:12:47.925940 31809 raft_consensus.cc:348] T b12ceaf7f8d04fad8b07b36459da7116 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:47.926119 31809 raft_consensus.cc:374] T b12ceaf7f8d04fad8b07b36459da7116 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:47.926194 31809 raft_consensus.cc:729] T b12ceaf7f8d04fad8b07b36459da7116 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:47.926416 31809 consensus_queue.cc:260] T b12ceaf7f8d04fad8b07b36459da7116 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:47.926930 31809 ts_tablet_manager.cc:1398] T b12ceaf7f8d04fad8b07b36459da7116 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:47.987850 27859 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:47.993229 31745 raft_consensus.cc:482] T 45f47478f49149b895060791655503ae P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:47.993420 31745 raft_consensus.cc:504] T 45f47478f49149b895060791655503ae P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:47.994169 31745 leader_election.cc:288] T 45f47478f49149b895060791655503ae P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:47.994555 28131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45f47478f49149b895060791655503ae" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:12:47.994800 28131 raft_consensus.cc:2454] T 45f47478f49149b895060791655503ae P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:47.995376 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45f47478f49149b895060791655503ae" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:47.995599 31027 raft_consensus.cc:2454] T 45f47478f49149b895060791655503ae P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:47.995833 30741 tablet_service.cc:1467] Processing CreateTablet for tablet dbd0cada7fe64da588e0cb5b4f9668c4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0f892d4db3044188bdb00b55c3b81389]), partition=
I20240503 08:12:47.996207 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbd0cada7fe64da588e0cb5b4f9668c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:47.996237 27937 leader_election.cc:302] T 45f47478f49149b895060791655503ae P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:47.996973 31745 raft_consensus.cc:2781] T 45f47478f49149b895060791655503ae P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:47.997058 31745 raft_consensus.cc:482] T 45f47478f49149b895060791655503ae P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:47.997149 31745 raft_consensus.cc:3037] T 45f47478f49149b895060791655503ae P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:47.997213 30874 tablet_service.cc:1467] Processing CreateTablet for tablet dbd0cada7fe64da588e0cb5b4f9668c4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0f892d4db3044188bdb00b55c3b81389]), partition=
I20240503 08:12:47.997503 30874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbd0cada7fe64da588e0cb5b4f9668c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:47.999720 31745 raft_consensus.cc:504] T 45f47478f49149b895060791655503ae P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:48.001235 31745 leader_election.cc:288] T 45f47478f49149b895060791655503ae P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Requested vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:48.002244 28131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45f47478f49149b895060791655503ae" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:12:48.002249 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45f47478f49149b895060791655503ae" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:12:48.002522 28131 raft_consensus.cc:3037] T 45f47478f49149b895060791655503ae P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:48.002534 31027 raft_consensus.cc:3037] T 45f47478f49149b895060791655503ae P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:48.002722 31007 tablet_service.cc:1467] Processing CreateTablet for tablet dbd0cada7fe64da588e0cb5b4f9668c4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0f892d4db3044188bdb00b55c3b81389]), partition=
I20240503 08:12:48.003149 31007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbd0cada7fe64da588e0cb5b4f9668c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:48.005721 31027 raft_consensus.cc:2454] T 45f47478f49149b895060791655503ae P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:48.006644 27937 leader_election.cc:302] T 45f47478f49149b895060791655503ae P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:48.006767 31810 tablet_bootstrap.cc:492] T dbd0cada7fe64da588e0cb5b4f9668c4 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:48.007095 31745 raft_consensus.cc:2781] T 45f47478f49149b895060791655503ae P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:48.007302 31745 raft_consensus.cc:686] T 45f47478f49149b895060791655503ae P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Becoming Leader. State: Replica: 833582ff68c94269be364ebd187a75db, State: Running, Role: LEADER
I20240503 08:12:48.005715 28131 raft_consensus.cc:2454] T 45f47478f49149b895060791655503ae P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:48.008299 31810 tablet_bootstrap.cc:654] T dbd0cada7fe64da588e0cb5b4f9668c4 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:48.008904 31745 consensus_queue.cc:237] T 45f47478f49149b895060791655503ae P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:48.010488 31814 tablet_bootstrap.cc:492] T dbd0cada7fe64da588e0cb5b4f9668c4 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:48.011873 31813 tablet_bootstrap.cc:492] T dbd0cada7fe64da588e0cb5b4f9668c4 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:48.011987 31814 tablet_bootstrap.cc:654] T dbd0cada7fe64da588e0cb5b4f9668c4 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:48.012082 31810 tablet_bootstrap.cc:492] T dbd0cada7fe64da588e0cb5b4f9668c4 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:48.012229 31810 ts_tablet_manager.cc:1367] T dbd0cada7fe64da588e0cb5b4f9668c4 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 08:12:48.013883 31813 tablet_bootstrap.cc:654] T dbd0cada7fe64da588e0cb5b4f9668c4 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:48.013926 31810 raft_consensus.cc:348] T dbd0cada7fe64da588e0cb5b4f9668c4 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:48.014196 31810 raft_consensus.cc:374] T dbd0cada7fe64da588e0cb5b4f9668c4 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:48.014319 31810 raft_consensus.cc:729] T dbd0cada7fe64da588e0cb5b4f9668c4 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:48.014421 31814 tablet_bootstrap.cc:492] T dbd0cada7fe64da588e0cb5b4f9668c4 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:48.014951 31814 ts_tablet_manager.cc:1367] T dbd0cada7fe64da588e0cb5b4f9668c4 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240503 08:12:48.015008 27859 catalog_manager.cc:5484] T 45f47478f49149b895060791655503ae P 833582ff68c94269be364ebd187a75db reported cstate change: term changed from 0 to 1, leader changed from <none> to 833582ff68c94269be364ebd187a75db (127.20.230.65). New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:12:48.015756 31814 raft_consensus.cc:348] T dbd0cada7fe64da588e0cb5b4f9668c4 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:48.016000 31814 raft_consensus.cc:374] T dbd0cada7fe64da588e0cb5b4f9668c4 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:48.016091 31814 raft_consensus.cc:729] T dbd0cada7fe64da588e0cb5b4f9668c4 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:48.014652 31810 consensus_queue.cc:260] T dbd0cada7fe64da588e0cb5b4f9668c4 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:48.016386 31814 consensus_queue.cc:260] T dbd0cada7fe64da588e0cb5b4f9668c4 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:48.016820 31810 ts_tablet_manager.cc:1398] T dbd0cada7fe64da588e0cb5b4f9668c4 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:48.016956 31814 ts_tablet_manager.cc:1398] T dbd0cada7fe64da588e0cb5b4f9668c4 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:12:48.017671 31813 tablet_bootstrap.cc:492] T dbd0cada7fe64da588e0cb5b4f9668c4 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:48.017848 31813 ts_tablet_manager.cc:1367] T dbd0cada7fe64da588e0cb5b4f9668c4 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20240503 08:12:48.018594 31813 raft_consensus.cc:348] T dbd0cada7fe64da588e0cb5b4f9668c4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:48.018957 31813 raft_consensus.cc:374] T dbd0cada7fe64da588e0cb5b4f9668c4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:48.019117 31813 raft_consensus.cc:729] T dbd0cada7fe64da588e0cb5b4f9668c4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:48.019382 31813 consensus_queue.cc:260] T dbd0cada7fe64da588e0cb5b4f9668c4 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:48.019856 31813 ts_tablet_manager.cc:1398] T dbd0cada7fe64da588e0cb5b4f9668c4 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:48.043188 31783 raft_consensus.cc:482] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:48.043351 31783 raft_consensus.cc:504] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:48.044152 31783 leader_election.cc:288] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:12:48.044660 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b12ceaf7f8d04fad8b07b36459da7116" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:12:48.044919 28002 raft_consensus.cc:2454] T b12ceaf7f8d04fad8b07b36459da7116 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 0.
I20240503 08:12:48.045343 30962 leader_election.cc:302] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:48.045707 31783 raft_consensus.cc:2781] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:48.045851 31783 raft_consensus.cc:482] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:48.045971 31783 raft_consensus.cc:3037] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:48.048871 31783 raft_consensus.cc:504] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:48.049463 31783 leader_election.cc:288] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:12:48.050014 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b12ceaf7f8d04fad8b07b36459da7116" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:12:48.050226 28002 raft_consensus.cc:3037] T b12ceaf7f8d04fad8b07b36459da7116 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:48.052847 28002 raft_consensus.cc:2454] T b12ceaf7f8d04fad8b07b36459da7116 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 1.
I20240503 08:12:48.053316 30962 leader_election.cc:302] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:48.053730 31783 raft_consensus.cc:2781] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:48.054877 31783 raft_consensus.cc:686] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 LEADER]: Becoming Leader. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Running, Role: LEADER
I20240503 08:12:48.055284 31783 consensus_queue.cc:237] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:48.060727 27856 catalog_manager.cc:5484] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71). New cstate: current_term: 1 leader_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:12:48.062566 28261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b12ceaf7f8d04fad8b07b36459da7116" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:12:48.062690 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b12ceaf7f8d04fad8b07b36459da7116" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:12:48.062803 28261 raft_consensus.cc:2454] T b12ceaf7f8d04fad8b07b36459da7116 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 0.
I20240503 08:12:48.084821 31771 raft_consensus.cc:482] T dbd0cada7fe64da588e0cb5b4f9668c4 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:48.085001 31771 raft_consensus.cc:504] T dbd0cada7fe64da588e0cb5b4f9668c4 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:48.085633 31771 leader_election.cc:288] T dbd0cada7fe64da588e0cb5b4f9668c4 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:48.086172 30894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbd0cada7fe64da588e0cb5b4f9668c4" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:12:48.086223 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbd0cada7fe64da588e0cb5b4f9668c4" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:48.086416 31027 raft_consensus.cc:2454] T dbd0cada7fe64da588e0cb5b4f9668c4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:12:48.086441 30894 raft_consensus.cc:2454] T dbd0cada7fe64da588e0cb5b4f9668c4 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:12:48.086865 30695 leader_election.cc:302] T dbd0cada7fe64da588e0cb5b4f9668c4 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:48.087216 31771 raft_consensus.cc:2781] T dbd0cada7fe64da588e0cb5b4f9668c4 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:48.087319 31771 raft_consensus.cc:482] T dbd0cada7fe64da588e0cb5b4f9668c4 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:48.087406 31771 raft_consensus.cc:3037] T dbd0cada7fe64da588e0cb5b4f9668c4 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:48.089392 31771 raft_consensus.cc:504] T dbd0cada7fe64da588e0cb5b4f9668c4 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:48.089975 31771 leader_election.cc:288] T dbd0cada7fe64da588e0cb5b4f9668c4 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Requested vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:48.090339 30894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbd0cada7fe64da588e0cb5b4f9668c4" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:12:48.090443 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbd0cada7fe64da588e0cb5b4f9668c4" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:12:48.090541 30894 raft_consensus.cc:3037] T dbd0cada7fe64da588e0cb5b4f9668c4 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:48.090634 31027 raft_consensus.cc:3037] T dbd0cada7fe64da588e0cb5b4f9668c4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:48.092383 30894 raft_consensus.cc:2454] T dbd0cada7fe64da588e0cb5b4f9668c4 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:12:48.092753 30694 leader_election.cc:302] T dbd0cada7fe64da588e0cb5b4f9668c4 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:48.092909 31027 raft_consensus.cc:2454] T dbd0cada7fe64da588e0cb5b4f9668c4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:12:48.093052 31771 raft_consensus.cc:2781] T dbd0cada7fe64da588e0cb5b4f9668c4 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:48.093178 31771 raft_consensus.cc:686] T dbd0cada7fe64da588e0cb5b4f9668c4 P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Becoming Leader. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Running, Role: LEADER
I20240503 08:12:48.093466 31771 consensus_queue.cc:237] T dbd0cada7fe64da588e0cb5b4f9668c4 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:48.096778 27862 catalog_manager.cc:5484] T dbd0cada7fe64da588e0cb5b4f9668c4 P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69). New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:12:48.216080 27862 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:48.222504 30894 raft_consensus.cc:1261] T dbd0cada7fe64da588e0cb5b4f9668c4 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:48.222709 31027 raft_consensus.cc:1261] T dbd0cada7fe64da588e0cb5b4f9668c4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:48.223378 31771 consensus_queue.cc:1035] T dbd0cada7fe64da588e0cb5b4f9668c4 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:48.223743 31688 consensus_queue.cc:1035] T dbd0cada7fe64da588e0cb5b4f9668c4 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:48.230003 27862 catalog_manager.cc:5576] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915) does not have the latest schema for tablet dbd0cada7fe64da588e0cb5b4f9668c4 (table rebalancer_dynamic_table [id=0f892d4db3044188bdb00b55c3b81389]). Expected version 1 got 0
I20240503 08:12:48.238348 31840 tablet.cc:1722] T dbd0cada7fe64da588e0cb5b4f9668c4 P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:48.238593 27862 catalog_manager.cc:5576] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747) does not have the latest schema for tablet dbd0cada7fe64da588e0cb5b4f9668c4 (table rebalancer_dynamic_table [id=0f892d4db3044188bdb00b55c3b81389]). Expected version 1 got 0
I20240503 08:12:48.239049 31840 tablet.cc:1620] T dbd0cada7fe64da588e0cb5b4f9668c4 P ced21ca983b749d0b4d4f6a0e2426bf8: MemRowSet was empty: no flush needed.
I20240503 08:12:48.241199 31798 tablet.cc:1722] T dbd0cada7fe64da588e0cb5b4f9668c4 P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:48.241691 31798 tablet.cc:1620] T dbd0cada7fe64da588e0cb5b4f9668c4 P ce1496a47d084365ad786cd09e472a2c: MemRowSet was empty: no flush needed.
I20240503 08:12:48.241497 31839 tablet.cc:1722] T dbd0cada7fe64da588e0cb5b4f9668c4 P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:48.243244 31839 tablet.cc:1620] T dbd0cada7fe64da588e0cb5b4f9668c4 P 6d1d7de19459435fbde1999e3dfd226b: MemRowSet was empty: no flush needed.
I20240503 08:12:48.245947 27856 catalog_manager.cc:5576] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747) does not have the latest schema for tablet dbd0cada7fe64da588e0cb5b4f9668c4 (table rebalancer_dynamic_table [id=0f892d4db3044188bdb00b55c3b81389]). Expected version 1 got 0
I20240503 08:12:48.247786 27862 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=0f892d4db3044188bdb00b55c3b81389] alter complete (version 1)
I20240503 08:12:48.299976 31677 raft_consensus.cc:482] T 549f64ad9ef64a3da619c5df5a411e1c P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:48.300146 31677 raft_consensus.cc:504] T 549f64ad9ef64a3da619c5df5a411e1c P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:48.300740 31677 leader_election.cc:288] T 549f64ad9ef64a3da619c5df5a411e1c P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:12:48.301198 30894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "549f64ad9ef64a3da619c5df5a411e1c" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:12:48.301211 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "549f64ad9ef64a3da619c5df5a411e1c" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:12:48.301419 30894 raft_consensus.cc:2454] T 549f64ad9ef64a3da619c5df5a411e1c P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 0.
I20240503 08:12:48.301461 30761 raft_consensus.cc:2454] T 549f64ad9ef64a3da619c5df5a411e1c P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 0.
I20240503 08:12:48.301903 31094 leader_election.cc:302] T 549f64ad9ef64a3da619c5df5a411e1c P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:48.302191 31677 raft_consensus.cc:2781] T 549f64ad9ef64a3da619c5df5a411e1c P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:48.302301 31677 raft_consensus.cc:482] T 549f64ad9ef64a3da619c5df5a411e1c P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:48.302393 31677 raft_consensus.cc:3037] T 549f64ad9ef64a3da619c5df5a411e1c P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:48.304448 31677 raft_consensus.cc:504] T 549f64ad9ef64a3da619c5df5a411e1c P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:48.305047 31677 leader_election.cc:288] T 549f64ad9ef64a3da619c5df5a411e1c P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 election: Requested vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:12:48.305487 30894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "549f64ad9ef64a3da619c5df5a411e1c" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:12:48.305596 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "549f64ad9ef64a3da619c5df5a411e1c" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:12:48.305687 30894 raft_consensus.cc:3037] T 549f64ad9ef64a3da619c5df5a411e1c P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:48.305830 30761 raft_consensus.cc:3037] T 549f64ad9ef64a3da619c5df5a411e1c P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:48.307657 30894 raft_consensus.cc:2454] T 549f64ad9ef64a3da619c5df5a411e1c P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 1.
I20240503 08:12:48.307816 30761 raft_consensus.cc:2454] T 549f64ad9ef64a3da619c5df5a411e1c P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 1.
I20240503 08:12:48.308305 31096 leader_election.cc:302] T 549f64ad9ef64a3da619c5df5a411e1c P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, b00f244c69df401997ea99d492218a98, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:48.308670 31677 raft_consensus.cc:2781] T 549f64ad9ef64a3da619c5df5a411e1c P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:48.308876 31677 raft_consensus.cc:686] T 549f64ad9ef64a3da619c5df5a411e1c P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Becoming Leader. State: Replica: b00f244c69df401997ea99d492218a98, State: Running, Role: LEADER
I20240503 08:12:48.309242 31677 consensus_queue.cc:237] T 549f64ad9ef64a3da619c5df5a411e1c P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:48.315951 27856 catalog_manager.cc:5484] T 549f64ad9ef64a3da619c5df5a411e1c P b00f244c69df401997ea99d492218a98 reported cstate change: term changed from 0 to 1, leader changed from <none> to b00f244c69df401997ea99d492218a98 (127.20.230.72). New cstate: current_term: 1 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:12:48.353981 27856 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:48.361724 31839 tablet.cc:1722] T dbd0cada7fe64da588e0cb5b4f9668c4 P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:48.362267 31839 tablet.cc:1620] T dbd0cada7fe64da588e0cb5b4f9668c4 P 6d1d7de19459435fbde1999e3dfd226b: MemRowSet was empty: no flush needed.
I20240503 08:12:48.362514 31840 tablet.cc:1722] T dbd0cada7fe64da588e0cb5b4f9668c4 P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:48.362766 31798 tablet.cc:1722] T dbd0cada7fe64da588e0cb5b4f9668c4 P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:48.363092 31840 tablet.cc:1620] T dbd0cada7fe64da588e0cb5b4f9668c4 P ced21ca983b749d0b4d4f6a0e2426bf8: MemRowSet was empty: no flush needed.
I20240503 08:12:48.363255 31798 tablet.cc:1620] T dbd0cada7fe64da588e0cb5b4f9668c4 P ce1496a47d084365ad786cd09e472a2c: MemRowSet was empty: no flush needed.
I20240503 08:12:48.372004 27862 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=0f892d4db3044188bdb00b55c3b81389] alter complete (version 2)
I20240503 08:12:48.448236 31745 consensus_queue.cc:1035] T 45f47478f49149b895060791655503ae P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:48.453028 31745 consensus_queue.cc:1035] T 45f47478f49149b895060791655503ae P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:48.475478 27859 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:48.483683 31798 tablet.cc:1722] T dbd0cada7fe64da588e0cb5b4f9668c4 P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:48.483909 31839 tablet.cc:1722] T dbd0cada7fe64da588e0cb5b4f9668c4 P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:48.484891 31840 tablet.cc:1722] T dbd0cada7fe64da588e0cb5b4f9668c4 P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:48.494868 27859 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table_new_name [id=0f892d4db3044188bdb00b55c3b81389] alter complete (version 3)
I20240503 08:12:48.520961 31783 consensus_queue.cc:1035] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:48.524695 28261 raft_consensus.cc:3037] T b12ceaf7f8d04fad8b07b36459da7116 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:48.528380 31832 consensus_queue.cc:1035] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:48.598325 27859 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:48.598631 27859 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:48.601739 27859 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet dbd0cada7fe64da588e0cb5b4f9668c4
I20240503 08:12:48.602628 31007 tablet_service.cc:1514] Processing DeleteTablet for tablet dbd0cada7fe64da588e0cb5b4f9668c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:48 UTC) from {username='slave'} at 127.0.0.1:47020
I20240503 08:12:48.602635 30874 tablet_service.cc:1514] Processing DeleteTablet for tablet dbd0cada7fe64da588e0cb5b4f9668c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:48 UTC) from {username='slave'} at 127.0.0.1:34426
I20240503 08:12:48.603169 31854 tablet_replica.cc:330] T dbd0cada7fe64da588e0cb5b4f9668c4 P ced21ca983b749d0b4d4f6a0e2426bf8: stopping tablet replica
I20240503 08:12:48.603121 30741 tablet_service.cc:1514] Processing DeleteTablet for tablet dbd0cada7fe64da588e0cb5b4f9668c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:48 UTC) from {username='slave'} at 127.0.0.1:46794
I20240503 08:12:48.603391 31854 raft_consensus.cc:2229] T dbd0cada7fe64da588e0cb5b4f9668c4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:48.603610 31854 raft_consensus.cc:2258] T dbd0cada7fe64da588e0cb5b4f9668c4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:48.604342 31854 ts_tablet_manager.cc:1875] T dbd0cada7fe64da588e0cb5b4f9668c4 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:48.609304 31854 ts_tablet_manager.cc:1888] T dbd0cada7fe64da588e0cb5b4f9668c4 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:48.609468 31854 log.cc:1196] T dbd0cada7fe64da588e0cb5b4f9668c4 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal/wals/dbd0cada7fe64da588e0cb5b4f9668c4
I20240503 08:12:48.609925 31854 ts_tablet_manager.cc:1909] T dbd0cada7fe64da588e0cb5b4f9668c4 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting consensus metadata
I20240503 08:12:48.610877 27847 catalog_manager.cc:4830] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): tablet dbd0cada7fe64da588e0cb5b4f9668c4 (table rebalancer_dynamic_table_new_name [id=0f892d4db3044188bdb00b55c3b81389]) successfully deleted
I20240503 08:12:48.614133 31855 tablet_replica.cc:330] T dbd0cada7fe64da588e0cb5b4f9668c4 P ce1496a47d084365ad786cd09e472a2c: stopping tablet replica
I20240503 08:12:48.614343 31855 raft_consensus.cc:2229] T dbd0cada7fe64da588e0cb5b4f9668c4 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:48.614552 31855 raft_consensus.cc:2258] T dbd0cada7fe64da588e0cb5b4f9668c4 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:48.615348 31855 ts_tablet_manager.cc:1875] T dbd0cada7fe64da588e0cb5b4f9668c4 P ce1496a47d084365ad786cd09e472a2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:48.619733 31855 ts_tablet_manager.cc:1888] T dbd0cada7fe64da588e0cb5b4f9668c4 P ce1496a47d084365ad786cd09e472a2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:48.619911 31855 log.cc:1196] T dbd0cada7fe64da588e0cb5b4f9668c4 P ce1496a47d084365ad786cd09e472a2c: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal/wals/dbd0cada7fe64da588e0cb5b4f9668c4
I20240503 08:12:48.620357 31855 ts_tablet_manager.cc:1909] T dbd0cada7fe64da588e0cb5b4f9668c4 P ce1496a47d084365ad786cd09e472a2c: Deleting consensus metadata
I20240503 08:12:48.621454 27846 catalog_manager.cc:4830] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): tablet dbd0cada7fe64da588e0cb5b4f9668c4 (table rebalancer_dynamic_table_new_name [id=0f892d4db3044188bdb00b55c3b81389]) successfully deleted
I20240503 08:12:48.626169 31856 tablet_replica.cc:330] T dbd0cada7fe64da588e0cb5b4f9668c4 P 6d1d7de19459435fbde1999e3dfd226b: stopping tablet replica
I20240503 08:12:48.626368 31856 raft_consensus.cc:2229] T dbd0cada7fe64da588e0cb5b4f9668c4 P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:48.626688 31856 raft_consensus.cc:2258] T dbd0cada7fe64da588e0cb5b4f9668c4 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:48.627403 31856 ts_tablet_manager.cc:1875] T dbd0cada7fe64da588e0cb5b4f9668c4 P 6d1d7de19459435fbde1999e3dfd226b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:48.631896 31856 ts_tablet_manager.cc:1888] T dbd0cada7fe64da588e0cb5b4f9668c4 P 6d1d7de19459435fbde1999e3dfd226b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:48.632030 31856 log.cc:1196] T dbd0cada7fe64da588e0cb5b4f9668c4 P 6d1d7de19459435fbde1999e3dfd226b: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal/wals/dbd0cada7fe64da588e0cb5b4f9668c4
I20240503 08:12:48.632433 31856 ts_tablet_manager.cc:1909] T dbd0cada7fe64da588e0cb5b4f9668c4 P 6d1d7de19459435fbde1999e3dfd226b: Deleting consensus metadata
I20240503 08:12:48.633293 27848 catalog_manager.cc:4830] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): tablet dbd0cada7fe64da588e0cb5b4f9668c4 (table rebalancer_dynamic_table_new_name [id=0f892d4db3044188bdb00b55c3b81389]) successfully deleted
I20240503 08:12:48.704118 27859 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:48.711282 31140 tablet_service.cc:1467] Processing CreateTablet for tablet 6e5e541c506a4d7aaef7bf46b24ba0d7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ab9ca11f0d204e0c95fe8293dc112cb2]), partition=
I20240503 08:12:48.711370 30874 tablet_service.cc:1467] Processing CreateTablet for tablet 6e5e541c506a4d7aaef7bf46b24ba0d7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ab9ca11f0d204e0c95fe8293dc112cb2]), partition=
I20240503 08:12:48.711689 31140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e5e541c506a4d7aaef7bf46b24ba0d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:48.711705 30874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e5e541c506a4d7aaef7bf46b24ba0d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:48.714622 28372 tablet_service.cc:1467] Processing CreateTablet for tablet 6e5e541c506a4d7aaef7bf46b24ba0d7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ab9ca11f0d204e0c95fe8293dc112cb2]), partition=
I20240503 08:12:48.714960 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e5e541c506a4d7aaef7bf46b24ba0d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:48.723897 31859 tablet_bootstrap.cc:492] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:48.724215 31860 tablet_bootstrap.cc:492] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:48.725584 31860 tablet_bootstrap.cc:654] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:48.725579 31859 tablet_bootstrap.cc:654] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:48.726583 31858 tablet_bootstrap.cc:492] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:48.727455 31860 tablet_bootstrap.cc:492] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:48.727699 31860 ts_tablet_manager.cc:1367] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:12:48.728474 31860 raft_consensus.cc:348] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:48.728693 31860 raft_consensus.cc:374] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:48.728780 31860 raft_consensus.cc:729] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:48.729033 31860 consensus_queue.cc:260] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:48.729171 31858 tablet_bootstrap.cc:654] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:48.729370 31859 tablet_bootstrap.cc:492] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:48.729590 31860 ts_tablet_manager.cc:1398] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:48.730214 31859 ts_tablet_manager.cc:1367] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 08:12:48.731323 31858 tablet_bootstrap.cc:492] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:48.731503 31858 ts_tablet_manager.cc:1367] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 08:12:48.731319 31859 raft_consensus.cc:348] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:48.731743 31859 raft_consensus.cc:374] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:48.731937 31859 raft_consensus.cc:729] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:48.732275 31858 raft_consensus.cc:348] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:48.732517 31858 raft_consensus.cc:374] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:48.732638 31858 raft_consensus.cc:729] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:48.732869 31858 consensus_queue.cc:260] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:48.733537 31858 ts_tablet_manager.cc:1398] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:48.732448 31859 consensus_queue.cc:260] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:48.734153 31859 ts_tablet_manager.cc:1398] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:48.760718 31677 consensus_queue.cc:1035] T 549f64ad9ef64a3da619c5df5a411e1c P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:48.764914 31841 consensus_queue.cc:1035] T 549f64ad9ef64a3da619c5df5a411e1c P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:48.819762 31841 raft_consensus.cc:482] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:48.819962 31841 raft_consensus.cc:504] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:48.820516 31841 leader_election.cc:288] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:48.821002 28393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e5e541c506a4d7aaef7bf46b24ba0d7" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:48.821022 30894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e5e541c506a4d7aaef7bf46b24ba0d7" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:12:48.821202 30894 raft_consensus.cc:2454] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 0.
I20240503 08:12:48.821221 28393 raft_consensus.cc:2454] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 0.
I20240503 08:12:48.821600 31094 leader_election.cc:302] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:48.821892 31841 raft_consensus.cc:2781] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:48.821971 31841 raft_consensus.cc:482] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:48.822044 31841 raft_consensus.cc:3037] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:48.823961 31841 raft_consensus.cc:504] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:48.824805 31841 leader_election.cc:288] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 election: Requested vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:48.824903 30894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e5e541c506a4d7aaef7bf46b24ba0d7" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:12:48.824998 28393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e5e541c506a4d7aaef7bf46b24ba0d7" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:12:48.825076 30894 raft_consensus.cc:3037] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:48.825202 28393 raft_consensus.cc:3037] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:48.826956 30894 raft_consensus.cc:2454] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 1.
I20240503 08:12:48.827075 28393 raft_consensus.cc:2454] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 1.
I20240503 08:12:48.827318 31094 leader_election.cc:302] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:48.827538 31841 raft_consensus.cc:2781] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:48.827697 31841 raft_consensus.cc:686] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Becoming Leader. State: Replica: b00f244c69df401997ea99d492218a98, State: Running, Role: LEADER
I20240503 08:12:48.827926 31841 consensus_queue.cc:237] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:48.830710 27862 catalog_manager.cc:5484] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P b00f244c69df401997ea99d492218a98 reported cstate change: term changed from 0 to 1, leader changed from <none> to b00f244c69df401997ea99d492218a98 (127.20.230.72). New cstate: current_term: 1 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:12:48.834913 27862 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_extra_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:48.842329 30874 tablet_service.cc:1467] Processing CreateTablet for tablet 68b1a80883614b1ca54fbd88bc769dff (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=e4d6c0dac75042f984dc1aa8fa801cee]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:48.842793 30874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68b1a80883614b1ca54fbd88bc769dff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:48.842819 28372 tablet_service.cc:1467] Processing CreateTablet for tablet 68b1a80883614b1ca54fbd88bc769dff (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=e4d6c0dac75042f984dc1aa8fa801cee]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:48.842850 31140 tablet_service.cc:1467] Processing CreateTablet for tablet 68b1a80883614b1ca54fbd88bc769dff (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=e4d6c0dac75042f984dc1aa8fa801cee]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:48.842993 30741 tablet_service.cc:1467] Processing CreateTablet for tablet f53acf725b62447382e9bae95e6e74d5 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=e4d6c0dac75042f984dc1aa8fa801cee]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:48.843267 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68b1a80883614b1ca54fbd88bc769dff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:48.843261 31140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68b1a80883614b1ca54fbd88bc769dff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:48.843427 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f53acf725b62447382e9bae95e6e74d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:48.847641 30873 tablet_service.cc:1467] Processing CreateTablet for tablet f53acf725b62447382e9bae95e6e74d5 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=e4d6c0dac75042f984dc1aa8fa801cee]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:48.848037 30873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f53acf725b62447382e9bae95e6e74d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:48.849123 31007 tablet_service.cc:1467] Processing CreateTablet for tablet f53acf725b62447382e9bae95e6e74d5 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=e4d6c0dac75042f984dc1aa8fa801cee]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:48.849586 31007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f53acf725b62447382e9bae95e6e74d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:48.849925 31858 tablet_bootstrap.cc:492] T 68b1a80883614b1ca54fbd88bc769dff P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:48.850616 31859 tablet_bootstrap.cc:492] T 68b1a80883614b1ca54fbd88bc769dff P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:48.851554 31858 tablet_bootstrap.cc:654] T 68b1a80883614b1ca54fbd88bc769dff P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:48.851764 30740 tablet_service.cc:1467] Processing CreateTablet for tablet 50e4d2c569344dde915eec12f6f3d747 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=e4d6c0dac75042f984dc1aa8fa801cee]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:48.852149 30740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50e4d2c569344dde915eec12f6f3d747. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:48.852159 31859 tablet_bootstrap.cc:654] T 68b1a80883614b1ca54fbd88bc769dff P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:48.853482 31139 tablet_service.cc:1467] Processing CreateTablet for tablet 50e4d2c569344dde915eec12f6f3d747 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=e4d6c0dac75042f984dc1aa8fa801cee]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:48.853857 31139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50e4d2c569344dde915eec12f6f3d747. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:48.856676 31008 tablet_service.cc:1467] Processing CreateTablet for tablet 50e4d2c569344dde915eec12f6f3d747 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=e4d6c0dac75042f984dc1aa8fa801cee]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:48.857051 31008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50e4d2c569344dde915eec12f6f3d747. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:48.858801 31860 tablet_bootstrap.cc:492] T 68b1a80883614b1ca54fbd88bc769dff P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:48.860318 31860 tablet_bootstrap.cc:654] T 68b1a80883614b1ca54fbd88bc769dff P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:48.860620 31859 tablet_bootstrap.cc:492] T 68b1a80883614b1ca54fbd88bc769dff P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:48.860738 31859 ts_tablet_manager.cc:1367] T 68b1a80883614b1ca54fbd88bc769dff P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20240503 08:12:48.861444 31859 raft_consensus.cc:348] T 68b1a80883614b1ca54fbd88bc769dff P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:48.861672 31859 raft_consensus.cc:374] T 68b1a80883614b1ca54fbd88bc769dff P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:48.861747 31859 raft_consensus.cc:729] T 68b1a80883614b1ca54fbd88bc769dff P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:48.862017 31859 consensus_queue.cc:260] T 68b1a80883614b1ca54fbd88bc769dff P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:48.862852 31859 ts_tablet_manager.cc:1398] T 68b1a80883614b1ca54fbd88bc769dff P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:48.865310 31859 tablet_bootstrap.cc:492] T 50e4d2c569344dde915eec12f6f3d747 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:48.867048 31859 tablet_bootstrap.cc:654] T 50e4d2c569344dde915eec12f6f3d747 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:48.867224 31872 tablet_bootstrap.cc:492] T 50e4d2c569344dde915eec12f6f3d747 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:48.870764 31860 tablet_bootstrap.cc:492] T 68b1a80883614b1ca54fbd88bc769dff P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:48.871021 31860 ts_tablet_manager.cc:1367] T 68b1a80883614b1ca54fbd88bc769dff P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240503 08:12:48.872175 31872 tablet_bootstrap.cc:654] T 50e4d2c569344dde915eec12f6f3d747 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:48.873960 31860 raft_consensus.cc:348] T 68b1a80883614b1ca54fbd88bc769dff P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:48.874279 31860 raft_consensus.cc:374] T 68b1a80883614b1ca54fbd88bc769dff P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:48.874403 31860 raft_consensus.cc:729] T 68b1a80883614b1ca54fbd88bc769dff P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:48.874454 31871 tablet_bootstrap.cc:492] T 50e4d2c569344dde915eec12f6f3d747 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:48.874765 31860 consensus_queue.cc:260] T 68b1a80883614b1ca54fbd88bc769dff P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:48.876410 31860 ts_tablet_manager.cc:1398] T 68b1a80883614b1ca54fbd88bc769dff P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:48.877197 31859 tablet_bootstrap.cc:492] T 50e4d2c569344dde915eec12f6f3d747 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:48.877393 31859 ts_tablet_manager.cc:1367] T 50e4d2c569344dde915eec12f6f3d747 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20240503 08:12:48.877743 31871 tablet_bootstrap.cc:654] T 50e4d2c569344dde915eec12f6f3d747 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:48.878556 31859 raft_consensus.cc:348] T 50e4d2c569344dde915eec12f6f3d747 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:48.878834 31859 raft_consensus.cc:374] T 50e4d2c569344dde915eec12f6f3d747 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:48.878973 31859 raft_consensus.cc:729] T 50e4d2c569344dde915eec12f6f3d747 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:48.879273 31859 consensus_queue.cc:260] T 50e4d2c569344dde915eec12f6f3d747 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:48.880070 31858 tablet_bootstrap.cc:492] T 68b1a80883614b1ca54fbd88bc769dff P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:48.880306 31858 ts_tablet_manager.cc:1367] T 68b1a80883614b1ca54fbd88bc769dff P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.004s
I20240503 08:12:48.880331 31859 ts_tablet_manager.cc:1398] T 50e4d2c569344dde915eec12f6f3d747 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:48.881203 31858 raft_consensus.cc:348] T 68b1a80883614b1ca54fbd88bc769dff P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:48.881520 31858 raft_consensus.cc:374] T 68b1a80883614b1ca54fbd88bc769dff P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:48.881652 31858 raft_consensus.cc:729] T 68b1a80883614b1ca54fbd88bc769dff P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:48.883658 31858 consensus_queue.cc:260] T 68b1a80883614b1ca54fbd88bc769dff P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:48.884482 31858 ts_tablet_manager.cc:1398] T 68b1a80883614b1ca54fbd88bc769dff P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 08:12:48.884899 31858 tablet_bootstrap.cc:492] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:48.886694 31858 tablet_bootstrap.cc:654] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:48.887799 31871 tablet_bootstrap.cc:492] T 50e4d2c569344dde915eec12f6f3d747 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:48.887959 31871 ts_tablet_manager.cc:1367] T 50e4d2c569344dde915eec12f6f3d747 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.003s
I20240503 08:12:48.888729 31872 tablet_bootstrap.cc:492] T 50e4d2c569344dde915eec12f6f3d747 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:48.888937 31872 ts_tablet_manager.cc:1367] T 50e4d2c569344dde915eec12f6f3d747 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.006s
I20240503 08:12:48.888917 31871 raft_consensus.cc:348] T 50e4d2c569344dde915eec12f6f3d747 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:48.889201 31871 raft_consensus.cc:374] T 50e4d2c569344dde915eec12f6f3d747 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:48.889361 31871 raft_consensus.cc:729] T 50e4d2c569344dde915eec12f6f3d747 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:48.889791 31871 consensus_queue.cc:260] T 50e4d2c569344dde915eec12f6f3d747 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:48.890610 31858 tablet_bootstrap.cc:492] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:48.890745 31858 ts_tablet_manager.cc:1367] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 08:12:48.891284 31871 ts_tablet_manager.cc:1398] T 50e4d2c569344dde915eec12f6f3d747 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:12:48.891465 31858 raft_consensus.cc:348] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:48.891664 31858 raft_consensus.cc:374] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:48.891615 31871 tablet_bootstrap.cc:492] T f53acf725b62447382e9bae95e6e74d5 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:48.891758 31858 raft_consensus.cc:729] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:48.891677 31872 raft_consensus.cc:348] T 50e4d2c569344dde915eec12f6f3d747 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:48.891857 31872 raft_consensus.cc:374] T 50e4d2c569344dde915eec12f6f3d747 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:48.891997 31872 raft_consensus.cc:729] T 50e4d2c569344dde915eec12f6f3d747 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:48.892004 31858 consensus_queue.cc:260] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:48.892298 31872 consensus_queue.cc:260] T 50e4d2c569344dde915eec12f6f3d747 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:48.892752 31858 ts_tablet_manager.cc:1398] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:48.893326 31872 ts_tablet_manager.cc:1398] T 50e4d2c569344dde915eec12f6f3d747 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:48.893641 31872 tablet_bootstrap.cc:492] T f53acf725b62447382e9bae95e6e74d5 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:48.895849 31872 tablet_bootstrap.cc:654] T f53acf725b62447382e9bae95e6e74d5 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:48.899818 31871 tablet_bootstrap.cc:654] T f53acf725b62447382e9bae95e6e74d5 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:48.899808 31699 raft_consensus.cc:482] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:48.899926 31872 tablet_bootstrap.cc:492] T f53acf725b62447382e9bae95e6e74d5 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:48.899942 31699 raft_consensus.cc:504] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:48.900211 31872 ts_tablet_manager.cc:1367] T f53acf725b62447382e9bae95e6e74d5 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20240503 08:12:48.900651 31699 leader_election.cc:288] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:48.901077 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53acf725b62447382e9bae95e6e74d5" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:12:48.901083 31872 raft_consensus.cc:348] T f53acf725b62447382e9bae95e6e74d5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:48.901381 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53acf725b62447382e9bae95e6e74d5" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:48.901403 31872 raft_consensus.cc:374] T f53acf725b62447382e9bae95e6e74d5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240503 08:12:48.901645 30829 leader_election.cc:341] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:48.901703 31872 raft_consensus.cc:729] T f53acf725b62447382e9bae95e6e74d5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:48.901971 31872 consensus_queue.cc:260] T f53acf725b62447382e9bae95e6e74d5 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:48.902264 31027 raft_consensus.cc:2454] T f53acf725b62447382e9bae95e6e74d5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:12:48.902529 31872 ts_tablet_manager.cc:1398] T f53acf725b62447382e9bae95e6e74d5 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:12:48.902599 31871 tablet_bootstrap.cc:492] T f53acf725b62447382e9bae95e6e74d5 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:48.902653 30828 leader_election.cc:302] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c, ced21ca983b749d0b4d4f6a0e2426bf8; no voters: 6d1d7de19459435fbde1999e3dfd226b
I20240503 08:12:48.902745 31871 ts_tablet_manager.cc:1367] T f53acf725b62447382e9bae95e6e74d5 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.002s
I20240503 08:12:48.903123 31699 raft_consensus.cc:2781] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:48.903236 31699 raft_consensus.cc:482] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:48.903322 31699 raft_consensus.cc:3037] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:48.903568 31871 raft_consensus.cc:348] T f53acf725b62447382e9bae95e6e74d5 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:48.903714 31871 raft_consensus.cc:374] T f53acf725b62447382e9bae95e6e74d5 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:48.903756 31871 raft_consensus.cc:729] T f53acf725b62447382e9bae95e6e74d5 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:48.903949 31871 consensus_queue.cc:260] T f53acf725b62447382e9bae95e6e74d5 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:48.904384 31871 ts_tablet_manager.cc:1398] T f53acf725b62447382e9bae95e6e74d5 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:48.905521 31699 raft_consensus.cc:504] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:48.906690 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53acf725b62447382e9bae95e6e74d5" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:12:48.906890 31027 raft_consensus.cc:3037] T f53acf725b62447382e9bae95e6e74d5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:48.907147 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53acf725b62447382e9bae95e6e74d5" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:12:48.907352 30761 raft_consensus.cc:3037] T f53acf725b62447382e9bae95e6e74d5 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:48.908850 31027 raft_consensus.cc:2454] T f53acf725b62447382e9bae95e6e74d5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:12:48.909219 30828 leader_election.cc:302] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:48.909328 30761 raft_consensus.cc:2454] T f53acf725b62447382e9bae95e6e74d5 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:12:48.911224 31699 leader_election.cc:288] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Requested vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:48.911825 31699 raft_consensus.cc:2781] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:48.911993 31699 raft_consensus.cc:686] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Becoming Leader. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Running, Role: LEADER
I20240503 08:12:48.912334 31699 consensus_queue.cc:237] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:48.915340 27859 catalog_manager.cc:5484] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c reported cstate change: term changed from 0 to 1, leader changed from <none> to ce1496a47d084365ad786cd09e472a2c (127.20.230.70). New cstate: current_term: 1 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:12:48.966743 27859 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:48.975879 30894 raft_consensus.cc:1261] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:48.976148 28393 raft_consensus.cc:1261] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:48.976470 31677 consensus_queue.cc:1035] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:48.977533 31677 consensus_queue.cc:1035] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:48.984257 31798 tablet.cc:1722] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:48.984792 31798 tablet.cc:1620] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P ce1496a47d084365ad786cd09e472a2c: MemRowSet was empty: no flush needed.
I20240503 08:12:48.989850 27862 catalog_manager.cc:5576] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747) does not have the latest schema for tablet 6e5e541c506a4d7aaef7bf46b24ba0d7 (table rebalancer_dynamic_table [id=ab9ca11f0d204e0c95fe8293dc112cb2]). Expected version 1 got 0
I20240503 08:12:48.990654 31883 tablet.cc:1722] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P b00f244c69df401997ea99d492218a98: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:48.991257 31883 tablet.cc:1620] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P b00f244c69df401997ea99d492218a98: MemRowSet was empty: no flush needed.
I20240503 08:12:49.001339 27862 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=ab9ca11f0d204e0c95fe8293dc112cb2] alter complete (version 1)
I20240503 08:12:49.003057 27855 catalog_manager.cc:5576] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211) does not have the latest schema for tablet 6e5e541c506a4d7aaef7bf46b24ba0d7 (table rebalancer_dynamic_table [id=ab9ca11f0d204e0c95fe8293dc112cb2]). Expected version 1 got 0
I20240503 08:12:49.005184 31885 tablet.cc:1722] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P d09a11a03989421b9f28019dc1c14cc2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:49.007153 27855 catalog_manager.cc:5576] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211) does not have the latest schema for tablet 6e5e541c506a4d7aaef7bf46b24ba0d7 (table rebalancer_dynamic_table [id=ab9ca11f0d204e0c95fe8293dc112cb2]). Expected version 1 got 0
I20240503 08:12:49.007786 31885 tablet.cc:1620] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P d09a11a03989421b9f28019dc1c14cc2: MemRowSet was empty: no flush needed.
I20240503 08:12:49.084576 31601 raft_consensus.cc:482] T 68b1a80883614b1ca54fbd88bc769dff P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:49.084764 31601 raft_consensus.cc:504] T 68b1a80883614b1ca54fbd88bc769dff P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:49.085453 31601 leader_election.cc:288] T 68b1a80883614b1ca54fbd88bc769dff P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:12:49.085959 31161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68b1a80883614b1ca54fbd88bc769dff" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:12:49.085992 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68b1a80883614b1ca54fbd88bc769dff" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:12:49.086088 31832 raft_consensus.cc:482] T 50e4d2c569344dde915eec12f6f3d747 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:49.086220 31161 raft_consensus.cc:2454] T 68b1a80883614b1ca54fbd88bc769dff P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:12:49.086230 30895 raft_consensus.cc:2454] T 68b1a80883614b1ca54fbd88bc769dff P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:12:49.086234 31832 raft_consensus.cc:504] T 50e4d2c569344dde915eec12f6f3d747 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:49.086691 28326 leader_election.cc:302] T 68b1a80883614b1ca54fbd88bc769dff P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:49.086841 31832 leader_election.cc:288] T 50e4d2c569344dde915eec12f6f3d747 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:12:49.087177 31601 raft_consensus.cc:2781] T 68b1a80883614b1ca54fbd88bc769dff P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:49.087286 31601 raft_consensus.cc:482] T 68b1a80883614b1ca54fbd88bc769dff P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:49.087359 31601 raft_consensus.cc:3037] T 68b1a80883614b1ca54fbd88bc769dff P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:49.087517 31161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50e4d2c569344dde915eec12f6f3d747" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:12:49.087512 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50e4d2c569344dde915eec12f6f3d747" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:12:49.087714 31161 raft_consensus.cc:2454] T 50e4d2c569344dde915eec12f6f3d747 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 0.
I20240503 08:12:49.087744 30761 raft_consensus.cc:2454] T 50e4d2c569344dde915eec12f6f3d747 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 0.
I20240503 08:12:49.088172 30961 leader_election.cc:302] T 50e4d2c569344dde915eec12f6f3d747 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:49.088478 31832 raft_consensus.cc:2781] T 50e4d2c569344dde915eec12f6f3d747 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:49.088585 31832 raft_consensus.cc:482] T 50e4d2c569344dde915eec12f6f3d747 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:49.088688 31832 raft_consensus.cc:3037] T 50e4d2c569344dde915eec12f6f3d747 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:49.090173 31601 raft_consensus.cc:504] T 68b1a80883614b1ca54fbd88bc769dff P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:49.090799 31601 leader_election.cc:288] T 68b1a80883614b1ca54fbd88bc769dff P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Requested vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:12:49.090757 31832 raft_consensus.cc:504] T 50e4d2c569344dde915eec12f6f3d747 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:49.091324 31161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68b1a80883614b1ca54fbd88bc769dff" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:12:49.091320 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68b1a80883614b1ca54fbd88bc769dff" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:12:49.091532 30895 raft_consensus.cc:3037] T 68b1a80883614b1ca54fbd88bc769dff P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:49.091516 31161 raft_consensus.cc:3037] T 68b1a80883614b1ca54fbd88bc769dff P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:49.092509 31832 leader_election.cc:288] T 50e4d2c569344dde915eec12f6f3d747 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 election: Requested vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:12:49.094166 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50e4d2c569344dde915eec12f6f3d747" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:12:49.094321 31161 raft_consensus.cc:2454] T 68b1a80883614b1ca54fbd88bc769dff P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:12:49.094386 31159 raft_consensus.cc:3037] T 50e4d2c569344dde915eec12f6f3d747 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:49.094338 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50e4d2c569344dde915eec12f6f3d747" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:12:49.094525 30761 raft_consensus.cc:3037] T 50e4d2c569344dde915eec12f6f3d747 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:49.094738 28326 leader_election.cc:302] T 68b1a80883614b1ca54fbd88bc769dff P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:49.095085 31601 raft_consensus.cc:2781] T 68b1a80883614b1ca54fbd88bc769dff P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:49.095516 31601 raft_consensus.cc:686] T 68b1a80883614b1ca54fbd88bc769dff P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Becoming Leader. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Running, Role: LEADER
I20240503 08:12:49.095798 31601 consensus_queue.cc:237] T 68b1a80883614b1ca54fbd88bc769dff P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:49.097049 31159 raft_consensus.cc:2454] T 50e4d2c569344dde915eec12f6f3d747 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 1.
I20240503 08:12:49.097085 30761 raft_consensus.cc:2454] T 50e4d2c569344dde915eec12f6f3d747 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 1.
I20240503 08:12:49.097635 30959 leader_election.cc:302] T 50e4d2c569344dde915eec12f6f3d747 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:49.097932 31832 raft_consensus.cc:2781] T 50e4d2c569344dde915eec12f6f3d747 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:49.098057 31832 raft_consensus.cc:686] T 50e4d2c569344dde915eec12f6f3d747 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 LEADER]: Becoming Leader. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Running, Role: LEADER
I20240503 08:12:49.098271 31832 consensus_queue.cc:237] T 50e4d2c569344dde915eec12f6f3d747 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:49.098781 30895 raft_consensus.cc:2454] T 68b1a80883614b1ca54fbd88bc769dff P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:12:49.102684 27855 catalog_manager.cc:5484] T 50e4d2c569344dde915eec12f6f3d747 P ced21ca983b749d0b4d4f6a0e2426bf8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71). New cstate: current_term: 1 leader_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:12:49.103135 27862 catalog_manager.cc:5484] T 68b1a80883614b1ca54fbd88bc769dff P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68). New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:12:49.107705 27862 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:49.114398 31883 tablet.cc:1722] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P b00f244c69df401997ea99d492218a98: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:49.114815 31883 tablet.cc:1620] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P b00f244c69df401997ea99d492218a98: MemRowSet was empty: no flush needed.
I20240503 08:12:49.116221 31885 tablet.cc:1722] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P d09a11a03989421b9f28019dc1c14cc2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:49.116681 31885 tablet.cc:1620] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P d09a11a03989421b9f28019dc1c14cc2: MemRowSet was empty: no flush needed.
I20240503 08:12:49.117024 31798 tablet.cc:1722] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:49.117365 31798 tablet.cc:1620] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P ce1496a47d084365ad786cd09e472a2c: MemRowSet was empty: no flush needed.
I20240503 08:12:49.125782 27855 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=ab9ca11f0d204e0c95fe8293dc112cb2] alter complete (version 2)
I20240503 08:12:49.227561 27862 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:49.234772 31883 tablet.cc:1722] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P b00f244c69df401997ea99d492218a98: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:49.235478 31798 tablet.cc:1722] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:49.236287 31885 tablet.cc:1722] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P d09a11a03989421b9f28019dc1c14cc2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:49.243885 27859 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table_new_name [id=ab9ca11f0d204e0c95fe8293dc112cb2] alter complete (version 3)
I20240503 08:12:49.291107 31879 consensus_queue.cc:1035] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:49.296154 31879 consensus_queue.cc:1035] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:49.346836 27859 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:49.347102 27859 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:49.350011 27859 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet 6e5e541c506a4d7aaef7bf46b24ba0d7
I20240503 08:12:49.350831 31139 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e5e541c506a4d7aaef7bf46b24ba0d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:49 UTC) from {username='slave'} at 127.0.0.1:45814
I20240503 08:12:49.350929 30874 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e5e541c506a4d7aaef7bf46b24ba0d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:49 UTC) from {username='slave'} at 127.0.0.1:34426
I20240503 08:12:49.351051 28372 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e5e541c506a4d7aaef7bf46b24ba0d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:49 UTC) from {username='slave'} at 127.0.0.1:34786
I20240503 08:12:49.351619 31895 tablet_replica.cc:330] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P b00f244c69df401997ea99d492218a98: stopping tablet replica
I20240503 08:12:49.351943 31897 tablet_replica.cc:330] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P d09a11a03989421b9f28019dc1c14cc2: stopping tablet replica
I20240503 08:12:49.351897 31895 raft_consensus.cc:2229] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:49.351953 31896 tablet_replica.cc:330] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P ce1496a47d084365ad786cd09e472a2c: stopping tablet replica
I20240503 08:12:49.352182 31897 raft_consensus.cc:2229] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:49.352200 31896 raft_consensus.cc:2229] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:49.352386 31895 raft_consensus.cc:2258] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:49.352461 31897 raft_consensus.cc:2258] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:49.352562 31896 raft_consensus.cc:2258] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:49.353375 31895 ts_tablet_manager.cc:1875] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P b00f244c69df401997ea99d492218a98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:49.353391 31897 ts_tablet_manager.cc:1875] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P d09a11a03989421b9f28019dc1c14cc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:49.354019 31896 ts_tablet_manager.cc:1875] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P ce1496a47d084365ad786cd09e472a2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:49.360489 31895 ts_tablet_manager.cc:1888] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P b00f244c69df401997ea99d492218a98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:49.360689 31895 log.cc:1196] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P b00f244c69df401997ea99d492218a98: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal/wals/6e5e541c506a4d7aaef7bf46b24ba0d7
I20240503 08:12:49.361037 31896 ts_tablet_manager.cc:1888] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P ce1496a47d084365ad786cd09e472a2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:49.361152 31895 ts_tablet_manager.cc:1909] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P b00f244c69df401997ea99d492218a98: Deleting consensus metadata
I20240503 08:12:49.361196 31896 log.cc:1196] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P ce1496a47d084365ad786cd09e472a2c: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal/wals/6e5e541c506a4d7aaef7bf46b24ba0d7
I20240503 08:12:49.361616 31896 ts_tablet_manager.cc:1909] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P ce1496a47d084365ad786cd09e472a2c: Deleting consensus metadata
I20240503 08:12:49.361858 31897 ts_tablet_manager.cc:1888] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P d09a11a03989421b9f28019dc1c14cc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:49.362001 31897 log.cc:1196] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P d09a11a03989421b9f28019dc1c14cc2: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/wals/6e5e541c506a4d7aaef7bf46b24ba0d7
I20240503 08:12:49.362342 27846 catalog_manager.cc:4830] TS b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): tablet 6e5e541c506a4d7aaef7bf46b24ba0d7 (table rebalancer_dynamic_table_new_name [id=ab9ca11f0d204e0c95fe8293dc112cb2]) successfully deleted
I20240503 08:12:49.362424 31897 ts_tablet_manager.cc:1909] T 6e5e541c506a4d7aaef7bf46b24ba0d7 P d09a11a03989421b9f28019dc1c14cc2: Deleting consensus metadata
I20240503 08:12:49.362664 27846 catalog_manager.cc:4830] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): tablet 6e5e541c506a4d7aaef7bf46b24ba0d7 (table rebalancer_dynamic_table_new_name [id=ab9ca11f0d204e0c95fe8293dc112cb2]) successfully deleted
I20240503 08:12:49.363360 27849 catalog_manager.cc:4830] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): tablet 6e5e541c506a4d7aaef7bf46b24ba0d7 (table rebalancer_dynamic_table_new_name [id=ab9ca11f0d204e0c95fe8293dc112cb2]) successfully deleted
I20240503 08:12:49.452733 27859 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:49.460388 30874 tablet_service.cc:1467] Processing CreateTablet for tablet 1f0640e7130340358f58b573a66400c5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=78456f5bf3834d2b817113fd5513e6de]), partition=
I20240503 08:12:49.460675 31007 tablet_service.cc:1467] Processing CreateTablet for tablet 1f0640e7130340358f58b573a66400c5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=78456f5bf3834d2b817113fd5513e6de]), partition=
I20240503 08:12:49.460791 30874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f0640e7130340358f58b573a66400c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:49.461004 31007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f0640e7130340358f58b573a66400c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:49.465970 30740 tablet_service.cc:1467] Processing CreateTablet for tablet 1f0640e7130340358f58b573a66400c5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=78456f5bf3834d2b817113fd5513e6de]), partition=
I20240503 08:12:49.466037 31899 tablet_bootstrap.cc:492] T 1f0640e7130340358f58b573a66400c5 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:49.466346 30740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f0640e7130340358f58b573a66400c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:49.466687 31898 tablet_bootstrap.cc:492] T 1f0640e7130340358f58b573a66400c5 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:49.467516 31899 tablet_bootstrap.cc:654] T 1f0640e7130340358f58b573a66400c5 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:49.468433 31898 tablet_bootstrap.cc:654] T 1f0640e7130340358f58b573a66400c5 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:49.470153 31898 tablet_bootstrap.cc:492] T 1f0640e7130340358f58b573a66400c5 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:49.470340 31898 ts_tablet_manager.cc:1367] T 1f0640e7130340358f58b573a66400c5 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:12:49.471168 31898 raft_consensus.cc:348] T 1f0640e7130340358f58b573a66400c5 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:49.471664 31898 raft_consensus.cc:374] T 1f0640e7130340358f58b573a66400c5 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:49.471778 31898 raft_consensus.cc:729] T 1f0640e7130340358f58b573a66400c5 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:49.472144 31898 consensus_queue.cc:260] T 1f0640e7130340358f58b573a66400c5 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:49.474077 31898 ts_tablet_manager.cc:1398] T 1f0640e7130340358f58b573a66400c5 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:49.474704 31899 tablet_bootstrap.cc:492] T 1f0640e7130340358f58b573a66400c5 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:49.474839 31899 ts_tablet_manager.cc:1367] T 1f0640e7130340358f58b573a66400c5 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20240503 08:12:49.475508 31899 raft_consensus.cc:348] T 1f0640e7130340358f58b573a66400c5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:49.475735 31899 raft_consensus.cc:374] T 1f0640e7130340358f58b573a66400c5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:49.475827 31899 raft_consensus.cc:729] T 1f0640e7130340358f58b573a66400c5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:49.476033 31899 consensus_queue.cc:260] T 1f0640e7130340358f58b573a66400c5 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:49.477456 31899 ts_tablet_manager.cc:1398] T 1f0640e7130340358f58b573a66400c5 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 08:12:49.479935 31902 tablet_bootstrap.cc:492] T 1f0640e7130340358f58b573a66400c5 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:49.482250 31902 tablet_bootstrap.cc:654] T 1f0640e7130340358f58b573a66400c5 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:49.486287 31902 tablet_bootstrap.cc:492] T 1f0640e7130340358f58b573a66400c5 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:49.486440 31902 ts_tablet_manager.cc:1367] T 1f0640e7130340358f58b573a66400c5 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240503 08:12:49.487156 31902 raft_consensus.cc:348] T 1f0640e7130340358f58b573a66400c5 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:49.487371 31902 raft_consensus.cc:374] T 1f0640e7130340358f58b573a66400c5 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:49.487478 31902 raft_consensus.cc:729] T 1f0640e7130340358f58b573a66400c5 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:49.487679 31902 consensus_queue.cc:260] T 1f0640e7130340358f58b573a66400c5 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:49.488255 31902 ts_tablet_manager.cc:1398] T 1f0640e7130340358f58b573a66400c5 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:49.534381 31832 consensus_queue.cc:1035] T 50e4d2c569344dde915eec12f6f3d747 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:49.540954 31832 consensus_queue.cc:1035] T 50e4d2c569344dde915eec12f6f3d747 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:49.575219 31601 consensus_queue.cc:1035] T 68b1a80883614b1ca54fbd88bc769dff P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:49.579910 31800 consensus_queue.cc:1035] T 68b1a80883614b1ca54fbd88bc769dff P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:49.615177 27859 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_extra_table_20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:49.625121 30740 tablet_service.cc:1467] Processing CreateTablet for tablet 1b53494abdd349158c897af8379ac0b0 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=e37b245cb78d400384ac439a89fb8706]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:49.625342 31007 tablet_service.cc:1467] Processing CreateTablet for tablet 200387e6edce4ee9bf11331380923005 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=e37b245cb78d400384ac439a89fb8706]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:49.625607 30740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b53494abdd349158c897af8379ac0b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:49.625778 31007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 200387e6edce4ee9bf11331380923005. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:49.630134 30874 tablet_service.cc:1467] Processing CreateTablet for tablet 200387e6edce4ee9bf11331380923005 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=e37b245cb78d400384ac439a89fb8706]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:49.630124 31008 tablet_service.cc:1467] Processing CreateTablet for tablet 1b53494abdd349158c897af8379ac0b0 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=e37b245cb78d400384ac439a89fb8706]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:49.630468 30874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 200387e6edce4ee9bf11331380923005. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:49.630503 31008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b53494abdd349158c897af8379ac0b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:49.632026 31139 tablet_service.cc:1467] Processing CreateTablet for tablet 200387e6edce4ee9bf11331380923005 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=e37b245cb78d400384ac439a89fb8706]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:49.632512 31139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 200387e6edce4ee9bf11331380923005. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:49.633718 31899 tablet_bootstrap.cc:492] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:49.635252 31899 tablet_bootstrap.cc:654] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:49.636119 31898 tablet_bootstrap.cc:492] T 200387e6edce4ee9bf11331380923005 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:49.636948 31140 tablet_service.cc:1467] Processing CreateTablet for tablet 045b3ce4b1d44f94a802b41a12e2704d (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=e37b245cb78d400384ac439a89fb8706]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:49.637352 31140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 045b3ce4b1d44f94a802b41a12e2704d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:49.637593 31898 tablet_bootstrap.cc:654] T 200387e6edce4ee9bf11331380923005 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:49.638170 27983 tablet_service.cc:1467] Processing CreateTablet for tablet 1b53494abdd349158c897af8379ac0b0 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=e37b245cb78d400384ac439a89fb8706]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:49.638187 28243 tablet_service.cc:1467] Processing CreateTablet for tablet 045b3ce4b1d44f94a802b41a12e2704d (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=e37b245cb78d400384ac439a89fb8706]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:49.638588 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 045b3ce4b1d44f94a802b41a12e2704d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:49.638594 27983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b53494abdd349158c897af8379ac0b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:49.642590 30873 tablet_service.cc:1467] Processing CreateTablet for tablet 045b3ce4b1d44f94a802b41a12e2704d (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=e37b245cb78d400384ac439a89fb8706]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:49.643007 30873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 045b3ce4b1d44f94a802b41a12e2704d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:49.644052 31902 tablet_bootstrap.cc:492] T 1b53494abdd349158c897af8379ac0b0 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:49.645655 31902 tablet_bootstrap.cc:654] T 1b53494abdd349158c897af8379ac0b0 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:49.647944 31902 tablet_bootstrap.cc:492] T 1b53494abdd349158c897af8379ac0b0 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:49.648113 31902 ts_tablet_manager.cc:1367] T 1b53494abdd349158c897af8379ac0b0 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 08:12:49.648864 31902 raft_consensus.cc:348] T 1b53494abdd349158c897af8379ac0b0 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:49.649088 31902 raft_consensus.cc:374] T 1b53494abdd349158c897af8379ac0b0 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:49.649169 31902 raft_consensus.cc:729] T 1b53494abdd349158c897af8379ac0b0 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:49.649402 31902 consensus_queue.cc:260] T 1b53494abdd349158c897af8379ac0b0 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:49.650393 31902 ts_tablet_manager.cc:1398] T 1b53494abdd349158c897af8379ac0b0 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:49.662104 31915 tablet_bootstrap.cc:492] T 045b3ce4b1d44f94a802b41a12e2704d P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:49.664762 31898 tablet_bootstrap.cc:492] T 200387e6edce4ee9bf11331380923005 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:49.665007 31898 ts_tablet_manager.cc:1367] T 200387e6edce4ee9bf11331380923005 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.006s
I20240503 08:12:49.665218 31899 tablet_bootstrap.cc:492] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:49.665464 31899 ts_tablet_manager.cc:1367] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.032s user 0.011s sys 0.001s
I20240503 08:12:49.666347 31899 raft_consensus.cc:348] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:49.666707 31899 raft_consensus.cc:374] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:49.666817 31899 raft_consensus.cc:729] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:49.667129 31899 consensus_queue.cc:260] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:49.668182 31913 tablet_bootstrap.cc:492] T 200387e6edce4ee9bf11331380923005 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:49.666669 31898 raft_consensus.cc:348] T 200387e6edce4ee9bf11331380923005 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:49.668485 31898 raft_consensus.cc:374] T 200387e6edce4ee9bf11331380923005 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:49.668627 31898 raft_consensus.cc:729] T 200387e6edce4ee9bf11331380923005 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:49.670728 31899 ts_tablet_manager.cc:1398] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:12:49.671075 31899 tablet_bootstrap.cc:492] T 1b53494abdd349158c897af8379ac0b0 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:49.671973 31912 tablet_bootstrap.cc:492] T 1b53494abdd349158c897af8379ac0b0 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:49.672240 31851 raft_consensus.cc:482] T 1f0640e7130340358f58b573a66400c5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:49.672369 31851 raft_consensus.cc:504] T 1f0640e7130340358f58b573a66400c5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:49.668936 31898 consensus_queue.cc:260] T 200387e6edce4ee9bf11331380923005 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:49.674180 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f0640e7130340358f58b573a66400c5" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:12:49.674229 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f0640e7130340358f58b573a66400c5" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:12:49.674401 30895 raft_consensus.cc:2454] T 1f0640e7130340358f58b573a66400c5 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 0.
I20240503 08:12:49.674485 30762 raft_consensus.cc:2454] T 1f0640e7130340358f58b573a66400c5 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 0.
I20240503 08:12:49.675129 30959 leader_election.cc:302] T 1f0640e7130340358f58b573a66400c5 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:49.675705 31851 leader_election.cc:288] T 1f0640e7130340358f58b573a66400c5 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:12:49.675944 31912 tablet_bootstrap.cc:654] T 1b53494abdd349158c897af8379ac0b0 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:49.676112 31913 tablet_bootstrap.cc:654] T 200387e6edce4ee9bf11331380923005 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:49.677868 31832 raft_consensus.cc:2781] T 1f0640e7130340358f58b573a66400c5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:49.677978 31832 raft_consensus.cc:482] T 1f0640e7130340358f58b573a66400c5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:49.678129 31832 raft_consensus.cc:3037] T 1f0640e7130340358f58b573a66400c5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:49.680727 31832 raft_consensus.cc:504] T 1f0640e7130340358f58b573a66400c5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:49.681744 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f0640e7130340358f58b573a66400c5" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:12:49.681980 30762 raft_consensus.cc:3037] T 1f0640e7130340358f58b573a66400c5 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:49.684787 30762 raft_consensus.cc:2454] T 1f0640e7130340358f58b573a66400c5 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 1.
I20240503 08:12:49.685369 31899 tablet_bootstrap.cc:654] T 1b53494abdd349158c897af8379ac0b0 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:49.685518 30961 leader_election.cc:302] T 1f0640e7130340358f58b573a66400c5 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:49.685979 31851 raft_consensus.cc:2781] T 1f0640e7130340358f58b573a66400c5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:49.686127 31851 raft_consensus.cc:686] T 1f0640e7130340358f58b573a66400c5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 LEADER]: Becoming Leader. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Running, Role: LEADER
I20240503 08:12:49.686405 31851 consensus_queue.cc:237] T 1f0640e7130340358f58b573a66400c5 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:49.687639 31832 leader_election.cc:288] T 1f0640e7130340358f58b573a66400c5 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 election: Requested vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:12:49.690359 31898 ts_tablet_manager.cc:1398] T 200387e6edce4ee9bf11331380923005 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.024s user 0.001s sys 0.000s
I20240503 08:12:49.690724 31898 tablet_bootstrap.cc:492] T 045b3ce4b1d44f94a802b41a12e2704d P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:49.692365 31898 tablet_bootstrap.cc:654] T 045b3ce4b1d44f94a802b41a12e2704d P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:49.695806 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f0640e7130340358f58b573a66400c5" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:12:49.696028 30895 raft_consensus.cc:3037] T 1f0640e7130340358f58b573a66400c5 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:49.696822 31916 raft_consensus.cc:482] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:49.696986 31916 raft_consensus.cc:504] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:49.697824 31916 leader_election.cc:288] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:49.698930 30895 raft_consensus.cc:2454] T 1f0640e7130340358f58b573a66400c5 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 1.
I20240503 08:12:49.700524 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "200387e6edce4ee9bf11331380923005" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:12:49.700726 30895 raft_consensus.cc:2454] T 200387e6edce4ee9bf11331380923005 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 0.
I20240503 08:12:49.701566 31913 tablet_bootstrap.cc:492] T 200387e6edce4ee9bf11331380923005 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:49.701735 31913 ts_tablet_manager.cc:1367] T 200387e6edce4ee9bf11331380923005 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.007s
I20240503 08:12:49.701784 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "200387e6edce4ee9bf11331380923005" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:12:49.702136 30959 leader_election.cc:302] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:49.702598 31912 tablet_bootstrap.cc:492] T 1b53494abdd349158c897af8379ac0b0 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:49.702610 31913 raft_consensus.cc:348] T 200387e6edce4ee9bf11331380923005 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:49.702793 31912 ts_tablet_manager.cc:1367] T 1b53494abdd349158c897af8379ac0b0 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.000s
I20240503 08:12:49.702884 31913 raft_consensus.cc:374] T 200387e6edce4ee9bf11331380923005 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:49.703023 31913 raft_consensus.cc:729] T 200387e6edce4ee9bf11331380923005 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:49.703333 31913 consensus_queue.cc:260] T 200387e6edce4ee9bf11331380923005 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:49.703639 27859 catalog_manager.cc:5484] T 1f0640e7130340358f58b573a66400c5 P ced21ca983b749d0b4d4f6a0e2426bf8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71). New cstate: current_term: 1 leader_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:12:49.704684 31851 raft_consensus.cc:2781] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:49.704799 31851 raft_consensus.cc:482] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:49.704896 31851 raft_consensus.cc:3037] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:49.705000 31913 ts_tablet_manager.cc:1398] T 200387e6edce4ee9bf11331380923005 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:49.705266 31913 tablet_bootstrap.cc:492] T 045b3ce4b1d44f94a802b41a12e2704d P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:49.706761 31913 tablet_bootstrap.cc:654] T 045b3ce4b1d44f94a802b41a12e2704d P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:49.703706 31912 raft_consensus.cc:348] T 1b53494abdd349158c897af8379ac0b0 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:49.707619 31912 raft_consensus.cc:374] T 1b53494abdd349158c897af8379ac0b0 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:49.707726 31912 raft_consensus.cc:729] T 1b53494abdd349158c897af8379ac0b0 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:49.707876 31851 raft_consensus.cc:504] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:49.708060 31912 consensus_queue.cc:260] T 1b53494abdd349158c897af8379ac0b0 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:49.708768 31912 ts_tablet_manager.cc:1398] T 1b53494abdd349158c897af8379ac0b0 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:12:49.709827 31851 leader_election.cc:288] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 election: Requested vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
W20240503 08:12:49.710458 30959 leader_election.cc:341] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:49.711000 31915 tablet_bootstrap.cc:654] T 045b3ce4b1d44f94a802b41a12e2704d P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:49.711402 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "200387e6edce4ee9bf11331380923005" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:12:49.711688 30895 raft_consensus.cc:3037] T 200387e6edce4ee9bf11331380923005 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:49.712965 31899 tablet_bootstrap.cc:492] T 1b53494abdd349158c897af8379ac0b0 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:49.713163 31899 ts_tablet_manager.cc:1367] T 1b53494abdd349158c897af8379ac0b0 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.000s
I20240503 08:12:49.714152 31899 raft_consensus.cc:348] T 1b53494abdd349158c897af8379ac0b0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:49.714303 31899 raft_consensus.cc:374] T 1b53494abdd349158c897af8379ac0b0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:49.714449 31899 raft_consensus.cc:729] T 1b53494abdd349158c897af8379ac0b0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:49.714540 30895 raft_consensus.cc:2454] T 200387e6edce4ee9bf11331380923005 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 1.
I20240503 08:12:49.714720 31899 consensus_queue.cc:260] T 1b53494abdd349158c897af8379ac0b0 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:49.715242 30959 leader_election.cc:302] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:49.715781 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "200387e6edce4ee9bf11331380923005" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:12:49.715996 31159 raft_consensus.cc:3037] T 200387e6edce4ee9bf11331380923005 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:49.716483 31913 tablet_bootstrap.cc:492] T 045b3ce4b1d44f94a802b41a12e2704d P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:49.716646 31913 ts_tablet_manager.cc:1367] T 045b3ce4b1d44f94a802b41a12e2704d P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20240503 08:12:49.717473 31913 raft_consensus.cc:348] T 045b3ce4b1d44f94a802b41a12e2704d P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:49.717671 31913 raft_consensus.cc:374] T 045b3ce4b1d44f94a802b41a12e2704d P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:49.717754 31913 raft_consensus.cc:729] T 045b3ce4b1d44f94a802b41a12e2704d P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:49.718039 31913 consensus_queue.cc:260] T 045b3ce4b1d44f94a802b41a12e2704d P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:49.718590 31913 ts_tablet_manager.cc:1398] T 045b3ce4b1d44f94a802b41a12e2704d P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:49.718827 31851 raft_consensus.cc:2781] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:49.718753 31159 raft_consensus.cc:2454] T 200387e6edce4ee9bf11331380923005 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 1.
I20240503 08:12:49.719007 31851 raft_consensus.cc:686] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 LEADER]: Becoming Leader. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Running, Role: LEADER
I20240503 08:12:49.719247 31898 tablet_bootstrap.cc:492] T 045b3ce4b1d44f94a802b41a12e2704d P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:49.719403 31898 ts_tablet_manager.cc:1367] T 045b3ce4b1d44f94a802b41a12e2704d P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.001s
I20240503 08:12:49.719270 31851 consensus_queue.cc:237] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:49.720033 31898 raft_consensus.cc:348] T 045b3ce4b1d44f94a802b41a12e2704d P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:49.720178 31898 raft_consensus.cc:374] T 045b3ce4b1d44f94a802b41a12e2704d P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:49.720662 31898 raft_consensus.cc:729] T 045b3ce4b1d44f94a802b41a12e2704d P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:49.720907 31898 consensus_queue.cc:260] T 045b3ce4b1d44f94a802b41a12e2704d P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:49.715307 31899 ts_tablet_manager.cc:1398] T 1b53494abdd349158c897af8379ac0b0 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:49.724220 31915 tablet_bootstrap.cc:492] T 045b3ce4b1d44f94a802b41a12e2704d P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:49.724399 31915 ts_tablet_manager.cc:1367] T 045b3ce4b1d44f94a802b41a12e2704d P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.062s user 0.003s sys 0.000s
I20240503 08:12:49.725162 31915 raft_consensus.cc:348] T 045b3ce4b1d44f94a802b41a12e2704d P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:49.725390 31915 raft_consensus.cc:374] T 045b3ce4b1d44f94a802b41a12e2704d P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:49.725494 31915 raft_consensus.cc:729] T 045b3ce4b1d44f94a802b41a12e2704d P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:49.725703 31915 consensus_queue.cc:260] T 045b3ce4b1d44f94a802b41a12e2704d P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:49.726280 31915 ts_tablet_manager.cc:1398] T 045b3ce4b1d44f94a802b41a12e2704d P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:49.726822 31898 ts_tablet_manager.cc:1398] T 045b3ce4b1d44f94a802b41a12e2704d P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240503 08:12:49.728615 27862 catalog_manager.cc:5484] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71). New cstate: current_term: 1 leader_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:12:49.742125 31879 raft_consensus.cc:482] T 045b3ce4b1d44f94a802b41a12e2704d P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:49.742269 31879 raft_consensus.cc:504] T 045b3ce4b1d44f94a802b41a12e2704d P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:49.743002 31879 leader_election.cc:288] T 045b3ce4b1d44f94a802b41a12e2704d P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:12:49.743429 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "045b3ce4b1d44f94a802b41a12e2704d" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:12:49.743645 31159 raft_consensus.cc:2454] T 045b3ce4b1d44f94a802b41a12e2704d P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:12:49.743563 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "045b3ce4b1d44f94a802b41a12e2704d" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:12:49.743784 28262 raft_consensus.cc:2454] T 045b3ce4b1d44f94a802b41a12e2704d P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:12:49.743992 30827 leader_election.cc:302] T 045b3ce4b1d44f94a802b41a12e2704d P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:49.744272 31879 raft_consensus.cc:2781] T 045b3ce4b1d44f94a802b41a12e2704d P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:49.744388 31879 raft_consensus.cc:482] T 045b3ce4b1d44f94a802b41a12e2704d P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:49.744474 31879 raft_consensus.cc:3037] T 045b3ce4b1d44f94a802b41a12e2704d P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:49.746464 31879 raft_consensus.cc:504] T 045b3ce4b1d44f94a802b41a12e2704d P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:49.747010 31879 leader_election.cc:288] T 045b3ce4b1d44f94a802b41a12e2704d P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Requested vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:12:49.747457 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "045b3ce4b1d44f94a802b41a12e2704d" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:12:49.747463 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "045b3ce4b1d44f94a802b41a12e2704d" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:12:49.747618 28262 raft_consensus.cc:3037] T 045b3ce4b1d44f94a802b41a12e2704d P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:49.747635 31159 raft_consensus.cc:3037] T 045b3ce4b1d44f94a802b41a12e2704d P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:49.749423 31159 raft_consensus.cc:2454] T 045b3ce4b1d44f94a802b41a12e2704d P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:12:49.749439 28262 raft_consensus.cc:2454] T 045b3ce4b1d44f94a802b41a12e2704d P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:12:49.749821 30827 leader_election.cc:302] T 045b3ce4b1d44f94a802b41a12e2704d P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:49.750109 31879 raft_consensus.cc:2781] T 045b3ce4b1d44f94a802b41a12e2704d P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:49.750232 31879 raft_consensus.cc:686] T 045b3ce4b1d44f94a802b41a12e2704d P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Becoming Leader. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Running, Role: LEADER
I20240503 08:12:49.750504 31879 consensus_queue.cc:237] T 045b3ce4b1d44f94a802b41a12e2704d P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:49.753561 27862 catalog_manager.cc:5484] T 045b3ce4b1d44f94a802b41a12e2704d P ce1496a47d084365ad786cd09e472a2c reported cstate change: term changed from 0 to 1, leader changed from <none> to ce1496a47d084365ad786cd09e472a2c (127.20.230.70). New cstate: current_term: 1 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:12:49.768141 31916 raft_consensus.cc:482] T 1b53494abdd349158c897af8379ac0b0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:49.768353 31916 raft_consensus.cc:504] T 1b53494abdd349158c897af8379ac0b0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:49.768944 31916 leader_election.cc:288] T 1b53494abdd349158c897af8379ac0b0 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:12:49.769361 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b53494abdd349158c897af8379ac0b0" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:12:49.769388 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b53494abdd349158c897af8379ac0b0" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:12:49.769551 30762 raft_consensus.cc:2454] T 1b53494abdd349158c897af8379ac0b0 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 0.
I20240503 08:12:49.769600 28002 raft_consensus.cc:2454] T 1b53494abdd349158c897af8379ac0b0 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 0.
I20240503 08:12:49.769908 30961 leader_election.cc:302] T 1b53494abdd349158c897af8379ac0b0 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:49.770203 31916 raft_consensus.cc:2781] T 1b53494abdd349158c897af8379ac0b0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:49.770283 31916 raft_consensus.cc:482] T 1b53494abdd349158c897af8379ac0b0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:49.770359 31916 raft_consensus.cc:3037] T 1b53494abdd349158c897af8379ac0b0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:49.772506 31916 raft_consensus.cc:504] T 1b53494abdd349158c897af8379ac0b0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:49.773093 31916 leader_election.cc:288] T 1b53494abdd349158c897af8379ac0b0 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 election: Requested vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:12:49.773458 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b53494abdd349158c897af8379ac0b0" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:12:49.773487 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b53494abdd349158c897af8379ac0b0" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:12:49.773635 28002 raft_consensus.cc:3037] T 1b53494abdd349158c897af8379ac0b0 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:49.773656 30762 raft_consensus.cc:3037] T 1b53494abdd349158c897af8379ac0b0 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:49.775595 28002 raft_consensus.cc:2454] T 1b53494abdd349158c897af8379ac0b0 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 1.
I20240503 08:12:49.775692 30762 raft_consensus.cc:2454] T 1b53494abdd349158c897af8379ac0b0 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 1.
I20240503 08:12:49.776003 30962 leader_election.cc:302] T 1b53494abdd349158c897af8379ac0b0 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:49.776288 31916 raft_consensus.cc:2781] T 1b53494abdd349158c897af8379ac0b0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:49.776454 31916 raft_consensus.cc:686] T 1b53494abdd349158c897af8379ac0b0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 LEADER]: Becoming Leader. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Running, Role: LEADER
I20240503 08:12:49.776726 31916 consensus_queue.cc:237] T 1b53494abdd349158c897af8379ac0b0 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:49.779760 27862 catalog_manager.cc:5484] T 1b53494abdd349158c897af8379ac0b0 P ced21ca983b749d0b4d4f6a0e2426bf8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71). New cstate: current_term: 1 leader_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:12:49.867525 27862 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:49.873421 30762 raft_consensus.cc:1261] T 1f0640e7130340358f58b573a66400c5 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer ced21ca983b749d0b4d4f6a0e2426bf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:49.873471 30895 raft_consensus.cc:1261] T 1f0640e7130340358f58b573a66400c5 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer ced21ca983b749d0b4d4f6a0e2426bf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:49.874178 31916 consensus_queue.cc:1035] T 1f0640e7130340358f58b573a66400c5 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:49.874554 31851 consensus_queue.cc:1035] T 1f0640e7130340358f58b573a66400c5 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:49.880290 31930 tablet.cc:1722] T 1f0640e7130340358f58b573a66400c5 P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:49.880908 31930 tablet.cc:1620] T 1f0640e7130340358f58b573a66400c5 P ced21ca983b749d0b4d4f6a0e2426bf8: MemRowSet was empty: no flush needed.
I20240503 08:12:49.882316 31931 tablet.cc:1722] T 1f0640e7130340358f58b573a66400c5 P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:49.883002 31931 tablet.cc:1620] T 1f0640e7130340358f58b573a66400c5 P 6d1d7de19459435fbde1999e3dfd226b: MemRowSet was empty: no flush needed.
I20240503 08:12:49.882977 31932 tablet.cc:1722] T 1f0640e7130340358f58b573a66400c5 P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:49.883540 31932 tablet.cc:1620] T 1f0640e7130340358f58b573a66400c5 P ce1496a47d084365ad786cd09e472a2c: MemRowSet was empty: no flush needed.
I20240503 08:12:49.883795 27862 catalog_manager.cc:5576] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725) does not have the latest schema for tablet 1f0640e7130340358f58b573a66400c5 (table rebalancer_dynamic_table [id=78456f5bf3834d2b817113fd5513e6de]). Expected version 1 got 0
I20240503 08:12:49.885077 27859 catalog_manager.cc:5576] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747) does not have the latest schema for tablet 1f0640e7130340358f58b573a66400c5 (table rebalancer_dynamic_table [id=78456f5bf3834d2b817113fd5513e6de]). Expected version 1 got 0
I20240503 08:12:49.893040 27859 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=78456f5bf3834d2b817113fd5513e6de] alter complete (version 1)
I20240503 08:12:49.996685 27859 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:50.004434 31930 tablet.cc:1722] T 1f0640e7130340358f58b573a66400c5 P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:50.004951 31930 tablet.cc:1620] T 1f0640e7130340358f58b573a66400c5 P ced21ca983b749d0b4d4f6a0e2426bf8: MemRowSet was empty: no flush needed.
I20240503 08:12:50.005081 31931 tablet.cc:1722] T 1f0640e7130340358f58b573a66400c5 P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:50.005509 31931 tablet.cc:1620] T 1f0640e7130340358f58b573a66400c5 P 6d1d7de19459435fbde1999e3dfd226b: MemRowSet was empty: no flush needed.
I20240503 08:12:50.006605 31932 tablet.cc:1722] T 1f0640e7130340358f58b573a66400c5 P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:50.007869 31932 tablet.cc:1620] T 1f0640e7130340358f58b573a66400c5 P ce1496a47d084365ad786cd09e472a2c: MemRowSet was empty: no flush needed.
I20240503 08:12:50.013861 27859 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=78456f5bf3834d2b817113fd5513e6de] alter complete (version 2)
I20240503 08:12:50.117225 27862 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:50.124684 31930 tablet.cc:1722] T 1f0640e7130340358f58b573a66400c5 P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:50.125308 31931 tablet.cc:1722] T 1f0640e7130340358f58b573a66400c5 P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:50.125861 31932 tablet.cc:1722] T 1f0640e7130340358f58b573a66400c5 P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:50.133464 27859 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table_new_name [id=78456f5bf3834d2b817113fd5513e6de] alter complete (version 3)
I20240503 08:12:50.235869 27862 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:50.236076 27862 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:50.238996 27862 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet 1f0640e7130340358f58b573a66400c5
I20240503 08:12:50.239787 30873 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f0640e7130340358f58b573a66400c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:50 UTC) from {username='slave'} at 127.0.0.1:34426
I20240503 08:12:50.239899 30740 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f0640e7130340358f58b573a66400c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:50 UTC) from {username='slave'} at 127.0.0.1:46794
I20240503 08:12:50.240348 31008 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f0640e7130340358f58b573a66400c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:50 UTC) from {username='slave'} at 127.0.0.1:47020
I20240503 08:12:50.240445 31934 tablet_replica.cc:330] T 1f0640e7130340358f58b573a66400c5 P ce1496a47d084365ad786cd09e472a2c: stopping tablet replica
I20240503 08:12:50.240660 31934 raft_consensus.cc:2229] T 1f0640e7130340358f58b573a66400c5 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:50.240898 31935 tablet_replica.cc:330] T 1f0640e7130340358f58b573a66400c5 P 6d1d7de19459435fbde1999e3dfd226b: stopping tablet replica
I20240503 08:12:50.240950 31934 raft_consensus.cc:2258] T 1f0640e7130340358f58b573a66400c5 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:50.241204 31935 raft_consensus.cc:2229] T 1f0640e7130340358f58b573a66400c5 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:50.241457 31935 raft_consensus.cc:2258] T 1f0640e7130340358f58b573a66400c5 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:50.242108 31936 tablet_replica.cc:330] T 1f0640e7130340358f58b573a66400c5 P ced21ca983b749d0b4d4f6a0e2426bf8: stopping tablet replica
I20240503 08:12:50.242238 31934 ts_tablet_manager.cc:1875] T 1f0640e7130340358f58b573a66400c5 P ce1496a47d084365ad786cd09e472a2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:50.242357 31936 raft_consensus.cc:2229] T 1f0640e7130340358f58b573a66400c5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:50.242652 31935 ts_tablet_manager.cc:1875] T 1f0640e7130340358f58b573a66400c5 P 6d1d7de19459435fbde1999e3dfd226b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:50.242990 31936 raft_consensus.cc:2258] T 1f0640e7130340358f58b573a66400c5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:50.243188 31699 consensus_queue.cc:1035] T 045b3ce4b1d44f94a802b41a12e2704d P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:50.244639 31936 ts_tablet_manager.cc:1875] T 1f0640e7130340358f58b573a66400c5 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:50.249047 31935 ts_tablet_manager.cc:1888] T 1f0640e7130340358f58b573a66400c5 P 6d1d7de19459435fbde1999e3dfd226b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:50.249223 31935 log.cc:1196] T 1f0640e7130340358f58b573a66400c5 P 6d1d7de19459435fbde1999e3dfd226b: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal/wals/1f0640e7130340358f58b573a66400c5
I20240503 08:12:50.249758 31935 ts_tablet_manager.cc:1909] T 1f0640e7130340358f58b573a66400c5 P 6d1d7de19459435fbde1999e3dfd226b: Deleting consensus metadata
I20240503 08:12:50.250954 27848 catalog_manager.cc:4830] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): tablet 1f0640e7130340358f58b573a66400c5 (table rebalancer_dynamic_table_new_name [id=78456f5bf3834d2b817113fd5513e6de]) successfully deleted
I20240503 08:12:50.254626 31934 ts_tablet_manager.cc:1888] T 1f0640e7130340358f58b573a66400c5 P ce1496a47d084365ad786cd09e472a2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:50.254736 31934 log.cc:1196] T 1f0640e7130340358f58b573a66400c5 P ce1496a47d084365ad786cd09e472a2c: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal/wals/1f0640e7130340358f58b573a66400c5
I20240503 08:12:50.255079 31934 ts_tablet_manager.cc:1909] T 1f0640e7130340358f58b573a66400c5 P ce1496a47d084365ad786cd09e472a2c: Deleting consensus metadata
I20240503 08:12:50.255960 27846 catalog_manager.cc:4830] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): tablet 1f0640e7130340358f58b573a66400c5 (table rebalancer_dynamic_table_new_name [id=78456f5bf3834d2b817113fd5513e6de]) successfully deleted
I20240503 08:12:50.257400 31936 ts_tablet_manager.cc:1888] T 1f0640e7130340358f58b573a66400c5 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:50.257746 31936 log.cc:1196] T 1f0640e7130340358f58b573a66400c5 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal/wals/1f0640e7130340358f58b573a66400c5
I20240503 08:12:50.258035 31879 consensus_queue.cc:1035] T 045b3ce4b1d44f94a802b41a12e2704d P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:50.258311 31936 ts_tablet_manager.cc:1909] T 1f0640e7130340358f58b573a66400c5 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting consensus metadata
I20240503 08:12:50.259542 27847 catalog_manager.cc:4830] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): tablet 1f0640e7130340358f58b573a66400c5 (table rebalancer_dynamic_table_new_name [id=78456f5bf3834d2b817113fd5513e6de]) successfully deleted
I20240503 08:12:50.276065 31916 consensus_queue.cc:1035] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:50.280793 31851 consensus_queue.cc:1035] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:50.289063 31851 consensus_queue.cc:1035] T 1b53494abdd349158c897af8379ac0b0 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:50.293089 31851 consensus_queue.cc:1035] T 1b53494abdd349158c897af8379ac0b0 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:50.296941 27856 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_extra_table_22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:50.305429 28372 tablet_service.cc:1467] Processing CreateTablet for tablet c9055668b86e4dc5aa321a4d29beaf74 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=11f85d35fe5f401d990c665eefacf247]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:50.305899 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9055668b86e4dc5aa321a4d29beaf74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:50.306375 30740 tablet_service.cc:1467] Processing CreateTablet for tablet 486e9843fd784148b9b49dfe27996dff (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=11f85d35fe5f401d990c665eefacf247]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:50.306744 30740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 486e9843fd784148b9b49dfe27996dff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:50.311235 31946 tablet_bootstrap.cc:492] T c9055668b86e4dc5aa321a4d29beaf74 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:50.311304 31008 tablet_service.cc:1467] Processing CreateTablet for tablet c9055668b86e4dc5aa321a4d29beaf74 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=11f85d35fe5f401d990c665eefacf247]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:50.311342 31007 tablet_service.cc:1467] Processing CreateTablet for tablet 486e9843fd784148b9b49dfe27996dff (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=11f85d35fe5f401d990c665eefacf247]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:50.311806 31008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9055668b86e4dc5aa321a4d29beaf74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:50.311967 30741 tablet_service.cc:1467] Processing CreateTablet for tablet 41e26e13355d4e0cb56bb526415a3998 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=11f85d35fe5f401d990c665eefacf247]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:50.312053 31007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 486e9843fd784148b9b49dfe27996dff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:50.312383 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41e26e13355d4e0cb56bb526415a3998. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:50.313333 31946 tablet_bootstrap.cc:654] T c9055668b86e4dc5aa321a4d29beaf74 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:50.314772 30873 tablet_service.cc:1467] Processing CreateTablet for tablet c9055668b86e4dc5aa321a4d29beaf74 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=11f85d35fe5f401d990c665eefacf247]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:50.314855 30874 tablet_service.cc:1467] Processing CreateTablet for tablet 41e26e13355d4e0cb56bb526415a3998 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=11f85d35fe5f401d990c665eefacf247]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:50.315227 30873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9055668b86e4dc5aa321a4d29beaf74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:50.315716 30874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41e26e13355d4e0cb56bb526415a3998. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:50.317281 31139 tablet_service.cc:1467] Processing CreateTablet for tablet 486e9843fd784148b9b49dfe27996dff (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=11f85d35fe5f401d990c665eefacf247]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:50.317709 31139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 486e9843fd784148b9b49dfe27996dff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:50.322554 31948 tablet_bootstrap.cc:492] T 486e9843fd784148b9b49dfe27996dff P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:50.325364 31948 tablet_bootstrap.cc:654] T 486e9843fd784148b9b49dfe27996dff P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:50.327999 31946 tablet_bootstrap.cc:492] T c9055668b86e4dc5aa321a4d29beaf74 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:50.328215 31946 ts_tablet_manager.cc:1367] T c9055668b86e4dc5aa321a4d29beaf74 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.007s
I20240503 08:12:50.329475 31946 raft_consensus.cc:348] T c9055668b86e4dc5aa321a4d29beaf74 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:50.329874 31946 raft_consensus.cc:374] T c9055668b86e4dc5aa321a4d29beaf74 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:50.330022 31946 raft_consensus.cc:729] T c9055668b86e4dc5aa321a4d29beaf74 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:50.330314 31946 consensus_queue.cc:260] T c9055668b86e4dc5aa321a4d29beaf74 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:50.330813 31008 tablet_service.cc:1467] Processing CreateTablet for tablet 41e26e13355d4e0cb56bb526415a3998 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=11f85d35fe5f401d990c665eefacf247]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:50.331079 31946 ts_tablet_manager.cc:1398] T c9055668b86e4dc5aa321a4d29beaf74 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 08:12:50.331238 31008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41e26e13355d4e0cb56bb526415a3998. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:50.332620 31948 tablet_bootstrap.cc:492] T 486e9843fd784148b9b49dfe27996dff P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:50.332809 31948 ts_tablet_manager.cc:1367] T 486e9843fd784148b9b49dfe27996dff P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.005s
I20240503 08:12:50.333670 31948 raft_consensus.cc:348] T 486e9843fd784148b9b49dfe27996dff P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:50.334976 31948 raft_consensus.cc:374] T 486e9843fd784148b9b49dfe27996dff P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:50.335117 31948 raft_consensus.cc:729] T 486e9843fd784148b9b49dfe27996dff P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:50.335400 31948 consensus_queue.cc:260] T 486e9843fd784148b9b49dfe27996dff P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:50.336330 31948 ts_tablet_manager.cc:1398] T 486e9843fd784148b9b49dfe27996dff P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:12:50.336820 31948 tablet_bootstrap.cc:492] T c9055668b86e4dc5aa321a4d29beaf74 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:50.338742 31948 tablet_bootstrap.cc:654] T c9055668b86e4dc5aa321a4d29beaf74 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:50.342635 27859 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:50.354796 31140 tablet_service.cc:1467] Processing CreateTablet for tablet a35a492630ad43cbb29b5fd31be15d4d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2f3321c714e04731b4231a71ccb9f205]), partition=
I20240503 08:12:50.355247 31140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a35a492630ad43cbb29b5fd31be15d4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:50.355377 30741 tablet_service.cc:1467] Processing CreateTablet for tablet a35a492630ad43cbb29b5fd31be15d4d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2f3321c714e04731b4231a71ccb9f205]), partition=
I20240503 08:12:50.355448 31008 tablet_service.cc:1467] Processing CreateTablet for tablet a35a492630ad43cbb29b5fd31be15d4d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2f3321c714e04731b4231a71ccb9f205]), partition=
I20240503 08:12:50.355733 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a35a492630ad43cbb29b5fd31be15d4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:50.355751 31008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a35a492630ad43cbb29b5fd31be15d4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:50.367415 31953 tablet_bootstrap.cc:492] T 41e26e13355d4e0cb56bb526415a3998 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:50.370536 31953 tablet_bootstrap.cc:654] T 41e26e13355d4e0cb56bb526415a3998 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:50.371481 31951 tablet_bootstrap.cc:492] T 486e9843fd784148b9b49dfe27996dff P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:50.373201 31951 tablet_bootstrap.cc:654] T 486e9843fd784148b9b49dfe27996dff P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:50.376003 31950 tablet_bootstrap.cc:492] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:50.378491 31950 tablet_bootstrap.cc:654] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:50.381024 31948 tablet_bootstrap.cc:492] T c9055668b86e4dc5aa321a4d29beaf74 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:50.381263 31948 ts_tablet_manager.cc:1367] T c9055668b86e4dc5aa321a4d29beaf74 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.045s user 0.009s sys 0.026s
I20240503 08:12:50.381706 31951 tablet_bootstrap.cc:492] T 486e9843fd784148b9b49dfe27996dff P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:50.381867 31951 ts_tablet_manager.cc:1367] T 486e9843fd784148b9b49dfe27996dff P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.010s user 0.010s sys 0.000s
I20240503 08:12:50.382050 31953 tablet_bootstrap.cc:492] T 41e26e13355d4e0cb56bb526415a3998 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:50.382331 31953 ts_tablet_manager.cc:1367] T 41e26e13355d4e0cb56bb526415a3998 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240503 08:12:50.382695 31951 raft_consensus.cc:348] T 486e9843fd784148b9b49dfe27996dff P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:50.382712 31948 raft_consensus.cc:348] T c9055668b86e4dc5aa321a4d29beaf74 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:50.382997 31951 raft_consensus.cc:374] T 486e9843fd784148b9b49dfe27996dff P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:50.383098 31951 raft_consensus.cc:729] T 486e9843fd784148b9b49dfe27996dff P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:50.383123 31953 raft_consensus.cc:348] T 41e26e13355d4e0cb56bb526415a3998 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:50.383280 31948 raft_consensus.cc:374] T c9055668b86e4dc5aa321a4d29beaf74 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:50.383378 31948 raft_consensus.cc:729] T c9055668b86e4dc5aa321a4d29beaf74 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:50.383347 31951 consensus_queue.cc:260] T 486e9843fd784148b9b49dfe27996dff P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:50.383611 31948 consensus_queue.cc:260] T c9055668b86e4dc5aa321a4d29beaf74 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:50.384146 31948 ts_tablet_manager.cc:1398] T c9055668b86e4dc5aa321a4d29beaf74 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 08:12:50.384171 31953 raft_consensus.cc:374] T 41e26e13355d4e0cb56bb526415a3998 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:50.384261 31953 raft_consensus.cc:729] T 41e26e13355d4e0cb56bb526415a3998 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:50.384543 31948 tablet_bootstrap.cc:492] T 41e26e13355d4e0cb56bb526415a3998 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:50.384601 31953 consensus_queue.cc:260] T 41e26e13355d4e0cb56bb526415a3998 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:50.385738 31953 ts_tablet_manager.cc:1398] T 41e26e13355d4e0cb56bb526415a3998 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:12:50.386107 31948 tablet_bootstrap.cc:654] T 41e26e13355d4e0cb56bb526415a3998 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:50.386415 31953 tablet_bootstrap.cc:492] T c9055668b86e4dc5aa321a4d29beaf74 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:50.388321 31948 tablet_bootstrap.cc:492] T 41e26e13355d4e0cb56bb526415a3998 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:50.388458 31948 ts_tablet_manager.cc:1367] T 41e26e13355d4e0cb56bb526415a3998 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:50.389089 31951 ts_tablet_manager.cc:1398] T 486e9843fd784148b9b49dfe27996dff P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:12:50.389287 31948 raft_consensus.cc:348] T 41e26e13355d4e0cb56bb526415a3998 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:50.389489 31948 raft_consensus.cc:374] T 41e26e13355d4e0cb56bb526415a3998 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:50.389571 31948 raft_consensus.cc:729] T 41e26e13355d4e0cb56bb526415a3998 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:50.389581 31950 tablet_bootstrap.cc:492] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:50.389714 31950 ts_tablet_manager.cc:1367] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240503 08:12:50.389812 31948 consensus_queue.cc:260] T 41e26e13355d4e0cb56bb526415a3998 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:50.390410 31950 raft_consensus.cc:348] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:50.392155 31948 ts_tablet_manager.cc:1398] T 41e26e13355d4e0cb56bb526415a3998 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:50.392336 31953 tablet_bootstrap.cc:654] T c9055668b86e4dc5aa321a4d29beaf74 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:50.392426 31948 tablet_bootstrap.cc:492] T a35a492630ad43cbb29b5fd31be15d4d P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:50.393148 31950 raft_consensus.cc:374] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:50.393321 31950 raft_consensus.cc:729] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:50.393816 31948 tablet_bootstrap.cc:654] T a35a492630ad43cbb29b5fd31be15d4d P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:50.393741 31950 consensus_queue.cc:260] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:50.394012 31951 tablet_bootstrap.cc:492] T 41e26e13355d4e0cb56bb526415a3998 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:50.395539 31950 ts_tablet_manager.cc:1398] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240503 08:12:50.395833 31951 tablet_bootstrap.cc:654] T 41e26e13355d4e0cb56bb526415a3998 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:50.395897 31950 tablet_bootstrap.cc:492] T a35a492630ad43cbb29b5fd31be15d4d P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:50.397626 31950 tablet_bootstrap.cc:654] T a35a492630ad43cbb29b5fd31be15d4d P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:50.400331 31953 tablet_bootstrap.cc:492] T c9055668b86e4dc5aa321a4d29beaf74 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:50.400563 31953 ts_tablet_manager.cc:1367] T c9055668b86e4dc5aa321a4d29beaf74 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.003s
I20240503 08:12:50.401388 31953 raft_consensus.cc:348] T c9055668b86e4dc5aa321a4d29beaf74 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:50.401742 31948 tablet_bootstrap.cc:492] T a35a492630ad43cbb29b5fd31be15d4d P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:50.401895 31953 raft_consensus.cc:374] T c9055668b86e4dc5aa321a4d29beaf74 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:50.401916 31948 ts_tablet_manager.cc:1367] T a35a492630ad43cbb29b5fd31be15d4d P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20240503 08:12:50.402027 31953 raft_consensus.cc:729] T c9055668b86e4dc5aa321a4d29beaf74 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:50.402449 31953 consensus_queue.cc:260] T c9055668b86e4dc5aa321a4d29beaf74 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:50.402570 31948 raft_consensus.cc:348] T a35a492630ad43cbb29b5fd31be15d4d P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:50.402757 31948 raft_consensus.cc:374] T a35a492630ad43cbb29b5fd31be15d4d P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:50.402830 31948 raft_consensus.cc:729] T a35a492630ad43cbb29b5fd31be15d4d P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:50.403101 31951 tablet_bootstrap.cc:492] T 41e26e13355d4e0cb56bb526415a3998 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:50.403163 31953 ts_tablet_manager.cc:1398] T c9055668b86e4dc5aa321a4d29beaf74 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:50.403067 31948 consensus_queue.cc:260] T a35a492630ad43cbb29b5fd31be15d4d P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:50.403245 31951 ts_tablet_manager.cc:1367] T 41e26e13355d4e0cb56bb526415a3998 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240503 08:12:50.403661 31948 ts_tablet_manager.cc:1398] T a35a492630ad43cbb29b5fd31be15d4d P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:12:50.403970 31951 raft_consensus.cc:348] T 41e26e13355d4e0cb56bb526415a3998 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:50.404168 31951 raft_consensus.cc:374] T 41e26e13355d4e0cb56bb526415a3998 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:50.404232 31951 raft_consensus.cc:729] T 41e26e13355d4e0cb56bb526415a3998 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:50.404496 31951 consensus_queue.cc:260] T 41e26e13355d4e0cb56bb526415a3998 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:50.405962 31951 ts_tablet_manager.cc:1398] T 41e26e13355d4e0cb56bb526415a3998 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:50.406256 31951 tablet_bootstrap.cc:492] T a35a492630ad43cbb29b5fd31be15d4d P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:50.406678 31950 tablet_bootstrap.cc:492] T a35a492630ad43cbb29b5fd31be15d4d P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:50.407033 31950 ts_tablet_manager.cc:1367] T a35a492630ad43cbb29b5fd31be15d4d P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240503 08:12:50.407727 31951 tablet_bootstrap.cc:654] T a35a492630ad43cbb29b5fd31be15d4d P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:50.408485 31950 raft_consensus.cc:348] T a35a492630ad43cbb29b5fd31be15d4d P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:50.408763 31950 raft_consensus.cc:374] T a35a492630ad43cbb29b5fd31be15d4d P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:50.408883 31950 raft_consensus.cc:729] T a35a492630ad43cbb29b5fd31be15d4d P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:50.409139 31950 consensus_queue.cc:260] T a35a492630ad43cbb29b5fd31be15d4d P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:50.409605 31951 tablet_bootstrap.cc:492] T a35a492630ad43cbb29b5fd31be15d4d P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:50.409797 31951 ts_tablet_manager.cc:1367] T a35a492630ad43cbb29b5fd31be15d4d P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:12:50.410686 31950 ts_tablet_manager.cc:1398] T a35a492630ad43cbb29b5fd31be15d4d P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 08:12:50.410625 31951 raft_consensus.cc:348] T a35a492630ad43cbb29b5fd31be15d4d P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:50.410804 31951 raft_consensus.cc:374] T a35a492630ad43cbb29b5fd31be15d4d P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:50.410893 31951 raft_consensus.cc:729] T a35a492630ad43cbb29b5fd31be15d4d P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:50.411092 31951 consensus_queue.cc:260] T a35a492630ad43cbb29b5fd31be15d4d P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:50.411689 31951 ts_tablet_manager.cc:1398] T a35a492630ad43cbb29b5fd31be15d4d P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:50.461611 31849 raft_consensus.cc:482] T a35a492630ad43cbb29b5fd31be15d4d P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:50.461817 31849 raft_consensus.cc:504] T a35a492630ad43cbb29b5fd31be15d4d P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:50.462534 31849 leader_election.cc:288] T a35a492630ad43cbb29b5fd31be15d4d P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:50.462949 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a35a492630ad43cbb29b5fd31be15d4d" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:12:50.462994 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a35a492630ad43cbb29b5fd31be15d4d" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:50.463193 31027 raft_consensus.cc:2454] T a35a492630ad43cbb29b5fd31be15d4d P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:12:50.463184 31159 raft_consensus.cc:2454] T a35a492630ad43cbb29b5fd31be15d4d P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:12:50.463613 30695 leader_election.cc:302] T a35a492630ad43cbb29b5fd31be15d4d P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:50.463853 31849 raft_consensus.cc:2781] T a35a492630ad43cbb29b5fd31be15d4d P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:50.463963 31849 raft_consensus.cc:482] T a35a492630ad43cbb29b5fd31be15d4d P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:50.464071 31849 raft_consensus.cc:3037] T a35a492630ad43cbb29b5fd31be15d4d P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:50.466226 31849 raft_consensus.cc:504] T a35a492630ad43cbb29b5fd31be15d4d P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:50.466759 31849 leader_election.cc:288] T a35a492630ad43cbb29b5fd31be15d4d P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Requested vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:50.467159 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a35a492630ad43cbb29b5fd31be15d4d" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:12:50.467167 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a35a492630ad43cbb29b5fd31be15d4d" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:12:50.467361 31159 raft_consensus.cc:3037] T a35a492630ad43cbb29b5fd31be15d4d P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:50.467363 31027 raft_consensus.cc:3037] T a35a492630ad43cbb29b5fd31be15d4d P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:50.469990 31027 raft_consensus.cc:2454] T a35a492630ad43cbb29b5fd31be15d4d P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:12:50.469993 31159 raft_consensus.cc:2454] T a35a492630ad43cbb29b5fd31be15d4d P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:12:50.470360 30694 leader_election.cc:302] T a35a492630ad43cbb29b5fd31be15d4d P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, b00f244c69df401997ea99d492218a98, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:50.470628 31849 raft_consensus.cc:2781] T a35a492630ad43cbb29b5fd31be15d4d P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:50.470750 31849 raft_consensus.cc:686] T a35a492630ad43cbb29b5fd31be15d4d P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Becoming Leader. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Running, Role: LEADER
I20240503 08:12:50.471019 31879 raft_consensus.cc:482] T c9055668b86e4dc5aa321a4d29beaf74 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:50.471079 31849 consensus_queue.cc:237] T a35a492630ad43cbb29b5fd31be15d4d P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:50.471176 31879 raft_consensus.cc:504] T c9055668b86e4dc5aa321a4d29beaf74 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:50.471839 31879 leader_election.cc:288] T c9055668b86e4dc5aa321a4d29beaf74 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:50.472587 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9055668b86e4dc5aa321a4d29beaf74" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:50.472784 31027 raft_consensus.cc:2454] T c9055668b86e4dc5aa321a4d29beaf74 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:12:50.473229 30828 leader_election.cc:302] T c9055668b86e4dc5aa321a4d29beaf74 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:50.473505 31879 raft_consensus.cc:2781] T c9055668b86e4dc5aa321a4d29beaf74 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:50.473613 31879 raft_consensus.cc:482] T c9055668b86e4dc5aa321a4d29beaf74 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:50.473683 31879 raft_consensus.cc:3037] T c9055668b86e4dc5aa321a4d29beaf74 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:50.476416 31879 raft_consensus.cc:504] T c9055668b86e4dc5aa321a4d29beaf74 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:50.477028 31879 leader_election.cc:288] T c9055668b86e4dc5aa321a4d29beaf74 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Requested vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:50.478456 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9055668b86e4dc5aa321a4d29beaf74" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:12:50.478634 31027 raft_consensus.cc:3037] T c9055668b86e4dc5aa321a4d29beaf74 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:50.481395 31027 raft_consensus.cc:2454] T c9055668b86e4dc5aa321a4d29beaf74 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:12:50.482555 30828 leader_election.cc:302] T c9055668b86e4dc5aa321a4d29beaf74 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:50.483124 31699 raft_consensus.cc:2781] T c9055668b86e4dc5aa321a4d29beaf74 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:50.483235 31699 raft_consensus.cc:686] T c9055668b86e4dc5aa321a4d29beaf74 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Becoming Leader. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Running, Role: LEADER
I20240503 08:12:50.483511 31699 consensus_queue.cc:237] T c9055668b86e4dc5aa321a4d29beaf74 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:50.499135 27855 catalog_manager.cc:5484] T c9055668b86e4dc5aa321a4d29beaf74 P ce1496a47d084365ad786cd09e472a2c reported cstate change: term changed from 0 to 1, leader changed from <none> to ce1496a47d084365ad786cd09e472a2c (127.20.230.70). New cstate: current_term: 1 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:12:50.506515 27855 catalog_manager.cc:5484] T a35a492630ad43cbb29b5fd31be15d4d P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69). New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:12:50.514187 28392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9055668b86e4dc5aa321a4d29beaf74" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:12:50.514410 28392 raft_consensus.cc:3037] T c9055668b86e4dc5aa321a4d29beaf74 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:50.517055 28392 raft_consensus.cc:2454] T c9055668b86e4dc5aa321a4d29beaf74 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:12:50.514132 28393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9055668b86e4dc5aa321a4d29beaf74" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:50.517508 28393 raft_consensus.cc:2362] T c9055668b86e4dc5aa321a4d29beaf74 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1. Re-sending same reply.
I20240503 08:12:50.544728 31841 raft_consensus.cc:482] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:50.544956 31841 raft_consensus.cc:504] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:50.545693 31841 leader_election.cc:288] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:12:50.546202 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "486e9843fd784148b9b49dfe27996dff" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:12:50.546216 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "486e9843fd784148b9b49dfe27996dff" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:50.546466 30761 raft_consensus.cc:2454] T 486e9843fd784148b9b49dfe27996dff P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 0.
I20240503 08:12:50.546480 31027 raft_consensus.cc:2454] T 486e9843fd784148b9b49dfe27996dff P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 0.
I20240503 08:12:50.546990 31096 leader_election.cc:302] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, b00f244c69df401997ea99d492218a98; no voters:
I20240503 08:12:50.547607 31971 raft_consensus.cc:2781] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:50.547762 31971 raft_consensus.cc:482] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:50.548218 31971 raft_consensus.cc:3037] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:50.550907 31971 raft_consensus.cc:504] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:50.551498 31971 leader_election.cc:288] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 election: Requested vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:12:50.551936 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "486e9843fd784148b9b49dfe27996dff" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:12:50.552172 31027 raft_consensus.cc:3037] T 486e9843fd784148b9b49dfe27996dff P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:50.552307 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "486e9843fd784148b9b49dfe27996dff" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:12:50.552515 30761 raft_consensus.cc:3037] T 486e9843fd784148b9b49dfe27996dff P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:50.554724 31027 raft_consensus.cc:2454] T 486e9843fd784148b9b49dfe27996dff P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 1.
I20240503 08:12:50.555133 31095 leader_election.cc:302] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:50.555178 30761 raft_consensus.cc:2454] T 486e9843fd784148b9b49dfe27996dff P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 1.
I20240503 08:12:50.555372 31971 raft_consensus.cc:2781] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:50.555583 31971 raft_consensus.cc:686] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Becoming Leader. State: Replica: b00f244c69df401997ea99d492218a98, State: Running, Role: LEADER
I20240503 08:12:50.555874 31971 consensus_queue.cc:237] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:50.558981 27855 catalog_manager.cc:5484] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98 reported cstate change: term changed from 0 to 1, leader changed from <none> to b00f244c69df401997ea99d492218a98 (127.20.230.72). New cstate: current_term: 1 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:12:50.632328 27855 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:50.637790 31027 raft_consensus.cc:1261] T a35a492630ad43cbb29b5fd31be15d4d P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:50.638420 31963 consensus_queue.cc:1035] T a35a492630ad43cbb29b5fd31be15d4d P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:50.638670 31159 raft_consensus.cc:1261] T a35a492630ad43cbb29b5fd31be15d4d P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:50.641497 31963 consensus_queue.cc:1035] T a35a492630ad43cbb29b5fd31be15d4d P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:50.650789 31976 tablet.cc:1722] T a35a492630ad43cbb29b5fd31be15d4d P b00f244c69df401997ea99d492218a98: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:50.651414 31976 tablet.cc:1620] T a35a492630ad43cbb29b5fd31be15d4d P b00f244c69df401997ea99d492218a98: MemRowSet was empty: no flush needed.
I20240503 08:12:50.652021 31975 tablet.cc:1722] T a35a492630ad43cbb29b5fd31be15d4d P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:50.652525 27859 catalog_manager.cc:5576] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915) does not have the latest schema for tablet a35a492630ad43cbb29b5fd31be15d4d (table rebalancer_dynamic_table [id=2f3321c714e04731b4231a71ccb9f205]). Expected version 1 got 0
I20240503 08:12:50.652599 31975 tablet.cc:1620] T a35a492630ad43cbb29b5fd31be15d4d P 6d1d7de19459435fbde1999e3dfd226b: MemRowSet was empty: no flush needed.
I20240503 08:12:50.656780 27855 catalog_manager.cc:5576] TS b00f244c69df401997ea99d492218a98 (127.20.230.72:40893) does not have the latest schema for tablet a35a492630ad43cbb29b5fd31be15d4d (table rebalancer_dynamic_table [id=2f3321c714e04731b4231a71ccb9f205]). Expected version 1 got 0
I20240503 08:12:50.658967 31977 tablet.cc:1722] T a35a492630ad43cbb29b5fd31be15d4d P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:50.659588 31977 tablet.cc:1620] T a35a492630ad43cbb29b5fd31be15d4d P ced21ca983b749d0b4d4f6a0e2426bf8: MemRowSet was empty: no flush needed.
I20240503 08:12:50.660369 27855 catalog_manager.cc:5576] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915) does not have the latest schema for tablet a35a492630ad43cbb29b5fd31be15d4d (table rebalancer_dynamic_table [id=2f3321c714e04731b4231a71ccb9f205]). Expected version 1 got 0
I20240503 08:12:50.664793 27859 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=2f3321c714e04731b4231a71ccb9f205] alter complete (version 1)
I20240503 08:12:50.665867 31963 raft_consensus.cc:482] T 41e26e13355d4e0cb56bb526415a3998 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:50.666019 31963 raft_consensus.cc:504] T 41e26e13355d4e0cb56bb526415a3998 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:50.666764 31963 leader_election.cc:288] T 41e26e13355d4e0cb56bb526415a3998 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:50.667716 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41e26e13355d4e0cb56bb526415a3998" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:50.667750 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41e26e13355d4e0cb56bb526415a3998" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:12:50.667930 30895 raft_consensus.cc:2454] T 41e26e13355d4e0cb56bb526415a3998 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:12:50.667918 31027 raft_consensus.cc:2454] T 41e26e13355d4e0cb56bb526415a3998 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:12:50.668246 30695 leader_election.cc:302] T 41e26e13355d4e0cb56bb526415a3998 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:50.668474 31963 raft_consensus.cc:2781] T 41e26e13355d4e0cb56bb526415a3998 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:50.668576 31963 raft_consensus.cc:482] T 41e26e13355d4e0cb56bb526415a3998 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:50.668658 31963 raft_consensus.cc:3037] T 41e26e13355d4e0cb56bb526415a3998 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:50.670449 31963 raft_consensus.cc:504] T 41e26e13355d4e0cb56bb526415a3998 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:50.670967 31963 leader_election.cc:288] T 41e26e13355d4e0cb56bb526415a3998 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Requested vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:50.671356 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41e26e13355d4e0cb56bb526415a3998" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:12:50.671532 30895 raft_consensus.cc:3037] T 41e26e13355d4e0cb56bb526415a3998 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:50.673295 30895 raft_consensus.cc:2454] T 41e26e13355d4e0cb56bb526415a3998 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:12:50.673633 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41e26e13355d4e0cb56bb526415a3998" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:12:50.673662 30694 leader_election.cc:302] T 41e26e13355d4e0cb56bb526415a3998 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:50.673838 31027 raft_consensus.cc:3037] T 41e26e13355d4e0cb56bb526415a3998 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:50.673961 31963 raft_consensus.cc:2781] T 41e26e13355d4e0cb56bb526415a3998 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:50.674101 31963 raft_consensus.cc:686] T 41e26e13355d4e0cb56bb526415a3998 P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Becoming Leader. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Running, Role: LEADER
I20240503 08:12:50.674408 31963 consensus_queue.cc:237] T 41e26e13355d4e0cb56bb526415a3998 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:50.676143 31027 raft_consensus.cc:2454] T 41e26e13355d4e0cb56bb526415a3998 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:12:50.677578 27859 catalog_manager.cc:5484] T 41e26e13355d4e0cb56bb526415a3998 P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69). New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:12:50.770262 27859 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:50.777042 31975 tablet.cc:1722] T a35a492630ad43cbb29b5fd31be15d4d P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:50.777412 31976 tablet.cc:1722] T a35a492630ad43cbb29b5fd31be15d4d P b00f244c69df401997ea99d492218a98: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:50.777711 31975 tablet.cc:1620] T a35a492630ad43cbb29b5fd31be15d4d P 6d1d7de19459435fbde1999e3dfd226b: MemRowSet was empty: no flush needed.
I20240503 08:12:50.777854 31976 tablet.cc:1620] T a35a492630ad43cbb29b5fd31be15d4d P b00f244c69df401997ea99d492218a98: MemRowSet was empty: no flush needed.
I20240503 08:12:50.777878 31977 tablet.cc:1722] T a35a492630ad43cbb29b5fd31be15d4d P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:50.778313 31977 tablet.cc:1620] T a35a492630ad43cbb29b5fd31be15d4d P ced21ca983b749d0b4d4f6a0e2426bf8: MemRowSet was empty: no flush needed.
I20240503 08:12:50.787057 27859 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=2f3321c714e04731b4231a71ccb9f205] alter complete (version 2)
I20240503 08:12:50.888935 27862 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:50.896291 31975 tablet.cc:1722] T a35a492630ad43cbb29b5fd31be15d4d P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:50.896920 31976 tablet.cc:1722] T a35a492630ad43cbb29b5fd31be15d4d P b00f244c69df401997ea99d492218a98: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:50.897392 31977 tablet.cc:1722] T a35a492630ad43cbb29b5fd31be15d4d P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:50.905179 27862 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table_new_name [id=2f3321c714e04731b4231a71ccb9f205] alter complete (version 3)
I20240503 08:12:51.001035 31699 consensus_queue.cc:1035] T c9055668b86e4dc5aa321a4d29beaf74 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:51.007424 31971 consensus_queue.cc:1035] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:51.010799 27862 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:51.011338 27862 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:51.014823 27862 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet a35a492630ad43cbb29b5fd31be15d4d
I20240503 08:12:51.015666 31139 tablet_service.cc:1514] Processing DeleteTablet for tablet a35a492630ad43cbb29b5fd31be15d4d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:51 UTC) from {username='slave'} at 127.0.0.1:45814
I20240503 08:12:51.016448 30740 tablet_service.cc:1514] Processing DeleteTablet for tablet a35a492630ad43cbb29b5fd31be15d4d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:51 UTC) from {username='slave'} at 127.0.0.1:46794
I20240503 08:12:51.016649 31987 tablet_replica.cc:330] T a35a492630ad43cbb29b5fd31be15d4d P b00f244c69df401997ea99d492218a98: stopping tablet replica
I20240503 08:12:51.016860 31987 raft_consensus.cc:2229] T a35a492630ad43cbb29b5fd31be15d4d P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:51.017160 31987 raft_consensus.cc:2258] T a35a492630ad43cbb29b5fd31be15d4d P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:51.018211 31987 ts_tablet_manager.cc:1875] T a35a492630ad43cbb29b5fd31be15d4d P b00f244c69df401997ea99d492218a98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:51.020395 31699 consensus_queue.cc:1035] T c9055668b86e4dc5aa321a4d29beaf74 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:51.021322 31008 tablet_service.cc:1514] Processing DeleteTablet for tablet a35a492630ad43cbb29b5fd31be15d4d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:51 UTC) from {username='slave'} at 127.0.0.1:47020
I20240503 08:12:51.022363 31841 consensus_queue.cc:1035] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:51.049436 31987 ts_tablet_manager.cc:1888] T a35a492630ad43cbb29b5fd31be15d4d P b00f244c69df401997ea99d492218a98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:51.049705 31987 log.cc:1196] T a35a492630ad43cbb29b5fd31be15d4d P b00f244c69df401997ea99d492218a98: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal/wals/a35a492630ad43cbb29b5fd31be15d4d
I20240503 08:12:51.050283 31987 ts_tablet_manager.cc:1909] T a35a492630ad43cbb29b5fd31be15d4d P b00f244c69df401997ea99d492218a98: Deleting consensus metadata
I20240503 08:12:51.053164 27846 catalog_manager.cc:4830] TS b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): tablet a35a492630ad43cbb29b5fd31be15d4d (table rebalancer_dynamic_table_new_name [id=2f3321c714e04731b4231a71ccb9f205]) successfully deleted
I20240503 08:12:51.059913 31989 tablet_replica.cc:330] T a35a492630ad43cbb29b5fd31be15d4d P ced21ca983b749d0b4d4f6a0e2426bf8: stopping tablet replica
I20240503 08:12:51.060182 31989 raft_consensus.cc:2229] T a35a492630ad43cbb29b5fd31be15d4d P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:51.060431 31989 raft_consensus.cc:2258] T a35a492630ad43cbb29b5fd31be15d4d P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:51.061949 31988 tablet_replica.cc:330] T a35a492630ad43cbb29b5fd31be15d4d P 6d1d7de19459435fbde1999e3dfd226b: stopping tablet replica
I20240503 08:12:51.061993 31989 ts_tablet_manager.cc:1875] T a35a492630ad43cbb29b5fd31be15d4d P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:51.062279 31988 raft_consensus.cc:2229] T a35a492630ad43cbb29b5fd31be15d4d P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:51.062685 31988 raft_consensus.cc:2258] T a35a492630ad43cbb29b5fd31be15d4d P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:51.063587 31988 ts_tablet_manager.cc:1875] T a35a492630ad43cbb29b5fd31be15d4d P 6d1d7de19459435fbde1999e3dfd226b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 08:12:51.063766 28069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.68:46211, user_credentials={real_user=slave}} blocked reactor thread for 55205.6us
I20240503 08:12:51.068887 31989 ts_tablet_manager.cc:1888] T a35a492630ad43cbb29b5fd31be15d4d P ced21ca983b749d0b4d4f6a0e2426bf8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:51.069123 31989 log.cc:1196] T a35a492630ad43cbb29b5fd31be15d4d P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal/wals/a35a492630ad43cbb29b5fd31be15d4d
I20240503 08:12:51.069653 31989 ts_tablet_manager.cc:1909] T a35a492630ad43cbb29b5fd31be15d4d P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting consensus metadata
I20240503 08:12:51.070778 27847 catalog_manager.cc:4830] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): tablet a35a492630ad43cbb29b5fd31be15d4d (table rebalancer_dynamic_table_new_name [id=2f3321c714e04731b4231a71ccb9f205]) successfully deleted
I20240503 08:12:51.071520 31988 ts_tablet_manager.cc:1888] T a35a492630ad43cbb29b5fd31be15d4d P 6d1d7de19459435fbde1999e3dfd226b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:51.071643 31988 log.cc:1196] T a35a492630ad43cbb29b5fd31be15d4d P 6d1d7de19459435fbde1999e3dfd226b: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal/wals/a35a492630ad43cbb29b5fd31be15d4d
I20240503 08:12:51.072065 31988 ts_tablet_manager.cc:1909] T a35a492630ad43cbb29b5fd31be15d4d P 6d1d7de19459435fbde1999e3dfd226b: Deleting consensus metadata
I20240503 08:12:51.073067 27848 catalog_manager.cc:4830] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): tablet a35a492630ad43cbb29b5fd31be15d4d (table rebalancer_dynamic_table_new_name [id=2f3321c714e04731b4231a71ccb9f205]) successfully deleted
I20240503 08:12:51.124229 27855 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:51.131496 30740 tablet_service.cc:1467] Processing CreateTablet for tablet 46d7203013bf4d2ba96ea50ae720d3de (DEFAULT_TABLE table=rebalancer_dynamic_table [id=18e03813ffeb43e892ab2db9b92159ba]), partition=
I20240503 08:12:51.131637 31008 tablet_service.cc:1467] Processing CreateTablet for tablet 46d7203013bf4d2ba96ea50ae720d3de (DEFAULT_TABLE table=rebalancer_dynamic_table [id=18e03813ffeb43e892ab2db9b92159ba]), partition=
I20240503 08:12:51.131886 30740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d7203013bf4d2ba96ea50ae720d3de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:51.131964 31008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d7203013bf4d2ba96ea50ae720d3de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:51.134667 28243 tablet_service.cc:1467] Processing CreateTablet for tablet 46d7203013bf4d2ba96ea50ae720d3de (DEFAULT_TABLE table=rebalancer_dynamic_table [id=18e03813ffeb43e892ab2db9b92159ba]), partition=
I20240503 08:12:51.134990 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d7203013bf4d2ba96ea50ae720d3de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:51.136680 31993 tablet_bootstrap.cc:492] T 46d7203013bf4d2ba96ea50ae720d3de P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:51.137125 31994 tablet_bootstrap.cc:492] T 46d7203013bf4d2ba96ea50ae720d3de P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:51.138334 31993 tablet_bootstrap.cc:654] T 46d7203013bf4d2ba96ea50ae720d3de P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:51.138681 31994 tablet_bootstrap.cc:654] T 46d7203013bf4d2ba96ea50ae720d3de P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:51.140194 31993 tablet_bootstrap.cc:492] T 46d7203013bf4d2ba96ea50ae720d3de P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:51.140385 31993 ts_tablet_manager.cc:1367] T 46d7203013bf4d2ba96ea50ae720d3de P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:12:51.141181 31993 raft_consensus.cc:348] T 46d7203013bf4d2ba96ea50ae720d3de P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:51.141469 31993 raft_consensus.cc:374] T 46d7203013bf4d2ba96ea50ae720d3de P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:51.141587 31993 raft_consensus.cc:729] T 46d7203013bf4d2ba96ea50ae720d3de P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:51.141913 31993 consensus_queue.cc:260] T 46d7203013bf4d2ba96ea50ae720d3de P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:51.142189 31997 tablet_bootstrap.cc:492] T 46d7203013bf4d2ba96ea50ae720d3de P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:51.143579 31994 tablet_bootstrap.cc:492] T 46d7203013bf4d2ba96ea50ae720d3de P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:51.143678 31993 ts_tablet_manager.cc:1398] T 46d7203013bf4d2ba96ea50ae720d3de P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:51.143997 31997 tablet_bootstrap.cc:654] T 46d7203013bf4d2ba96ea50ae720d3de P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:51.144587 31994 ts_tablet_manager.cc:1367] T 46d7203013bf4d2ba96ea50ae720d3de P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240503 08:12:51.145197 31994 raft_consensus.cc:348] T 46d7203013bf4d2ba96ea50ae720d3de P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:51.145421 31994 raft_consensus.cc:374] T 46d7203013bf4d2ba96ea50ae720d3de P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:51.145555 31994 raft_consensus.cc:729] T 46d7203013bf4d2ba96ea50ae720d3de P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:51.145823 31994 consensus_queue.cc:260] T 46d7203013bf4d2ba96ea50ae720d3de P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:51.146411 31994 ts_tablet_manager.cc:1398] T 46d7203013bf4d2ba96ea50ae720d3de P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:51.147204 31997 tablet_bootstrap.cc:492] T 46d7203013bf4d2ba96ea50ae720d3de P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:51.147382 31997 ts_tablet_manager.cc:1367] T 46d7203013bf4d2ba96ea50ae720d3de P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 08:12:51.148424 31997 raft_consensus.cc:348] T 46d7203013bf4d2ba96ea50ae720d3de P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:51.148602 31997 raft_consensus.cc:374] T 46d7203013bf4d2ba96ea50ae720d3de P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:51.148723 31997 raft_consensus.cc:729] T 46d7203013bf4d2ba96ea50ae720d3de P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:51.148988 31997 consensus_queue.cc:260] T 46d7203013bf4d2ba96ea50ae720d3de P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:51.149839 31997 ts_tablet_manager.cc:1398] T 46d7203013bf4d2ba96ea50ae720d3de P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:51.167742 31832 raft_consensus.cc:482] T 46d7203013bf4d2ba96ea50ae720d3de P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:51.167899 31832 raft_consensus.cc:504] T 46d7203013bf4d2ba96ea50ae720d3de P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:51.168422 31832 leader_election.cc:288] T 46d7203013bf4d2ba96ea50ae720d3de P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:12:51.168823 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d7203013bf4d2ba96ea50ae720d3de" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:12:51.168864 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d7203013bf4d2ba96ea50ae720d3de" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:12:51.168998 30762 raft_consensus.cc:2454] T 46d7203013bf4d2ba96ea50ae720d3de P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 0.
I20240503 08:12:51.169081 28262 raft_consensus.cc:2454] T 46d7203013bf4d2ba96ea50ae720d3de P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 0.
I20240503 08:12:51.169425 30961 leader_election.cc:302] T 46d7203013bf4d2ba96ea50ae720d3de P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:51.169708 31832 raft_consensus.cc:2781] T 46d7203013bf4d2ba96ea50ae720d3de P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:51.169827 31832 raft_consensus.cc:482] T 46d7203013bf4d2ba96ea50ae720d3de P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:51.169929 31832 raft_consensus.cc:3037] T 46d7203013bf4d2ba96ea50ae720d3de P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:51.172099 31832 raft_consensus.cc:504] T 46d7203013bf4d2ba96ea50ae720d3de P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:51.173029 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d7203013bf4d2ba96ea50ae720d3de" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:12:51.173223 30762 raft_consensus.cc:3037] T 46d7203013bf4d2ba96ea50ae720d3de P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:51.173228 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d7203013bf4d2ba96ea50ae720d3de" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:12:51.173424 28262 raft_consensus.cc:3037] T 46d7203013bf4d2ba96ea50ae720d3de P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:51.175998 30762 raft_consensus.cc:2454] T 46d7203013bf4d2ba96ea50ae720d3de P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 1.
I20240503 08:12:51.175982 28262 raft_consensus.cc:2454] T 46d7203013bf4d2ba96ea50ae720d3de P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 1.
I20240503 08:12:51.172813 31832 leader_election.cc:288] T 46d7203013bf4d2ba96ea50ae720d3de P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 election: Requested vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:12:51.176354 30961 leader_election.cc:302] T 46d7203013bf4d2ba96ea50ae720d3de P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:51.176627 31832 raft_consensus.cc:2781] T 46d7203013bf4d2ba96ea50ae720d3de P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:51.177994 31832 raft_consensus.cc:686] T 46d7203013bf4d2ba96ea50ae720d3de P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 LEADER]: Becoming Leader. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Running, Role: LEADER
I20240503 08:12:51.178300 31832 consensus_queue.cc:237] T 46d7203013bf4d2ba96ea50ae720d3de P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:51.181041 27862 catalog_manager.cc:5484] T 46d7203013bf4d2ba96ea50ae720d3de P ced21ca983b749d0b4d4f6a0e2426bf8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71). New cstate: current_term: 1 leader_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:12:51.202531 27862 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_extra_table_24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:51.210568 31849 consensus_queue.cc:1035] T 41e26e13355d4e0cb56bb526415a3998 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:51.211740 28111 tablet_service.cc:1467] Processing CreateTablet for tablet d0657f27fd474a41b0c33db55989daa3 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=3cd9bb4078874a2e8da7dea4f43db999]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:51.212339 28111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0657f27fd474a41b0c33db55989daa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:51.213917 31139 tablet_service.cc:1467] Processing CreateTablet for tablet d0657f27fd474a41b0c33db55989daa3 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=3cd9bb4078874a2e8da7dea4f43db999]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:51.214342 31139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0657f27fd474a41b0c33db55989daa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:51.215132 31008 tablet_service.cc:1467] Processing CreateTablet for tablet 2c8df43c895d49cfa758eb04382261d4 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=3cd9bb4078874a2e8da7dea4f43db999]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:51.215564 31008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c8df43c895d49cfa758eb04382261d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:51.218519 30873 tablet_service.cc:1467] Processing CreateTablet for tablet e3e58f895edb4095ba3d77653c9197af (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=3cd9bb4078874a2e8da7dea4f43db999]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:51.218963 30873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3e58f895edb4095ba3d77653c9197af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:51.219242 31140 tablet_service.cc:1467] Processing CreateTablet for tablet e3e58f895edb4095ba3d77653c9197af (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=3cd9bb4078874a2e8da7dea4f43db999]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:51.219616 31140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3e58f895edb4095ba3d77653c9197af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:51.221578 31994 tablet_bootstrap.cc:492] T 2c8df43c895d49cfa758eb04382261d4 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:51.218529 30874 tablet_service.cc:1467] Processing CreateTablet for tablet d0657f27fd474a41b0c33db55989daa3 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=3cd9bb4078874a2e8da7dea4f43db999]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:51.222080 31141 tablet_service.cc:1467] Processing CreateTablet for tablet 2c8df43c895d49cfa758eb04382261d4 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=3cd9bb4078874a2e8da7dea4f43db999]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:51.222187 30874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0657f27fd474a41b0c33db55989daa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:51.222429 31141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c8df43c895d49cfa758eb04382261d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:51.223320 31994 tablet_bootstrap.cc:654] T 2c8df43c895d49cfa758eb04382261d4 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:51.227769 28372 tablet_service.cc:1467] Processing CreateTablet for tablet 2c8df43c895d49cfa758eb04382261d4 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=3cd9bb4078874a2e8da7dea4f43db999]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:51.228207 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c8df43c895d49cfa758eb04382261d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:51.228632 30740 tablet_service.cc:1467] Processing CreateTablet for tablet e3e58f895edb4095ba3d77653c9197af (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=3cd9bb4078874a2e8da7dea4f43db999]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:51.229028 30740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3e58f895edb4095ba3d77653c9197af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:51.232050 31994 tablet_bootstrap.cc:492] T 2c8df43c895d49cfa758eb04382261d4 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:51.233146 31994 ts_tablet_manager.cc:1367] T 2c8df43c895d49cfa758eb04382261d4 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.002s
I20240503 08:12:51.234120 31994 raft_consensus.cc:348] T 2c8df43c895d49cfa758eb04382261d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:51.236114 31994 raft_consensus.cc:374] T 2c8df43c895d49cfa758eb04382261d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:51.236289 31994 raft_consensus.cc:729] T 2c8df43c895d49cfa758eb04382261d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:51.236626 31994 consensus_queue.cc:260] T 2c8df43c895d49cfa758eb04382261d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:51.238116 31993 tablet_bootstrap.cc:492] T e3e58f895edb4095ba3d77653c9197af P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:51.240010 31993 tablet_bootstrap.cc:654] T e3e58f895edb4095ba3d77653c9197af P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:51.242677 31994 ts_tablet_manager.cc:1398] T 2c8df43c895d49cfa758eb04382261d4 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 08:12:51.245388 31969 consensus_queue.cc:1035] T 41e26e13355d4e0cb56bb526415a3998 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:51.245645 32005 tablet_bootstrap.cc:492] T e3e58f895edb4095ba3d77653c9197af P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:51.247802 32005 tablet_bootstrap.cc:654] T e3e58f895edb4095ba3d77653c9197af P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:51.254743 32006 tablet_bootstrap.cc:492] T 2c8df43c895d49cfa758eb04382261d4 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:51.257010 32006 tablet_bootstrap.cc:654] T 2c8df43c895d49cfa758eb04382261d4 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:51.258658 32002 tablet_bootstrap.cc:492] T d0657f27fd474a41b0c33db55989daa3 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:51.260349 32005 tablet_bootstrap.cc:492] T e3e58f895edb4095ba3d77653c9197af P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:51.260555 32005 ts_tablet_manager.cc:1367] T e3e58f895edb4095ba3d77653c9197af P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.007s
I20240503 08:12:51.261391 32005 raft_consensus.cc:348] T e3e58f895edb4095ba3d77653c9197af P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:51.261547 32002 tablet_bootstrap.cc:654] T d0657f27fd474a41b0c33db55989daa3 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:51.261603 32005 raft_consensus.cc:374] T e3e58f895edb4095ba3d77653c9197af P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:51.261756 32005 raft_consensus.cc:729] T e3e58f895edb4095ba3d77653c9197af P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:51.262137 32005 consensus_queue.cc:260] T e3e58f895edb4095ba3d77653c9197af P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:51.262715 32005 ts_tablet_manager.cc:1398] T e3e58f895edb4095ba3d77653c9197af P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:51.263119 32005 tablet_bootstrap.cc:492] T d0657f27fd474a41b0c33db55989daa3 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:51.263250 31992 raft_consensus.cc:482] T e3e58f895edb4095ba3d77653c9197af P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:51.263698 31992 raft_consensus.cc:504] T e3e58f895edb4095ba3d77653c9197af P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:51.263829 32006 tablet_bootstrap.cc:492] T 2c8df43c895d49cfa758eb04382261d4 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:51.263950 32006 ts_tablet_manager.cc:1367] T 2c8df43c895d49cfa758eb04382261d4 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.000s
I20240503 08:12:51.264433 31992 leader_election.cc:288] T e3e58f895edb4095ba3d77653c9197af P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:51.264829 32006 raft_consensus.cc:348] T 2c8df43c895d49cfa758eb04382261d4 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:51.265084 32006 raft_consensus.cc:374] T 2c8df43c895d49cfa758eb04382261d4 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:51.265053 31161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3e58f895edb4095ba3d77653c9197af" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:12:51.265363 32005 tablet_bootstrap.cc:654] T d0657f27fd474a41b0c33db55989daa3 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
W20240503 08:12:51.265627 30827 leader_election.cc:341] T e3e58f895edb4095ba3d77653c9197af P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:51.265734 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3e58f895edb4095ba3d77653c9197af" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
W20240503 08:12:51.266544 30829 leader_election.cc:341] T e3e58f895edb4095ba3d77653c9197af P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:51.266670 30829 leader_election.cc:302] T e3e58f895edb4095ba3d77653c9197af P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c; no voters: 6d1d7de19459435fbde1999e3dfd226b, b00f244c69df401997ea99d492218a98
I20240503 08:12:51.267055 31992 raft_consensus.cc:2726] T e3e58f895edb4095ba3d77653c9197af P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:12:51.268870 32006 raft_consensus.cc:729] T 2c8df43c895d49cfa758eb04382261d4 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:51.269315 32006 consensus_queue.cc:260] T 2c8df43c895d49cfa758eb04382261d4 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:51.270541 31993 tablet_bootstrap.cc:492] T e3e58f895edb4095ba3d77653c9197af P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:51.270754 31993 ts_tablet_manager.cc:1367] T e3e58f895edb4095ba3d77653c9197af P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.005s
I20240503 08:12:51.271023 32006 ts_tablet_manager.cc:1398] T 2c8df43c895d49cfa758eb04382261d4 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:12:51.271279 32006 tablet_bootstrap.cc:492] T d0657f27fd474a41b0c33db55989daa3 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:51.272778 31993 raft_consensus.cc:348] T e3e58f895edb4095ba3d77653c9197af P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:51.273084 31993 raft_consensus.cc:374] T e3e58f895edb4095ba3d77653c9197af P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:51.273192 31993 raft_consensus.cc:729] T e3e58f895edb4095ba3d77653c9197af P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:51.273511 31993 consensus_queue.cc:260] T e3e58f895edb4095ba3d77653c9197af P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:51.273878 32006 tablet_bootstrap.cc:654] T d0657f27fd474a41b0c33db55989daa3 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:51.276238 32006 tablet_bootstrap.cc:492] T d0657f27fd474a41b0c33db55989daa3 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:51.276450 32006 ts_tablet_manager.cc:1367] T d0657f27fd474a41b0c33db55989daa3 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240503 08:12:51.277678 31993 ts_tablet_manager.cc:1398] T e3e58f895edb4095ba3d77653c9197af P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240503 08:12:51.277688 32006 raft_consensus.cc:348] T d0657f27fd474a41b0c33db55989daa3 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:51.278029 32006 raft_consensus.cc:374] T d0657f27fd474a41b0c33db55989daa3 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:51.278101 32006 raft_consensus.cc:729] T d0657f27fd474a41b0c33db55989daa3 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:51.278326 32006 consensus_queue.cc:260] T d0657f27fd474a41b0c33db55989daa3 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:51.278914 32006 ts_tablet_manager.cc:1398] T d0657f27fd474a41b0c33db55989daa3 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:12:51.279273 32006 tablet_bootstrap.cc:492] T e3e58f895edb4095ba3d77653c9197af P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:51.280800 32006 tablet_bootstrap.cc:654] T e3e58f895edb4095ba3d77653c9197af P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:51.286242 32007 tablet_bootstrap.cc:492] T 2c8df43c895d49cfa758eb04382261d4 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:51.287773 32006 tablet_bootstrap.cc:492] T e3e58f895edb4095ba3d77653c9197af P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:51.287909 32006 ts_tablet_manager.cc:1367] T e3e58f895edb4095ba3d77653c9197af P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.001s
I20240503 08:12:51.288020 32007 tablet_bootstrap.cc:654] T 2c8df43c895d49cfa758eb04382261d4 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:51.289798 32006 raft_consensus.cc:348] T e3e58f895edb4095ba3d77653c9197af P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:51.289965 32006 raft_consensus.cc:374] T e3e58f895edb4095ba3d77653c9197af P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:51.290565 32006 raft_consensus.cc:729] T e3e58f895edb4095ba3d77653c9197af P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:51.290980 32006 consensus_queue.cc:260] T e3e58f895edb4095ba3d77653c9197af P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:51.291630 32005 tablet_bootstrap.cc:492] T d0657f27fd474a41b0c33db55989daa3 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:51.291854 32005 ts_tablet_manager.cc:1367] T d0657f27fd474a41b0c33db55989daa3 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.029s user 0.007s sys 0.015s
I20240503 08:12:51.292815 32006 ts_tablet_manager.cc:1398] T e3e58f895edb4095ba3d77653c9197af P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 08:12:51.293053 32005 raft_consensus.cc:348] T d0657f27fd474a41b0c33db55989daa3 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:51.293272 32005 raft_consensus.cc:374] T d0657f27fd474a41b0c33db55989daa3 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:51.293359 32005 raft_consensus.cc:729] T d0657f27fd474a41b0c33db55989daa3 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:51.293613 32005 consensus_queue.cc:260] T d0657f27fd474a41b0c33db55989daa3 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:51.294274 27859 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:51.295164 32002 tablet_bootstrap.cc:492] T d0657f27fd474a41b0c33db55989daa3 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:51.295416 32002 ts_tablet_manager.cc:1367] T d0657f27fd474a41b0c33db55989daa3 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.005s
I20240503 08:12:51.296228 32002 raft_consensus.cc:348] T d0657f27fd474a41b0c33db55989daa3 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:51.296432 32007 tablet_bootstrap.cc:492] T 2c8df43c895d49cfa758eb04382261d4 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:51.296497 32002 raft_consensus.cc:374] T d0657f27fd474a41b0c33db55989daa3 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:51.296640 32007 ts_tablet_manager.cc:1367] T 2c8df43c895d49cfa758eb04382261d4 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240503 08:12:51.296645 32002 raft_consensus.cc:729] T d0657f27fd474a41b0c33db55989daa3 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:51.296851 32002 consensus_queue.cc:260] T d0657f27fd474a41b0c33db55989daa3 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:51.297219 32005 ts_tablet_manager.cc:1398] T d0657f27fd474a41b0c33db55989daa3 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 08:12:51.297488 32007 raft_consensus.cc:348] T 2c8df43c895d49cfa758eb04382261d4 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:51.297639 32007 raft_consensus.cc:374] T 2c8df43c895d49cfa758eb04382261d4 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:51.297695 32007 raft_consensus.cc:729] T 2c8df43c895d49cfa758eb04382261d4 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:51.297906 32007 consensus_queue.cc:260] T 2c8df43c895d49cfa758eb04382261d4 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:51.298455 32007 ts_tablet_manager.cc:1398] T 2c8df43c895d49cfa758eb04382261d4 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:12:51.298556 32002 ts_tablet_manager.cc:1398] T d0657f27fd474a41b0c33db55989daa3 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 08:12:51.302953 28261 raft_consensus.cc:1261] T 46d7203013bf4d2ba96ea50ae720d3de P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer ced21ca983b749d0b4d4f6a0e2426bf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:51.304358 30762 raft_consensus.cc:1261] T 46d7203013bf4d2ba96ea50ae720d3de P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer ced21ca983b749d0b4d4f6a0e2426bf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:51.305187 31999 consensus_queue.cc:1035] T 46d7203013bf4d2ba96ea50ae720d3de P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:51.305616 31999 consensus_queue.cc:1035] T 46d7203013bf4d2ba96ea50ae720d3de P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:51.311020 31977 tablet.cc:1722] T 46d7203013bf4d2ba96ea50ae720d3de P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:51.311587 31977 tablet.cc:1620] T 46d7203013bf4d2ba96ea50ae720d3de P ced21ca983b749d0b4d4f6a0e2426bf8: MemRowSet was empty: no flush needed.
I20240503 08:12:51.311915 31975 tablet.cc:1722] T 46d7203013bf4d2ba96ea50ae720d3de P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:51.312191 27859 catalog_manager.cc:5576] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725) does not have the latest schema for tablet 46d7203013bf4d2ba96ea50ae720d3de (table rebalancer_dynamic_table [id=18e03813ffeb43e892ab2db9b92159ba]). Expected version 1 got 0
I20240503 08:12:51.312402 31975 tablet.cc:1620] T 46d7203013bf4d2ba96ea50ae720d3de P 6d1d7de19459435fbde1999e3dfd226b: MemRowSet was empty: no flush needed.
I20240503 08:12:51.318006 27862 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=18e03813ffeb43e892ab2db9b92159ba] alter complete (version 1)
I20240503 08:12:51.323647 27855 catalog_manager.cc:5576] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919) does not have the latest schema for tablet 46d7203013bf4d2ba96ea50ae720d3de (table rebalancer_dynamic_table [id=18e03813ffeb43e892ab2db9b92159ba]). Expected version 1 got 0
I20240503 08:12:51.324986 32020 tablet.cc:1722] T 46d7203013bf4d2ba96ea50ae720d3de P aae860d7ae2b4002bf7794c278f9cc56: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:51.325817 32020 tablet.cc:1620] T 46d7203013bf4d2ba96ea50ae720d3de P aae860d7ae2b4002bf7794c278f9cc56: MemRowSet was empty: no flush needed.
I20240503 08:12:51.327044 27855 catalog_manager.cc:5576] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919) does not have the latest schema for tablet 46d7203013bf4d2ba96ea50ae720d3de (table rebalancer_dynamic_table [id=18e03813ffeb43e892ab2db9b92159ba]). Expected version 1 got 0
I20240503 08:12:51.327883 31969 raft_consensus.cc:482] T e3e58f895edb4095ba3d77653c9197af P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:51.328045 31969 raft_consensus.cc:504] T e3e58f895edb4095ba3d77653c9197af P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:51.328624 31969 leader_election.cc:288] T e3e58f895edb4095ba3d77653c9197af P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:51.329138 31161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3e58f895edb4095ba3d77653c9197af" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:12:51.329334 31161 raft_consensus.cc:2454] T e3e58f895edb4095ba3d77653c9197af P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:12:51.329450 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3e58f895edb4095ba3d77653c9197af" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:12:51.329667 30895 raft_consensus.cc:2454] T e3e58f895edb4095ba3d77653c9197af P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:12:51.329838 30694 leader_election.cc:302] T e3e58f895edb4095ba3d77653c9197af P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, b00f244c69df401997ea99d492218a98; no voters:
I20240503 08:12:51.330734 31969 raft_consensus.cc:2781] T e3e58f895edb4095ba3d77653c9197af P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:51.330837 31969 raft_consensus.cc:482] T e3e58f895edb4095ba3d77653c9197af P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:51.330936 31969 raft_consensus.cc:3037] T e3e58f895edb4095ba3d77653c9197af P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:51.332713 31969 raft_consensus.cc:504] T e3e58f895edb4095ba3d77653c9197af P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:51.333153 31969 leader_election.cc:288] T e3e58f895edb4095ba3d77653c9197af P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Requested vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:51.333563 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3e58f895edb4095ba3d77653c9197af" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:12:51.333633 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3e58f895edb4095ba3d77653c9197af" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:12:51.333796 30895 raft_consensus.cc:3037] T e3e58f895edb4095ba3d77653c9197af P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:51.333837 31159 raft_consensus.cc:3037] T e3e58f895edb4095ba3d77653c9197af P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:51.336529 30895 raft_consensus.cc:2454] T e3e58f895edb4095ba3d77653c9197af P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:12:51.336563 31159 raft_consensus.cc:2454] T e3e58f895edb4095ba3d77653c9197af P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:12:51.336933 30694 leader_election.cc:302] T e3e58f895edb4095ba3d77653c9197af P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, b00f244c69df401997ea99d492218a98; no voters:
I20240503 08:12:51.337383 31969 raft_consensus.cc:2781] T e3e58f895edb4095ba3d77653c9197af P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:51.337565 31969 raft_consensus.cc:686] T e3e58f895edb4095ba3d77653c9197af P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Becoming Leader. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Running, Role: LEADER
I20240503 08:12:51.337836 31969 consensus_queue.cc:237] T e3e58f895edb4095ba3d77653c9197af P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:51.341090 27855 catalog_manager.cc:5484] T e3e58f895edb4095ba3d77653c9197af P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69). New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:12:51.419740 27855 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:51.427579 31977 tablet.cc:1722] T 46d7203013bf4d2ba96ea50ae720d3de P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:51.427867 31975 tablet.cc:1722] T 46d7203013bf4d2ba96ea50ae720d3de P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:51.428340 31975 tablet.cc:1620] T 46d7203013bf4d2ba96ea50ae720d3de P 6d1d7de19459435fbde1999e3dfd226b: MemRowSet was empty: no flush needed.
I20240503 08:12:51.428512 31977 tablet.cc:1620] T 46d7203013bf4d2ba96ea50ae720d3de P ced21ca983b749d0b4d4f6a0e2426bf8: MemRowSet was empty: no flush needed.
I20240503 08:12:51.428555 32020 tablet.cc:1722] T 46d7203013bf4d2ba96ea50ae720d3de P aae860d7ae2b4002bf7794c278f9cc56: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:51.429670 32020 tablet.cc:1620] T 46d7203013bf4d2ba96ea50ae720d3de P aae860d7ae2b4002bf7794c278f9cc56: MemRowSet was empty: no flush needed.
I20240503 08:12:51.439457 27862 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=18e03813ffeb43e892ab2db9b92159ba] alter complete (version 2)
I20240503 08:12:51.544103 27855 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:51.545184 31992 raft_consensus.cc:482] T d0657f27fd474a41b0c33db55989daa3 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:51.545329 31992 raft_consensus.cc:504] T d0657f27fd474a41b0c33db55989daa3 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:51.546051 31992 leader_election.cc:288] T d0657f27fd474a41b0c33db55989daa3 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:51.546577 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0657f27fd474a41b0c33db55989daa3" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:12:51.546695 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0657f27fd474a41b0c33db55989daa3" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:12:51.546809 28132 raft_consensus.cc:2454] T d0657f27fd474a41b0c33db55989daa3 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:12:51.547289 30829 leader_election.cc:302] T d0657f27fd474a41b0c33db55989daa3 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:51.548745 31992 raft_consensus.cc:2781] T d0657f27fd474a41b0c33db55989daa3 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:51.548861 31992 raft_consensus.cc:482] T d0657f27fd474a41b0c33db55989daa3 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:51.548930 31992 raft_consensus.cc:3037] T d0657f27fd474a41b0c33db55989daa3 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:51.551525 31992 raft_consensus.cc:504] T d0657f27fd474a41b0c33db55989daa3 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:51.551822 31977 tablet.cc:1722] T 46d7203013bf4d2ba96ea50ae720d3de P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:51.546975 31159 raft_consensus.cc:2454] T d0657f27fd474a41b0c33db55989daa3 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:12:51.552736 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0657f27fd474a41b0c33db55989daa3" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:12:51.552970 31992 leader_election.cc:288] T d0657f27fd474a41b0c33db55989daa3 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Requested vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:51.552969 28132 raft_consensus.cc:3037] T d0657f27fd474a41b0c33db55989daa3 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:51.553908 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0657f27fd474a41b0c33db55989daa3" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:12:51.554157 31159 raft_consensus.cc:3037] T d0657f27fd474a41b0c33db55989daa3 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:51.555248 31975 tablet.cc:1722] T 46d7203013bf4d2ba96ea50ae720d3de P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:51.556237 28132 raft_consensus.cc:2454] T d0657f27fd474a41b0c33db55989daa3 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:12:51.556718 30829 leader_election.cc:302] T d0657f27fd474a41b0c33db55989daa3 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:51.557030 31992 raft_consensus.cc:2781] T d0657f27fd474a41b0c33db55989daa3 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:51.557189 31992 raft_consensus.cc:686] T d0657f27fd474a41b0c33db55989daa3 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Becoming Leader. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Running, Role: LEADER
I20240503 08:12:51.559217 31992 consensus_queue.cc:237] T d0657f27fd474a41b0c33db55989daa3 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:51.561538 32020 tablet.cc:1722] T 46d7203013bf4d2ba96ea50ae720d3de P aae860d7ae2b4002bf7794c278f9cc56: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:51.563614 31159 raft_consensus.cc:2454] T d0657f27fd474a41b0c33db55989daa3 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:12:51.564549 27859 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table_new_name [id=18e03813ffeb43e892ab2db9b92159ba] alter complete (version 3)
I20240503 08:12:51.565639 27856 catalog_manager.cc:5484] T d0657f27fd474a41b0c33db55989daa3 P ce1496a47d084365ad786cd09e472a2c reported cstate change: term changed from 0 to 1, leader changed from <none> to ce1496a47d084365ad786cd09e472a2c (127.20.230.70). New cstate: current_term: 1 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:12:51.571790 31999 raft_consensus.cc:482] T 2c8df43c895d49cfa758eb04382261d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:51.571938 31999 raft_consensus.cc:504] T 2c8df43c895d49cfa758eb04382261d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:51.572643 31999 leader_election.cc:288] T 2c8df43c895d49cfa758eb04382261d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:51.573325 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c8df43c895d49cfa758eb04382261d4" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:12:51.573560 31159 raft_consensus.cc:2454] T 2c8df43c895d49cfa758eb04382261d4 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 0.
I20240503 08:12:51.573949 30959 leader_election.cc:302] T 2c8df43c895d49cfa758eb04382261d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:51.574182 31999 raft_consensus.cc:2781] T 2c8df43c895d49cfa758eb04382261d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:51.574319 31999 raft_consensus.cc:482] T 2c8df43c895d49cfa758eb04382261d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:51.574419 31999 raft_consensus.cc:3037] T 2c8df43c895d49cfa758eb04382261d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:51.577198 31999 raft_consensus.cc:504] T 2c8df43c895d49cfa758eb04382261d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:51.577895 31999 leader_election.cc:288] T 2c8df43c895d49cfa758eb04382261d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 election: Requested vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:51.578310 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c8df43c895d49cfa758eb04382261d4" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:12:51.578532 31159 raft_consensus.cc:3037] T 2c8df43c895d49cfa758eb04382261d4 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:51.580660 31159 raft_consensus.cc:2454] T 2c8df43c895d49cfa758eb04382261d4 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 1.
I20240503 08:12:51.581002 30959 leader_election.cc:302] T 2c8df43c895d49cfa758eb04382261d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:51.581284 31999 raft_consensus.cc:2781] T 2c8df43c895d49cfa758eb04382261d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:51.581445 31999 raft_consensus.cc:686] T 2c8df43c895d49cfa758eb04382261d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 LEADER]: Becoming Leader. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Running, Role: LEADER
I20240503 08:12:51.581691 31999 consensus_queue.cc:237] T 2c8df43c895d49cfa758eb04382261d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:51.584681 27855 catalog_manager.cc:5484] T 2c8df43c895d49cfa758eb04382261d4 P ced21ca983b749d0b4d4f6a0e2426bf8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71). New cstate: current_term: 1 leader_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:12:51.585023 28392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c8df43c895d49cfa758eb04382261d4" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:51.585204 28392 raft_consensus.cc:2454] T 2c8df43c895d49cfa758eb04382261d4 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 0.
I20240503 08:12:51.585240 28393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c8df43c895d49cfa758eb04382261d4" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:12:51.585439 28393 raft_consensus.cc:3037] T 2c8df43c895d49cfa758eb04382261d4 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:51.588313 28393 raft_consensus.cc:2454] T 2c8df43c895d49cfa758eb04382261d4 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 1.
I20240503 08:12:51.666080 27855 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:51.666298 27855 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:51.668926 27855 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet 46d7203013bf4d2ba96ea50ae720d3de
I20240503 08:12:51.669615 30740 tablet_service.cc:1514] Processing DeleteTablet for tablet 46d7203013bf4d2ba96ea50ae720d3de with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:51 UTC) from {username='slave'} at 127.0.0.1:46794
I20240503 08:12:51.669884 31008 tablet_service.cc:1514] Processing DeleteTablet for tablet 46d7203013bf4d2ba96ea50ae720d3de with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:51 UTC) from {username='slave'} at 127.0.0.1:47020
I20240503 08:12:51.669974 28243 tablet_service.cc:1514] Processing DeleteTablet for tablet 46d7203013bf4d2ba96ea50ae720d3de with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:51 UTC) from {username='slave'} at 127.0.0.1:41960
I20240503 08:12:51.670238 32027 tablet_replica.cc:330] T 46d7203013bf4d2ba96ea50ae720d3de P 6d1d7de19459435fbde1999e3dfd226b: stopping tablet replica
I20240503 08:12:51.670503 32027 raft_consensus.cc:2229] T 46d7203013bf4d2ba96ea50ae720d3de P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:51.670537 32029 tablet_replica.cc:330] T 46d7203013bf4d2ba96ea50ae720d3de P aae860d7ae2b4002bf7794c278f9cc56: stopping tablet replica
I20240503 08:12:51.670737 32028 tablet_replica.cc:330] T 46d7203013bf4d2ba96ea50ae720d3de P ced21ca983b749d0b4d4f6a0e2426bf8: stopping tablet replica
I20240503 08:12:51.670730 32029 raft_consensus.cc:2229] T 46d7203013bf4d2ba96ea50ae720d3de P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:51.670761 32027 raft_consensus.cc:2258] T 46d7203013bf4d2ba96ea50ae720d3de P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:51.670926 32029 raft_consensus.cc:2258] T 46d7203013bf4d2ba96ea50ae720d3de P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:51.670925 32028 raft_consensus.cc:2229] T 46d7203013bf4d2ba96ea50ae720d3de P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:51.671229 32028 raft_consensus.cc:2258] T 46d7203013bf4d2ba96ea50ae720d3de P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:51.671578 32027 ts_tablet_manager.cc:1875] T 46d7203013bf4d2ba96ea50ae720d3de P 6d1d7de19459435fbde1999e3dfd226b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:51.671705 32029 ts_tablet_manager.cc:1875] T 46d7203013bf4d2ba96ea50ae720d3de P aae860d7ae2b4002bf7794c278f9cc56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:51.671895 32028 ts_tablet_manager.cc:1875] T 46d7203013bf4d2ba96ea50ae720d3de P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:51.676087 32028 ts_tablet_manager.cc:1888] T 46d7203013bf4d2ba96ea50ae720d3de P ced21ca983b749d0b4d4f6a0e2426bf8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:51.676204 32028 log.cc:1196] T 46d7203013bf4d2ba96ea50ae720d3de P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal/wals/46d7203013bf4d2ba96ea50ae720d3de
I20240503 08:12:51.676532 32028 ts_tablet_manager.cc:1909] T 46d7203013bf4d2ba96ea50ae720d3de P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting consensus metadata
I20240503 08:12:51.677300 27847 catalog_manager.cc:4830] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): tablet 46d7203013bf4d2ba96ea50ae720d3de (table rebalancer_dynamic_table_new_name [id=18e03813ffeb43e892ab2db9b92159ba]) successfully deleted
I20240503 08:12:51.677597 32029 ts_tablet_manager.cc:1888] T 46d7203013bf4d2ba96ea50ae720d3de P aae860d7ae2b4002bf7794c278f9cc56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:51.677716 32029 log.cc:1196] T 46d7203013bf4d2ba96ea50ae720d3de P aae860d7ae2b4002bf7794c278f9cc56: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal/wals/46d7203013bf4d2ba96ea50ae720d3de
I20240503 08:12:51.677707 32027 ts_tablet_manager.cc:1888] T 46d7203013bf4d2ba96ea50ae720d3de P 6d1d7de19459435fbde1999e3dfd226b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:51.677840 32027 log.cc:1196] T 46d7203013bf4d2ba96ea50ae720d3de P 6d1d7de19459435fbde1999e3dfd226b: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal/wals/46d7203013bf4d2ba96ea50ae720d3de
I20240503 08:12:51.678162 32029 ts_tablet_manager.cc:1909] T 46d7203013bf4d2ba96ea50ae720d3de P aae860d7ae2b4002bf7794c278f9cc56: Deleting consensus metadata
I20240503 08:12:51.678210 32027 ts_tablet_manager.cc:1909] T 46d7203013bf4d2ba96ea50ae720d3de P 6d1d7de19459435fbde1999e3dfd226b: Deleting consensus metadata
I20240503 08:12:51.679080 27847 catalog_manager.cc:4830] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): tablet 46d7203013bf4d2ba96ea50ae720d3de (table rebalancer_dynamic_table_new_name [id=18e03813ffeb43e892ab2db9b92159ba]) successfully deleted
I20240503 08:12:51.679100 27848 catalog_manager.cc:4830] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): tablet 46d7203013bf4d2ba96ea50ae720d3de (table rebalancer_dynamic_table_new_name [id=18e03813ffeb43e892ab2db9b92159ba]) successfully deleted
I20240503 08:12:51.761358 31969 consensus_queue.cc:1035] T e3e58f895edb4095ba3d77653c9197af P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:51.765218 31969 consensus_queue.cc:1035] T e3e58f895edb4095ba3d77653c9197af P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:51.771034 27856 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:51.780107 30740 tablet_service.cc:1467] Processing CreateTablet for tablet ade95de696ad4348b0ed5dbcf62023de (DEFAULT_TABLE table=rebalancer_dynamic_table [id=721a53fe4003458681e2b8783f0cb429]), partition=
I20240503 08:12:51.780494 28243 tablet_service.cc:1467] Processing CreateTablet for tablet ade95de696ad4348b0ed5dbcf62023de (DEFAULT_TABLE table=rebalancer_dynamic_table [id=721a53fe4003458681e2b8783f0cb429]), partition=
I20240503 08:12:51.780750 30740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ade95de696ad4348b0ed5dbcf62023de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:51.780790 31140 tablet_service.cc:1467] Processing CreateTablet for tablet ade95de696ad4348b0ed5dbcf62023de (DEFAULT_TABLE table=rebalancer_dynamic_table [id=721a53fe4003458681e2b8783f0cb429]), partition=
I20240503 08:12:51.781128 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ade95de696ad4348b0ed5dbcf62023de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:51.781276 31140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ade95de696ad4348b0ed5dbcf62023de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:51.790081 32006 tablet_bootstrap.cc:492] T ade95de696ad4348b0ed5dbcf62023de P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:51.792536 32006 tablet_bootstrap.cc:654] T ade95de696ad4348b0ed5dbcf62023de P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:51.792568 32037 tablet_bootstrap.cc:492] T ade95de696ad4348b0ed5dbcf62023de P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:51.794586 32037 tablet_bootstrap.cc:654] T ade95de696ad4348b0ed5dbcf62023de P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:51.792994 32036 tablet_bootstrap.cc:492] T ade95de696ad4348b0ed5dbcf62023de P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:51.798206 32036 tablet_bootstrap.cc:654] T ade95de696ad4348b0ed5dbcf62023de P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:51.800719 32006 tablet_bootstrap.cc:492] T ade95de696ad4348b0ed5dbcf62023de P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:51.800866 32006 ts_tablet_manager.cc:1367] T ade95de696ad4348b0ed5dbcf62023de P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.006s
I20240503 08:12:51.801069 32036 tablet_bootstrap.cc:492] T ade95de696ad4348b0ed5dbcf62023de P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:51.801535 32006 raft_consensus.cc:348] T ade95de696ad4348b0ed5dbcf62023de P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:51.801643 32036 ts_tablet_manager.cc:1367] T ade95de696ad4348b0ed5dbcf62023de P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240503 08:12:51.801751 32006 raft_consensus.cc:374] T ade95de696ad4348b0ed5dbcf62023de P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:51.801869 32006 raft_consensus.cc:729] T ade95de696ad4348b0ed5dbcf62023de P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:51.802026 32037 tablet_bootstrap.cc:492] T ade95de696ad4348b0ed5dbcf62023de P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:51.802166 32037 ts_tablet_manager.cc:1367] T ade95de696ad4348b0ed5dbcf62023de P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20240503 08:12:51.802112 32006 consensus_queue.cc:260] T ade95de696ad4348b0ed5dbcf62023de P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:51.802711 32006 ts_tablet_manager.cc:1398] T ade95de696ad4348b0ed5dbcf62023de P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:51.802652 32036 raft_consensus.cc:348] T ade95de696ad4348b0ed5dbcf62023de P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:51.802840 32036 raft_consensus.cc:374] T ade95de696ad4348b0ed5dbcf62023de P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:51.802922 32036 raft_consensus.cc:729] T ade95de696ad4348b0ed5dbcf62023de P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:51.802836 32037 raft_consensus.cc:348] T ade95de696ad4348b0ed5dbcf62023de P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:51.803027 32037 raft_consensus.cc:374] T ade95de696ad4348b0ed5dbcf62023de P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:51.803103 32037 raft_consensus.cc:729] T ade95de696ad4348b0ed5dbcf62023de P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:51.803144 32036 consensus_queue.cc:260] T ade95de696ad4348b0ed5dbcf62023de P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:51.803298 32037 consensus_queue.cc:260] T ade95de696ad4348b0ed5dbcf62023de P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:51.803831 32037 ts_tablet_manager.cc:1398] T ade95de696ad4348b0ed5dbcf62023de P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:12:51.803932 32036 ts_tablet_manager.cc:1398] T ade95de696ad4348b0ed5dbcf62023de P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:51.875718 31841 raft_consensus.cc:482] T ade95de696ad4348b0ed5dbcf62023de P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:51.875876 31841 raft_consensus.cc:504] T ade95de696ad4348b0ed5dbcf62023de P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:51.876456 31841 leader_election.cc:288] T ade95de696ad4348b0ed5dbcf62023de P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:12:51.876902 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade95de696ad4348b0ed5dbcf62023de" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:12:51.876946 28261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade95de696ad4348b0ed5dbcf62023de" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:12:51.877138 30762 raft_consensus.cc:2454] T ade95de696ad4348b0ed5dbcf62023de P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 0.
I20240503 08:12:51.877177 28261 raft_consensus.cc:2454] T ade95de696ad4348b0ed5dbcf62023de P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 0.
I20240503 08:12:51.877547 31096 leader_election.cc:302] T ade95de696ad4348b0ed5dbcf62023de P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, b00f244c69df401997ea99d492218a98; no voters:
I20240503 08:12:51.877825 31841 raft_consensus.cc:2781] T ade95de696ad4348b0ed5dbcf62023de P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:51.877936 31841 raft_consensus.cc:482] T ade95de696ad4348b0ed5dbcf62023de P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:51.878021 31841 raft_consensus.cc:3037] T ade95de696ad4348b0ed5dbcf62023de P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:51.879842 31841 raft_consensus.cc:504] T ade95de696ad4348b0ed5dbcf62023de P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:51.880322 31841 leader_election.cc:288] T ade95de696ad4348b0ed5dbcf62023de P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 election: Requested vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:12:51.880757 28261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade95de696ad4348b0ed5dbcf62023de" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:12:51.880743 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade95de696ad4348b0ed5dbcf62023de" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:12:51.880939 28261 raft_consensus.cc:3037] T ade95de696ad4348b0ed5dbcf62023de P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:51.880962 30762 raft_consensus.cc:3037] T ade95de696ad4348b0ed5dbcf62023de P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:51.883612 30762 raft_consensus.cc:2454] T ade95de696ad4348b0ed5dbcf62023de P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 1.
I20240503 08:12:51.883625 28261 raft_consensus.cc:2454] T ade95de696ad4348b0ed5dbcf62023de P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 1.
I20240503 08:12:51.884045 31096 leader_election.cc:302] T ade95de696ad4348b0ed5dbcf62023de P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, b00f244c69df401997ea99d492218a98; no voters:
I20240503 08:12:51.884317 31841 raft_consensus.cc:2781] T ade95de696ad4348b0ed5dbcf62023de P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:51.884654 31841 raft_consensus.cc:686] T ade95de696ad4348b0ed5dbcf62023de P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Becoming Leader. State: Replica: b00f244c69df401997ea99d492218a98, State: Running, Role: LEADER
I20240503 08:12:51.884938 31841 consensus_queue.cc:237] T ade95de696ad4348b0ed5dbcf62023de P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:51.887993 27859 catalog_manager.cc:5484] T ade95de696ad4348b0ed5dbcf62023de P b00f244c69df401997ea99d492218a98 reported cstate change: term changed from 0 to 1, leader changed from <none> to b00f244c69df401997ea99d492218a98 (127.20.230.72). New cstate: current_term: 1 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:12:52.011046 27859 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:52.016052 30762 raft_consensus.cc:1261] T ade95de696ad4348b0ed5dbcf62023de P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:52.016209 28261 raft_consensus.cc:1261] T ade95de696ad4348b0ed5dbcf62023de P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:52.016590 31841 consensus_queue.cc:1035] T ade95de696ad4348b0ed5dbcf62023de P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:52.016860 32041 consensus_queue.cc:1035] T ade95de696ad4348b0ed5dbcf62023de P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:52.021111 27855 catalog_manager.cc:5576] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919) does not have the latest schema for tablet ade95de696ad4348b0ed5dbcf62023de (table rebalancer_dynamic_table [id=721a53fe4003458681e2b8783f0cb429]). Expected version 1 got 0
I20240503 08:12:52.021816 32045 tablet.cc:1722] T ade95de696ad4348b0ed5dbcf62023de P b00f244c69df401997ea99d492218a98: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:52.023164 27859 catalog_manager.cc:5576] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725) does not have the latest schema for tablet ade95de696ad4348b0ed5dbcf62023de (table rebalancer_dynamic_table [id=721a53fe4003458681e2b8783f0cb429]). Expected version 1 got 0
I20240503 08:12:52.023418 31975 tablet.cc:1722] T ade95de696ad4348b0ed5dbcf62023de P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:52.023928 31975 tablet.cc:1620] T ade95de696ad4348b0ed5dbcf62023de P 6d1d7de19459435fbde1999e3dfd226b: MemRowSet was empty: no flush needed.
I20240503 08:12:52.023278 32045 tablet.cc:1620] T ade95de696ad4348b0ed5dbcf62023de P b00f244c69df401997ea99d492218a98: MemRowSet was empty: no flush needed.
I20240503 08:12:52.029464 27859 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=721a53fe4003458681e2b8783f0cb429] alter complete (version 1)
I20240503 08:12:52.035835 32020 tablet.cc:1722] T ade95de696ad4348b0ed5dbcf62023de P aae860d7ae2b4002bf7794c278f9cc56: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:52.036311 32020 tablet.cc:1620] T ade95de696ad4348b0ed5dbcf62023de P aae860d7ae2b4002bf7794c278f9cc56: MemRowSet was empty: no flush needed.
I20240503 08:12:52.038770 27859 catalog_manager.cc:5576] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919) does not have the latest schema for tablet ade95de696ad4348b0ed5dbcf62023de (table rebalancer_dynamic_table [id=721a53fe4003458681e2b8783f0cb429]). Expected version 1 got 0
I20240503 08:12:52.087378 31992 consensus_queue.cc:1035] T d0657f27fd474a41b0c33db55989daa3 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:52.093125 32048 consensus_queue.cc:1035] T d0657f27fd474a41b0c33db55989daa3 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:52.099959 31832 consensus_queue.cc:1035] T 2c8df43c895d49cfa758eb04382261d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:52.104480 31999 consensus_queue.cc:1035] T 2c8df43c895d49cfa758eb04382261d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:52.112692 27859 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_extra_table_26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:52.119879 28111 tablet_service.cc:1467] Processing CreateTablet for tablet 35c62760e1de4c2f88d06b97d0286c4c (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=1b577ea991674f799fea7ba7f25e80ad]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:52.120024 30874 tablet_service.cc:1467] Processing CreateTablet for tablet 35c62760e1de4c2f88d06b97d0286c4c (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=1b577ea991674f799fea7ba7f25e80ad]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:52.120337 28111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35c62760e1de4c2f88d06b97d0286c4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.120383 30874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35c62760e1de4c2f88d06b97d0286c4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.121868 28112 tablet_service.cc:1467] Processing CreateTablet for tablet fa7e18041c294a239eb7b8f735b66c53 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=1b577ea991674f799fea7ba7f25e80ad]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:52.122236 28112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa7e18041c294a239eb7b8f735b66c53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.122139 30740 tablet_service.cc:1467] Processing CreateTablet for tablet 35c62760e1de4c2f88d06b97d0286c4c (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=1b577ea991674f799fea7ba7f25e80ad]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:52.122548 30740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35c62760e1de4c2f88d06b97d0286c4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.126058 31140 tablet_service.cc:1467] Processing CreateTablet for tablet 9fc39354ba064392be1af1769513f738 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=1b577ea991674f799fea7ba7f25e80ad]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:52.126091 30742 tablet_service.cc:1467] Processing CreateTablet for tablet fa7e18041c294a239eb7b8f735b66c53 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=1b577ea991674f799fea7ba7f25e80ad]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:52.126422 30742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa7e18041c294a239eb7b8f735b66c53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.126437 31140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fc39354ba064392be1af1769513f738. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.130105 30741 tablet_service.cc:1467] Processing CreateTablet for tablet 9fc39354ba064392be1af1769513f738 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=1b577ea991674f799fea7ba7f25e80ad]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:52.130467 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fc39354ba064392be1af1769513f738. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.131614 32006 tablet_bootstrap.cc:492] T 9fc39354ba064392be1af1769513f738 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:52.133165 32006 tablet_bootstrap.cc:654] T 9fc39354ba064392be1af1769513f738 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:52.135707 30873 tablet_service.cc:1467] Processing CreateTablet for tablet 9fc39354ba064392be1af1769513f738 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=1b577ea991674f799fea7ba7f25e80ad]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:52.136157 30873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fc39354ba064392be1af1769513f738. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.136051 27859 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:52.136442 32055 tablet_bootstrap.cc:492] T 35c62760e1de4c2f88d06b97d0286c4c P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:52.138517 32055 tablet_bootstrap.cc:654] T 35c62760e1de4c2f88d06b97d0286c4c P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:52.148120 32045 tablet.cc:1722] T ade95de696ad4348b0ed5dbcf62023de P b00f244c69df401997ea99d492218a98: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:52.148557 32045 tablet.cc:1620] T ade95de696ad4348b0ed5dbcf62023de P b00f244c69df401997ea99d492218a98: MemRowSet was empty: no flush needed.
I20240503 08:12:52.149143 31975 tablet.cc:1722] T ade95de696ad4348b0ed5dbcf62023de P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:52.149283 30874 tablet_service.cc:1467] Processing CreateTablet for tablet fa7e18041c294a239eb7b8f735b66c53 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=1b577ea991674f799fea7ba7f25e80ad]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:52.149559 31975 tablet.cc:1620] T ade95de696ad4348b0ed5dbcf62023de P 6d1d7de19459435fbde1999e3dfd226b: MemRowSet was empty: no flush needed.
I20240503 08:12:52.149798 30874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa7e18041c294a239eb7b8f735b66c53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.152513 32020 tablet.cc:1722] T ade95de696ad4348b0ed5dbcf62023de P aae860d7ae2b4002bf7794c278f9cc56: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:52.152999 32020 tablet.cc:1620] T ade95de696ad4348b0ed5dbcf62023de P aae860d7ae2b4002bf7794c278f9cc56: MemRowSet was empty: no flush needed.
I20240503 08:12:52.159767 32036 tablet_bootstrap.cc:492] T 35c62760e1de4c2f88d06b97d0286c4c P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:52.161322 32036 tablet_bootstrap.cc:654] T 35c62760e1de4c2f88d06b97d0286c4c P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:52.164808 27855 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=721a53fe4003458681e2b8783f0cb429] alter complete (version 2)
I20240503 08:12:52.171241 32006 tablet_bootstrap.cc:492] T 9fc39354ba064392be1af1769513f738 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:52.171653 32006 ts_tablet_manager.cc:1367] T 9fc39354ba064392be1af1769513f738 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.040s user 0.005s sys 0.005s
I20240503 08:12:52.172627 32006 raft_consensus.cc:348] T 9fc39354ba064392be1af1769513f738 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:52.172920 32006 raft_consensus.cc:374] T 9fc39354ba064392be1af1769513f738 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:52.173051 32006 raft_consensus.cc:729] T 9fc39354ba064392be1af1769513f738 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:52.173354 32036 tablet_bootstrap.cc:492] T 35c62760e1de4c2f88d06b97d0286c4c P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:52.173520 32036 ts_tablet_manager.cc:1367] T 35c62760e1de4c2f88d06b97d0286c4c P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.007s
I20240503 08:12:52.173338 32006 consensus_queue.cc:260] T 9fc39354ba064392be1af1769513f738 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:52.174116 32006 ts_tablet_manager.cc:1398] T 9fc39354ba064392be1af1769513f738 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:52.174803 32056 tablet_bootstrap.cc:492] T 35c62760e1de4c2f88d06b97d0286c4c P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:52.175457 32036 raft_consensus.cc:348] T 35c62760e1de4c2f88d06b97d0286c4c P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:52.175704 32036 raft_consensus.cc:374] T 35c62760e1de4c2f88d06b97d0286c4c P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:52.175776 32036 raft_consensus.cc:729] T 35c62760e1de4c2f88d06b97d0286c4c P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:52.176101 32036 consensus_queue.cc:260] T 35c62760e1de4c2f88d06b97d0286c4c P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:52.176717 32056 tablet_bootstrap.cc:654] T 35c62760e1de4c2f88d06b97d0286c4c P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:52.176788 32036 ts_tablet_manager.cc:1398] T 35c62760e1de4c2f88d06b97d0286c4c P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:12:52.177080 32036 tablet_bootstrap.cc:492] T 9fc39354ba064392be1af1769513f738 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:52.178795 32056 tablet_bootstrap.cc:492] T 35c62760e1de4c2f88d06b97d0286c4c P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:52.178979 32056 ts_tablet_manager.cc:1367] T 35c62760e1de4c2f88d06b97d0286c4c P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:12:52.179690 32056 raft_consensus.cc:348] T 35c62760e1de4c2f88d06b97d0286c4c P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:52.179934 32056 raft_consensus.cc:374] T 35c62760e1de4c2f88d06b97d0286c4c P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:52.180469 32036 tablet_bootstrap.cc:654] T 9fc39354ba064392be1af1769513f738 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:52.180642 32056 raft_consensus.cc:729] T 35c62760e1de4c2f88d06b97d0286c4c P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:52.180871 32056 consensus_queue.cc:260] T 35c62760e1de4c2f88d06b97d0286c4c P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:52.181401 32056 ts_tablet_manager.cc:1398] T 35c62760e1de4c2f88d06b97d0286c4c P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:52.181802 32056 tablet_bootstrap.cc:492] T fa7e18041c294a239eb7b8f735b66c53 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:52.183262 32056 tablet_bootstrap.cc:654] T fa7e18041c294a239eb7b8f735b66c53 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:52.186681 32056 tablet_bootstrap.cc:492] T fa7e18041c294a239eb7b8f735b66c53 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:52.186818 32056 ts_tablet_manager.cc:1367] T fa7e18041c294a239eb7b8f735b66c53 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240503 08:12:52.187770 32056 raft_consensus.cc:348] T fa7e18041c294a239eb7b8f735b66c53 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:52.188093 32056 raft_consensus.cc:374] T fa7e18041c294a239eb7b8f735b66c53 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:52.188172 32056 raft_consensus.cc:729] T fa7e18041c294a239eb7b8f735b66c53 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:52.188421 32056 consensus_queue.cc:260] T fa7e18041c294a239eb7b8f735b66c53 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:52.189121 32056 ts_tablet_manager.cc:1398] T fa7e18041c294a239eb7b8f735b66c53 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:52.189886 32055 tablet_bootstrap.cc:492] T 35c62760e1de4c2f88d06b97d0286c4c P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:52.190073 32055 ts_tablet_manager.cc:1367] T 35c62760e1de4c2f88d06b97d0286c4c P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.054s user 0.007s sys 0.012s
I20240503 08:12:52.190838 32055 raft_consensus.cc:348] T 35c62760e1de4c2f88d06b97d0286c4c P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:52.191038 32055 raft_consensus.cc:374] T 35c62760e1de4c2f88d06b97d0286c4c P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:52.191130 32055 raft_consensus.cc:729] T 35c62760e1de4c2f88d06b97d0286c4c P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:52.191367 32055 consensus_queue.cc:260] T 35c62760e1de4c2f88d06b97d0286c4c P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:52.191705 32036 tablet_bootstrap.cc:492] T 9fc39354ba064392be1af1769513f738 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:52.191833 32036 ts_tablet_manager.cc:1367] T 9fc39354ba064392be1af1769513f738 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.002s
I20240503 08:12:52.191974 32055 ts_tablet_manager.cc:1398] T 35c62760e1de4c2f88d06b97d0286c4c P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:52.192217 32055 tablet_bootstrap.cc:492] T 9fc39354ba064392be1af1769513f738 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:52.192351 32036 raft_consensus.cc:348] T 9fc39354ba064392be1af1769513f738 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:52.192525 32036 raft_consensus.cc:374] T 9fc39354ba064392be1af1769513f738 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:52.192621 32036 raft_consensus.cc:729] T 9fc39354ba064392be1af1769513f738 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:52.192828 32036 consensus_queue.cc:260] T 9fc39354ba064392be1af1769513f738 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:52.193182 32036 ts_tablet_manager.cc:1398] T 9fc39354ba064392be1af1769513f738 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 08:12:52.193994 32036 tablet_bootstrap.cc:492] T fa7e18041c294a239eb7b8f735b66c53 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:52.195562 32036 tablet_bootstrap.cc:654] T fa7e18041c294a239eb7b8f735b66c53 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:52.197537 32055 tablet_bootstrap.cc:654] T 9fc39354ba064392be1af1769513f738 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:52.202910 32055 tablet_bootstrap.cc:492] T 9fc39354ba064392be1af1769513f738 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:52.203068 32055 ts_tablet_manager.cc:1367] T 9fc39354ba064392be1af1769513f738 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240503 08:12:52.203817 32055 raft_consensus.cc:348] T 9fc39354ba064392be1af1769513f738 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:52.204016 32055 raft_consensus.cc:374] T 9fc39354ba064392be1af1769513f738 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:52.204102 32055 raft_consensus.cc:729] T 9fc39354ba064392be1af1769513f738 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:52.204337 32055 consensus_queue.cc:260] T 9fc39354ba064392be1af1769513f738 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:52.204849 32055 ts_tablet_manager.cc:1398] T 9fc39354ba064392be1af1769513f738 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:52.205132 32055 tablet_bootstrap.cc:492] T fa7e18041c294a239eb7b8f735b66c53 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:52.206735 32055 tablet_bootstrap.cc:654] T fa7e18041c294a239eb7b8f735b66c53 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:52.209358 32055 tablet_bootstrap.cc:492] T fa7e18041c294a239eb7b8f735b66c53 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:52.209486 32055 ts_tablet_manager.cc:1367] T fa7e18041c294a239eb7b8f735b66c53 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240503 08:12:52.210119 32055 raft_consensus.cc:348] T fa7e18041c294a239eb7b8f735b66c53 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:52.210296 32055 raft_consensus.cc:374] T fa7e18041c294a239eb7b8f735b66c53 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:52.210393 32055 raft_consensus.cc:729] T fa7e18041c294a239eb7b8f735b66c53 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:52.210602 32055 consensus_queue.cc:260] T fa7e18041c294a239eb7b8f735b66c53 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:52.211113 32055 ts_tablet_manager.cc:1398] T fa7e18041c294a239eb7b8f735b66c53 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:52.222749 32036 tablet_bootstrap.cc:492] T fa7e18041c294a239eb7b8f735b66c53 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:52.222896 32036 ts_tablet_manager.cc:1367] T fa7e18041c294a239eb7b8f735b66c53 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.029s user 0.016s sys 0.011s
I20240503 08:12:52.223501 32036 raft_consensus.cc:348] T fa7e18041c294a239eb7b8f735b66c53 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:52.223711 32036 raft_consensus.cc:374] T fa7e18041c294a239eb7b8f735b66c53 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:52.223781 32036 raft_consensus.cc:729] T fa7e18041c294a239eb7b8f735b66c53 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:52.224005 32036 consensus_queue.cc:260] T fa7e18041c294a239eb7b8f735b66c53 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:52.224531 32036 ts_tablet_manager.cc:1398] T fa7e18041c294a239eb7b8f735b66c53 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:12:52.269337 27855 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:52.273259 32059 raft_consensus.cc:482] T 35c62760e1de4c2f88d06b97d0286c4c P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:52.273427 32059 raft_consensus.cc:504] T 35c62760e1de4c2f88d06b97d0286c4c P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:52.274165 32059 leader_election.cc:288] T 35c62760e1de4c2f88d06b97d0286c4c P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:12:52.274574 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35c62760e1de4c2f88d06b97d0286c4c" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:12:52.274587 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35c62760e1de4c2f88d06b97d0286c4c" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:12:52.274792 30762 raft_consensus.cc:2454] T 35c62760e1de4c2f88d06b97d0286c4c P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:52.274793 30895 raft_consensus.cc:2454] T 35c62760e1de4c2f88d06b97d0286c4c P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:52.275534 28068 leader_election.cc:302] T 35c62760e1de4c2f88d06b97d0286c4c P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 6d1d7de19459435fbde1999e3dfd226b; no voters:
I20240503 08:12:52.275803 32059 raft_consensus.cc:2781] T 35c62760e1de4c2f88d06b97d0286c4c P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:52.275905 32059 raft_consensus.cc:482] T 35c62760e1de4c2f88d06b97d0286c4c P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:52.275972 32059 raft_consensus.cc:3037] T 35c62760e1de4c2f88d06b97d0286c4c P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:52.277277 32045 tablet.cc:1722] T ade95de696ad4348b0ed5dbcf62023de P b00f244c69df401997ea99d492218a98: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:52.278548 32059 raft_consensus.cc:504] T 35c62760e1de4c2f88d06b97d0286c4c P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:52.279088 32020 tablet.cc:1722] T ade95de696ad4348b0ed5dbcf62023de P aae860d7ae2b4002bf7794c278f9cc56: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:52.279744 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35c62760e1de4c2f88d06b97d0286c4c" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:12:52.279906 30761 raft_consensus.cc:3037] T 35c62760e1de4c2f88d06b97d0286c4c P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:52.281603 32059 leader_election.cc:288] T 35c62760e1de4c2f88d06b97d0286c4c P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Requested vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:12:52.282362 30761 raft_consensus.cc:2454] T 35c62760e1de4c2f88d06b97d0286c4c P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:12:52.282380 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35c62760e1de4c2f88d06b97d0286c4c" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:12:52.282562 30895 raft_consensus.cc:3037] T 35c62760e1de4c2f88d06b97d0286c4c P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:52.282822 28068 leader_election.cc:302] T 35c62760e1de4c2f88d06b97d0286c4c P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 6d1d7de19459435fbde1999e3dfd226b; no voters:
I20240503 08:12:52.283345 31975 tablet.cc:1722] T ade95de696ad4348b0ed5dbcf62023de P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:52.288044 30895 raft_consensus.cc:2454] T 35c62760e1de4c2f88d06b97d0286c4c P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:12:52.288815 27855 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table_new_name [id=721a53fe4003458681e2b8783f0cb429] alter complete (version 3)
I20240503 08:12:52.289266 32059 raft_consensus.cc:2781] T 35c62760e1de4c2f88d06b97d0286c4c P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:52.289435 32059 raft_consensus.cc:686] T 35c62760e1de4c2f88d06b97d0286c4c P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Becoming Leader. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Running, Role: LEADER
I20240503 08:12:52.289723 32059 consensus_queue.cc:237] T 35c62760e1de4c2f88d06b97d0286c4c P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:52.295957 27855 catalog_manager.cc:5484] T 35c62760e1de4c2f88d06b97d0286c4c P 1b411c6bb25549368041773d7b710d48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b411c6bb25549368041773d7b710d48 (127.20.230.66). New cstate: current_term: 1 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:12:52.299818 32046 raft_consensus.cc:482] T 9fc39354ba064392be1af1769513f738 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:52.299965 32046 raft_consensus.cc:504] T 9fc39354ba064392be1af1769513f738 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:52.300637 32046 leader_election.cc:288] T 9fc39354ba064392be1af1769513f738 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:12:52.301164 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fc39354ba064392be1af1769513f738" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:12:52.301414 30761 raft_consensus.cc:2454] T 9fc39354ba064392be1af1769513f738 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 0.
I20240503 08:12:52.301376 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fc39354ba064392be1af1769513f738" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:12:52.301620 30895 raft_consensus.cc:2454] T 9fc39354ba064392be1af1769513f738 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 0.
I20240503 08:12:52.301839 31096 leader_election.cc:302] T 9fc39354ba064392be1af1769513f738 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, b00f244c69df401997ea99d492218a98; no voters:
I20240503 08:12:52.302125 32046 raft_consensus.cc:2781] T 9fc39354ba064392be1af1769513f738 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:52.302214 32046 raft_consensus.cc:482] T 9fc39354ba064392be1af1769513f738 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:52.302294 32046 raft_consensus.cc:3037] T 9fc39354ba064392be1af1769513f738 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:52.304383 32046 raft_consensus.cc:504] T 9fc39354ba064392be1af1769513f738 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:52.304947 32046 leader_election.cc:288] T 9fc39354ba064392be1af1769513f738 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 election: Requested vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:12:52.305390 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fc39354ba064392be1af1769513f738" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:12:52.305408 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fc39354ba064392be1af1769513f738" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:12:52.305583 30761 raft_consensus.cc:3037] T 9fc39354ba064392be1af1769513f738 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:52.305588 30895 raft_consensus.cc:3037] T 9fc39354ba064392be1af1769513f738 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:52.308094 30761 raft_consensus.cc:2454] T 9fc39354ba064392be1af1769513f738 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 1.
I20240503 08:12:52.308204 30895 raft_consensus.cc:2454] T 9fc39354ba064392be1af1769513f738 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 1.
I20240503 08:12:52.308462 31096 leader_election.cc:302] T 9fc39354ba064392be1af1769513f738 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, b00f244c69df401997ea99d492218a98; no voters:
I20240503 08:12:52.308743 32046 raft_consensus.cc:2781] T 9fc39354ba064392be1af1769513f738 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:52.308878 32059 raft_consensus.cc:482] T fa7e18041c294a239eb7b8f735b66c53 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:52.308938 32046 raft_consensus.cc:686] T 9fc39354ba064392be1af1769513f738 P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Becoming Leader. State: Replica: b00f244c69df401997ea99d492218a98, State: Running, Role: LEADER
I20240503 08:12:52.309062 32059 raft_consensus.cc:504] T fa7e18041c294a239eb7b8f735b66c53 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:52.309471 32046 consensus_queue.cc:237] T 9fc39354ba064392be1af1769513f738 P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:52.310058 32059 leader_election.cc:288] T fa7e18041c294a239eb7b8f735b66c53 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:12:52.310143 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e18041c294a239eb7b8f735b66c53" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:12:52.310122 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e18041c294a239eb7b8f735b66c53" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:12:52.310357 30761 raft_consensus.cc:2454] T fa7e18041c294a239eb7b8f735b66c53 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:52.310367 30895 raft_consensus.cc:2454] T fa7e18041c294a239eb7b8f735b66c53 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:52.310819 28068 leader_election.cc:302] T fa7e18041c294a239eb7b8f735b66c53 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 6d1d7de19459435fbde1999e3dfd226b, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:52.311108 32059 raft_consensus.cc:2781] T fa7e18041c294a239eb7b8f735b66c53 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:52.311272 32059 raft_consensus.cc:482] T fa7e18041c294a239eb7b8f735b66c53 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:52.311422 32059 raft_consensus.cc:3037] T fa7e18041c294a239eb7b8f735b66c53 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:52.313580 27855 catalog_manager.cc:5484] T 9fc39354ba064392be1af1769513f738 P b00f244c69df401997ea99d492218a98 reported cstate change: term changed from 0 to 1, leader changed from <none> to b00f244c69df401997ea99d492218a98 (127.20.230.72). New cstate: current_term: 1 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:12:52.314251 32059 raft_consensus.cc:504] T fa7e18041c294a239eb7b8f735b66c53 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:52.315804 32059 leader_election.cc:288] T fa7e18041c294a239eb7b8f735b66c53 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Requested vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:12:52.316171 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e18041c294a239eb7b8f735b66c53" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:12:52.316413 30895 raft_consensus.cc:3037] T fa7e18041c294a239eb7b8f735b66c53 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:52.316484 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e18041c294a239eb7b8f735b66c53" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:12:52.316690 30761 raft_consensus.cc:3037] T fa7e18041c294a239eb7b8f735b66c53 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:52.318437 30895 raft_consensus.cc:2454] T fa7e18041c294a239eb7b8f735b66c53 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:12:52.318555 30761 raft_consensus.cc:2454] T fa7e18041c294a239eb7b8f735b66c53 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:12:52.318903 28066 leader_election.cc:302] T fa7e18041c294a239eb7b8f735b66c53 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:52.319156 32059 raft_consensus.cc:2781] T fa7e18041c294a239eb7b8f735b66c53 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:52.319375 32059 raft_consensus.cc:686] T fa7e18041c294a239eb7b8f735b66c53 P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Becoming Leader. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Running, Role: LEADER
I20240503 08:12:52.319612 32059 consensus_queue.cc:237] T fa7e18041c294a239eb7b8f735b66c53 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:52.322786 27855 catalog_manager.cc:5484] T fa7e18041c294a239eb7b8f735b66c53 P 1b411c6bb25549368041773d7b710d48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b411c6bb25549368041773d7b710d48 (127.20.230.66). New cstate: current_term: 1 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:12:52.392912 27855 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:52.393172 27855 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:52.395952 27855 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet ade95de696ad4348b0ed5dbcf62023de
I20240503 08:12:52.396809 30741 tablet_service.cc:1514] Processing DeleteTablet for tablet ade95de696ad4348b0ed5dbcf62023de with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:52 UTC) from {username='slave'} at 127.0.0.1:46794
I20240503 08:12:52.396898 28243 tablet_service.cc:1514] Processing DeleteTablet for tablet ade95de696ad4348b0ed5dbcf62023de with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:52 UTC) from {username='slave'} at 127.0.0.1:41960
I20240503 08:12:52.397465 32070 tablet_replica.cc:330] T ade95de696ad4348b0ed5dbcf62023de P 6d1d7de19459435fbde1999e3dfd226b: stopping tablet replica
I20240503 08:12:52.397475 32071 tablet_replica.cc:330] T ade95de696ad4348b0ed5dbcf62023de P aae860d7ae2b4002bf7794c278f9cc56: stopping tablet replica
I20240503 08:12:52.397476 31140 tablet_service.cc:1514] Processing DeleteTablet for tablet ade95de696ad4348b0ed5dbcf62023de with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:52 UTC) from {username='slave'} at 127.0.0.1:45814
I20240503 08:12:52.397686 32070 raft_consensus.cc:2229] T ade95de696ad4348b0ed5dbcf62023de P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:52.397753 32071 raft_consensus.cc:2229] T ade95de696ad4348b0ed5dbcf62023de P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:52.398029 32070 raft_consensus.cc:2258] T ade95de696ad4348b0ed5dbcf62023de P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:52.398039 32071 raft_consensus.cc:2258] T ade95de696ad4348b0ed5dbcf62023de P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:52.398133 32072 tablet_replica.cc:330] T ade95de696ad4348b0ed5dbcf62023de P b00f244c69df401997ea99d492218a98: stopping tablet replica
I20240503 08:12:52.398365 32072 raft_consensus.cc:2229] T ade95de696ad4348b0ed5dbcf62023de P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:52.398835 32072 raft_consensus.cc:2258] T ade95de696ad4348b0ed5dbcf62023de P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:52.399070 32070 ts_tablet_manager.cc:1875] T ade95de696ad4348b0ed5dbcf62023de P 6d1d7de19459435fbde1999e3dfd226b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:52.399080 32071 ts_tablet_manager.cc:1875] T ade95de696ad4348b0ed5dbcf62023de P aae860d7ae2b4002bf7794c278f9cc56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:52.399696 32072 ts_tablet_manager.cc:1875] T ade95de696ad4348b0ed5dbcf62023de P b00f244c69df401997ea99d492218a98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:52.403836 32070 ts_tablet_manager.cc:1888] T ade95de696ad4348b0ed5dbcf62023de P 6d1d7de19459435fbde1999e3dfd226b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:52.403977 32070 log.cc:1196] T ade95de696ad4348b0ed5dbcf62023de P 6d1d7de19459435fbde1999e3dfd226b: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal/wals/ade95de696ad4348b0ed5dbcf62023de
I20240503 08:12:52.404464 32070 ts_tablet_manager.cc:1909] T ade95de696ad4348b0ed5dbcf62023de P 6d1d7de19459435fbde1999e3dfd226b: Deleting consensus metadata
I20240503 08:12:52.405241 32071 ts_tablet_manager.cc:1888] T ade95de696ad4348b0ed5dbcf62023de P aae860d7ae2b4002bf7794c278f9cc56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:52.405403 32071 log.cc:1196] T ade95de696ad4348b0ed5dbcf62023de P aae860d7ae2b4002bf7794c278f9cc56: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal/wals/ade95de696ad4348b0ed5dbcf62023de
I20240503 08:12:52.405474 27848 catalog_manager.cc:4830] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): tablet ade95de696ad4348b0ed5dbcf62023de (table rebalancer_dynamic_table_new_name [id=721a53fe4003458681e2b8783f0cb429]) successfully deleted
I20240503 08:12:52.406122 32072 ts_tablet_manager.cc:1888] T ade95de696ad4348b0ed5dbcf62023de P b00f244c69df401997ea99d492218a98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:52.406261 32072 log.cc:1196] T ade95de696ad4348b0ed5dbcf62023de P b00f244c69df401997ea99d492218a98: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal/wals/ade95de696ad4348b0ed5dbcf62023de
I20240503 08:12:52.406253 32071 ts_tablet_manager.cc:1909] T ade95de696ad4348b0ed5dbcf62023de P aae860d7ae2b4002bf7794c278f9cc56: Deleting consensus metadata
I20240503 08:12:52.406697 32072 ts_tablet_manager.cc:1909] T ade95de696ad4348b0ed5dbcf62023de P b00f244c69df401997ea99d492218a98: Deleting consensus metadata
I20240503 08:12:52.407279 27847 catalog_manager.cc:4830] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): tablet ade95de696ad4348b0ed5dbcf62023de (table rebalancer_dynamic_table_new_name [id=721a53fe4003458681e2b8783f0cb429]) successfully deleted
I20240503 08:12:52.407613 27846 catalog_manager.cc:4830] TS b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): tablet ade95de696ad4348b0ed5dbcf62023de (table rebalancer_dynamic_table_new_name [id=721a53fe4003458681e2b8783f0cb429]) successfully deleted
I20240503 08:12:52.498363 27855 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:52.510138 30741 tablet_service.cc:1467] Processing CreateTablet for tablet c1f4ca3e4aa24b3b9af806ea0de85823 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f8fabbc38d6b4222851841f69771ea30]), partition=
I20240503 08:12:52.510251 27983 tablet_service.cc:1467] Processing CreateTablet for tablet c1f4ca3e4aa24b3b9af806ea0de85823 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f8fabbc38d6b4222851841f69771ea30]), partition=
I20240503 08:12:52.510234 31140 tablet_service.cc:1467] Processing CreateTablet for tablet c1f4ca3e4aa24b3b9af806ea0de85823 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f8fabbc38d6b4222851841f69771ea30]), partition=
I20240503 08:12:52.510526 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1f4ca3e4aa24b3b9af806ea0de85823. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.510551 27983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1f4ca3e4aa24b3b9af806ea0de85823. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.510581 31140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1f4ca3e4aa24b3b9af806ea0de85823. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.516402 32036 tablet_bootstrap.cc:492] T c1f4ca3e4aa24b3b9af806ea0de85823 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:52.516810 32006 tablet_bootstrap.cc:492] T c1f4ca3e4aa24b3b9af806ea0de85823 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:52.518020 32036 tablet_bootstrap.cc:654] T c1f4ca3e4aa24b3b9af806ea0de85823 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:52.518329 32006 tablet_bootstrap.cc:654] T c1f4ca3e4aa24b3b9af806ea0de85823 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:52.526130 32006 tablet_bootstrap.cc:492] T c1f4ca3e4aa24b3b9af806ea0de85823 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:52.526494 32036 tablet_bootstrap.cc:492] T c1f4ca3e4aa24b3b9af806ea0de85823 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:52.526484 32006 ts_tablet_manager.cc:1367] T c1f4ca3e4aa24b3b9af806ea0de85823 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.002s
I20240503 08:12:52.526628 32036 ts_tablet_manager.cc:1367] T c1f4ca3e4aa24b3b9af806ea0de85823 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20240503 08:12:52.527173 32036 raft_consensus.cc:348] T c1f4ca3e4aa24b3b9af806ea0de85823 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:52.527174 32006 raft_consensus.cc:348] T c1f4ca3e4aa24b3b9af806ea0de85823 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:52.527341 32036 raft_consensus.cc:374] T c1f4ca3e4aa24b3b9af806ea0de85823 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:52.527426 32036 raft_consensus.cc:729] T c1f4ca3e4aa24b3b9af806ea0de85823 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:52.527427 32006 raft_consensus.cc:374] T c1f4ca3e4aa24b3b9af806ea0de85823 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:52.527547 32006 raft_consensus.cc:729] T c1f4ca3e4aa24b3b9af806ea0de85823 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:52.527624 32036 consensus_queue.cc:260] T c1f4ca3e4aa24b3b9af806ea0de85823 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:52.527832 32006 consensus_queue.cc:260] T c1f4ca3e4aa24b3b9af806ea0de85823 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:52.528815 32006 ts_tablet_manager.cc:1398] T c1f4ca3e4aa24b3b9af806ea0de85823 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:52.529279 32036 ts_tablet_manager.cc:1398] T c1f4ca3e4aa24b3b9af806ea0de85823 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 08:12:52.531451 32073 tablet_bootstrap.cc:492] T c1f4ca3e4aa24b3b9af806ea0de85823 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:52.532936 32073 tablet_bootstrap.cc:654] T c1f4ca3e4aa24b3b9af806ea0de85823 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:52.538530 32073 tablet_bootstrap.cc:492] T c1f4ca3e4aa24b3b9af806ea0de85823 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:52.538661 32073 ts_tablet_manager.cc:1367] T c1f4ca3e4aa24b3b9af806ea0de85823 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240503 08:12:52.539242 32073 raft_consensus.cc:348] T c1f4ca3e4aa24b3b9af806ea0de85823 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:52.539409 32073 raft_consensus.cc:374] T c1f4ca3e4aa24b3b9af806ea0de85823 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:52.539505 32073 raft_consensus.cc:729] T c1f4ca3e4aa24b3b9af806ea0de85823 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:52.539700 32073 consensus_queue.cc:260] T c1f4ca3e4aa24b3b9af806ea0de85823 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:52.540163 32073 ts_tablet_manager.cc:1398] T c1f4ca3e4aa24b3b9af806ea0de85823 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:12:52.609175 31969 raft_consensus.cc:482] T c1f4ca3e4aa24b3b9af806ea0de85823 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:52.609346 31969 raft_consensus.cc:504] T c1f4ca3e4aa24b3b9af806ea0de85823 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:52.610083 31969 leader_election.cc:288] T c1f4ca3e4aa24b3b9af806ea0de85823 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:12:52.610507 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f4ca3e4aa24b3b9af806ea0de85823" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:12:52.610728 31159 raft_consensus.cc:2454] T c1f4ca3e4aa24b3b9af806ea0de85823 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:12:52.610785 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f4ca3e4aa24b3b9af806ea0de85823" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:12:52.611012 28002 raft_consensus.cc:2454] T c1f4ca3e4aa24b3b9af806ea0de85823 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:12:52.611158 30694 leader_election.cc:302] T c1f4ca3e4aa24b3b9af806ea0de85823 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, b00f244c69df401997ea99d492218a98; no voters:
I20240503 08:12:52.611444 31969 raft_consensus.cc:2781] T c1f4ca3e4aa24b3b9af806ea0de85823 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:52.611559 31969 raft_consensus.cc:482] T c1f4ca3e4aa24b3b9af806ea0de85823 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:52.611644 31969 raft_consensus.cc:3037] T c1f4ca3e4aa24b3b9af806ea0de85823 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:52.613665 31969 raft_consensus.cc:504] T c1f4ca3e4aa24b3b9af806ea0de85823 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:52.614392 31969 leader_election.cc:288] T c1f4ca3e4aa24b3b9af806ea0de85823 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Requested vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:12:52.614847 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f4ca3e4aa24b3b9af806ea0de85823" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:12:52.615052 28003 raft_consensus.cc:3037] T c1f4ca3e4aa24b3b9af806ea0de85823 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:52.617703 28003 raft_consensus.cc:2454] T c1f4ca3e4aa24b3b9af806ea0de85823 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:12:52.618111 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f4ca3e4aa24b3b9af806ea0de85823" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:12:52.618331 31159 raft_consensus.cc:3037] T c1f4ca3e4aa24b3b9af806ea0de85823 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:52.621008 31159 raft_consensus.cc:2454] T c1f4ca3e4aa24b3b9af806ea0de85823 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:12:52.621416 30694 leader_election.cc:302] T c1f4ca3e4aa24b3b9af806ea0de85823 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, b00f244c69df401997ea99d492218a98; no voters:
I20240503 08:12:52.622004 32077 raft_consensus.cc:2781] T c1f4ca3e4aa24b3b9af806ea0de85823 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:52.633968 32077 raft_consensus.cc:686] T c1f4ca3e4aa24b3b9af806ea0de85823 P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Becoming Leader. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Running, Role: LEADER
I20240503 08:12:52.634341 32077 consensus_queue.cc:237] T c1f4ca3e4aa24b3b9af806ea0de85823 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:52.640281 27856 catalog_manager.cc:5484] T c1f4ca3e4aa24b3b9af806ea0de85823 P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69). New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:12:52.700497 32059 consensus_queue.cc:1035] T fa7e18041c294a239eb7b8f735b66c53 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:52.705178 32059 consensus_queue.cc:1035] T fa7e18041c294a239eb7b8f735b66c53 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:52.748198 27856 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:52.753304 28003 raft_consensus.cc:1261] T c1f4ca3e4aa24b3b9af806ea0de85823 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:52.753289 31159 raft_consensus.cc:1261] T c1f4ca3e4aa24b3b9af806ea0de85823 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:52.753835 32077 consensus_queue.cc:1035] T c1f4ca3e4aa24b3b9af806ea0de85823 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:52.754128 32078 consensus_queue.cc:1035] T c1f4ca3e4aa24b3b9af806ea0de85823 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:52.759739 31975 tablet.cc:1722] T c1f4ca3e4aa24b3b9af806ea0de85823 P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:52.760254 31975 tablet.cc:1620] T c1f4ca3e4aa24b3b9af806ea0de85823 P 6d1d7de19459435fbde1999e3dfd226b: MemRowSet was empty: no flush needed.
I20240503 08:12:52.761406 27859 catalog_manager.cc:5576] TS b00f244c69df401997ea99d492218a98 (127.20.230.72:40893) does not have the latest schema for tablet c1f4ca3e4aa24b3b9af806ea0de85823 (table rebalancer_dynamic_table [id=f8fabbc38d6b4222851841f69771ea30]). Expected version 1 got 0
I20240503 08:12:52.761839 27856 catalog_manager.cc:5576] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081) does not have the latest schema for tablet c1f4ca3e4aa24b3b9af806ea0de85823 (table rebalancer_dynamic_table [id=f8fabbc38d6b4222851841f69771ea30]). Expected version 1 got 0
I20240503 08:12:52.766984 32045 tablet.cc:1722] T c1f4ca3e4aa24b3b9af806ea0de85823 P b00f244c69df401997ea99d492218a98: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:52.767416 32045 tablet.cc:1620] T c1f4ca3e4aa24b3b9af806ea0de85823 P b00f244c69df401997ea99d492218a98: MemRowSet was empty: no flush needed.
I20240503 08:12:52.768074 32088 tablet.cc:1722] T c1f4ca3e4aa24b3b9af806ea0de85823 P 833582ff68c94269be364ebd187a75db: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:52.768723 32088 tablet.cc:1620] T c1f4ca3e4aa24b3b9af806ea0de85823 P 833582ff68c94269be364ebd187a75db: MemRowSet was empty: no flush needed.
I20240503 08:12:52.771528 27859 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=f8fabbc38d6b4222851841f69771ea30] alter complete (version 1)
I20240503 08:12:52.787082 32059 consensus_queue.cc:1035] T 35c62760e1de4c2f88d06b97d0286c4c P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:52.790989 32059 consensus_queue.cc:1035] T 35c62760e1de4c2f88d06b97d0286c4c P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:52.800444 32046 consensus_queue.cc:1035] T 9fc39354ba064392be1af1769513f738 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:52.804140 31841 consensus_queue.cc:1035] T 9fc39354ba064392be1af1769513f738 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:52.841531 27859 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_extra_table_28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:52.849124 27983 tablet_service.cc:1467] Processing CreateTablet for tablet dcc026d6d9bf40059fd6298c40b04c1a (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=49b49624ca8542a3a7429b4b6e32095d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:52.849315 31008 tablet_service.cc:1467] Processing CreateTablet for tablet dcc026d6d9bf40059fd6298c40b04c1a (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=49b49624ca8542a3a7429b4b6e32095d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:52.849565 27983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc026d6d9bf40059fd6298c40b04c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.849717 31008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc026d6d9bf40059fd6298c40b04c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.850143 31140 tablet_service.cc:1467] Processing CreateTablet for tablet dcc026d6d9bf40059fd6298c40b04c1a (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=49b49624ca8542a3a7429b4b6e32095d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:52.850531 31140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc026d6d9bf40059fd6298c40b04c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.851845 30741 tablet_service.cc:1467] Processing CreateTablet for tablet 2222ff7fb5184d6c86010e1475d6c1f7 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=49b49624ca8542a3a7429b4b6e32095d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:52.852226 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2222ff7fb5184d6c86010e1475d6c1f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.853523 31139 tablet_service.cc:1467] Processing CreateTablet for tablet 2222ff7fb5184d6c86010e1475d6c1f7 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=49b49624ca8542a3a7429b4b6e32095d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:52.853894 31139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2222ff7fb5184d6c86010e1475d6c1f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.857550 28243 tablet_service.cc:1467] Processing CreateTablet for tablet 2222ff7fb5184d6c86010e1475d6c1f7 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=49b49624ca8542a3a7429b4b6e32095d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:52.857941 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2222ff7fb5184d6c86010e1475d6c1f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.860761 32036 tablet_bootstrap.cc:492] T 2222ff7fb5184d6c86010e1475d6c1f7 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:52.862169 30874 tablet_service.cc:1467] Processing CreateTablet for tablet 3131a638a8bc4030ab93c9b4b9173b94 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=49b49624ca8542a3a7429b4b6e32095d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:52.862605 30874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3131a638a8bc4030ab93c9b4b9173b94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.865049 32006 tablet_bootstrap.cc:492] T dcc026d6d9bf40059fd6298c40b04c1a P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:52.865482 32094 tablet_bootstrap.cc:492] T dcc026d6d9bf40059fd6298c40b04c1a P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:52.866122 31141 tablet_service.cc:1467] Processing CreateTablet for tablet 3131a638a8bc4030ab93c9b4b9173b94 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=49b49624ca8542a3a7429b4b6e32095d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:52.866386 31141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3131a638a8bc4030ab93c9b4b9173b94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.866626 32006 tablet_bootstrap.cc:654] T dcc026d6d9bf40059fd6298c40b04c1a P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:52.867017 32094 tablet_bootstrap.cc:654] T dcc026d6d9bf40059fd6298c40b04c1a P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:52.869037 32006 tablet_bootstrap.cc:492] T dcc026d6d9bf40059fd6298c40b04c1a P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:52.869159 32006 ts_tablet_manager.cc:1367] T dcc026d6d9bf40059fd6298c40b04c1a P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:12:52.870136 32073 tablet_bootstrap.cc:492] T dcc026d6d9bf40059fd6298c40b04c1a P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:52.870030 32006 raft_consensus.cc:348] T dcc026d6d9bf40059fd6298c40b04c1a P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:52.870210 32006 raft_consensus.cc:374] T dcc026d6d9bf40059fd6298c40b04c1a P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:52.870286 32006 raft_consensus.cc:729] T dcc026d6d9bf40059fd6298c40b04c1a P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:52.870522 32006 consensus_queue.cc:260] T dcc026d6d9bf40059fd6298c40b04c1a P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:52.871011 32006 ts_tablet_manager.cc:1398] T dcc026d6d9bf40059fd6298c40b04c1a P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:52.871568 32073 tablet_bootstrap.cc:654] T dcc026d6d9bf40059fd6298c40b04c1a P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:52.871897 32094 tablet_bootstrap.cc:492] T dcc026d6d9bf40059fd6298c40b04c1a P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:52.872063 32094 ts_tablet_manager.cc:1367] T dcc026d6d9bf40059fd6298c40b04c1a P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 08:12:52.872742 32094 raft_consensus.cc:348] T dcc026d6d9bf40059fd6298c40b04c1a P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:52.872915 32094 raft_consensus.cc:374] T dcc026d6d9bf40059fd6298c40b04c1a P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:52.872993 32094 raft_consensus.cc:729] T dcc026d6d9bf40059fd6298c40b04c1a P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:52.873207 32094 consensus_queue.cc:260] T dcc026d6d9bf40059fd6298c40b04c1a P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:52.873817 32094 ts_tablet_manager.cc:1398] T dcc026d6d9bf40059fd6298c40b04c1a P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:52.874680 32095 tablet_bootstrap.cc:492] T 2222ff7fb5184d6c86010e1475d6c1f7 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:52.876178 32036 tablet_bootstrap.cc:654] T 2222ff7fb5184d6c86010e1475d6c1f7 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:52.876657 32095 tablet_bootstrap.cc:654] T 2222ff7fb5184d6c86010e1475d6c1f7 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:52.878199 27856 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:52.878788 28372 tablet_service.cc:1467] Processing CreateTablet for tablet 3131a638a8bc4030ab93c9b4b9173b94 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=49b49624ca8542a3a7429b4b6e32095d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:52.879202 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3131a638a8bc4030ab93c9b4b9173b94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.879921 32095 tablet_bootstrap.cc:492] T 2222ff7fb5184d6c86010e1475d6c1f7 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:52.880035 32095 ts_tablet_manager.cc:1367] T 2222ff7fb5184d6c86010e1475d6c1f7 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:12:52.880704 32095 raft_consensus.cc:348] T 2222ff7fb5184d6c86010e1475d6c1f7 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:52.880882 32095 raft_consensus.cc:374] T 2222ff7fb5184d6c86010e1475d6c1f7 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:52.880962 32095 raft_consensus.cc:729] T 2222ff7fb5184d6c86010e1475d6c1f7 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:52.881014 32073 tablet_bootstrap.cc:492] T dcc026d6d9bf40059fd6298c40b04c1a P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:52.881148 32073 ts_tablet_manager.cc:1367] T dcc026d6d9bf40059fd6298c40b04c1a P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.002s
I20240503 08:12:52.881155 32095 consensus_queue.cc:260] T 2222ff7fb5184d6c86010e1475d6c1f7 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:52.881729 32073 raft_consensus.cc:348] T dcc026d6d9bf40059fd6298c40b04c1a P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:52.882001 32073 raft_consensus.cc:374] T dcc026d6d9bf40059fd6298c40b04c1a P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:52.882113 32073 raft_consensus.cc:729] T dcc026d6d9bf40059fd6298c40b04c1a P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:52.882335 32073 consensus_queue.cc:260] T dcc026d6d9bf40059fd6298c40b04c1a P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:52.882489 32095 ts_tablet_manager.cc:1398] T 2222ff7fb5184d6c86010e1475d6c1f7 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:52.882822 32073 ts_tablet_manager.cc:1398] T dcc026d6d9bf40059fd6298c40b04c1a P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:52.893388 32101 tablet_bootstrap.cc:492] T 3131a638a8bc4030ab93c9b4b9173b94 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:52.894064 32102 tablet_bootstrap.cc:492] T 3131a638a8bc4030ab93c9b4b9173b94 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:52.894094 31975 tablet.cc:1722] T c1f4ca3e4aa24b3b9af806ea0de85823 P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:52.894601 32036 tablet_bootstrap.cc:492] T 2222ff7fb5184d6c86010e1475d6c1f7 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:52.894778 32036 ts_tablet_manager.cc:1367] T 2222ff7fb5184d6c86010e1475d6c1f7 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.000s
I20240503 08:12:52.895196 32101 tablet_bootstrap.cc:654] T 3131a638a8bc4030ab93c9b4b9173b94 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:52.895902 32036 raft_consensus.cc:348] T 2222ff7fb5184d6c86010e1475d6c1f7 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:52.896127 32036 raft_consensus.cc:374] T 2222ff7fb5184d6c86010e1475d6c1f7 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:52.896212 32036 raft_consensus.cc:729] T 2222ff7fb5184d6c86010e1475d6c1f7 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:52.896262 32102 tablet_bootstrap.cc:654] T 3131a638a8bc4030ab93c9b4b9173b94 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:52.896443 32036 consensus_queue.cc:260] T 2222ff7fb5184d6c86010e1475d6c1f7 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:52.894601 31975 tablet.cc:1620] T c1f4ca3e4aa24b3b9af806ea0de85823 P 6d1d7de19459435fbde1999e3dfd226b: MemRowSet was empty: no flush needed.
I20240503 08:12:52.897322 30652 raft_consensus.cc:482] T 2222ff7fb5184d6c86010e1475d6c1f7 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:52.897471 30652 raft_consensus.cc:504] T 2222ff7fb5184d6c86010e1475d6c1f7 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:52.898280 30652 leader_election.cc:288] T 2222ff7fb5184d6c86010e1475d6c1f7 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
W20240503 08:12:52.899935 28196 leader_election.cc:341] T 2222ff7fb5184d6c86010e1475d6c1f7 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): Not found: Tablet not found: 2222ff7fb5184d6c86010e1475d6c1f7
I20240503 08:12:52.902096 32036 ts_tablet_manager.cc:1398] T 2222ff7fb5184d6c86010e1475d6c1f7 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:12:52.903126 32088 tablet.cc:1722] T c1f4ca3e4aa24b3b9af806ea0de85823 P 833582ff68c94269be364ebd187a75db: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:52.903353 32045 tablet.cc:1722] T c1f4ca3e4aa24b3b9af806ea0de85823 P b00f244c69df401997ea99d492218a98: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:52.903599 32088 tablet.cc:1620] T c1f4ca3e4aa24b3b9af806ea0de85823 P 833582ff68c94269be364ebd187a75db: MemRowSet was empty: no flush needed.
I20240503 08:12:52.903851 32045 tablet.cc:1620] T c1f4ca3e4aa24b3b9af806ea0de85823 P b00f244c69df401997ea99d492218a98: MemRowSet was empty: no flush needed.
I20240503 08:12:52.904655 32101 tablet_bootstrap.cc:492] T 3131a638a8bc4030ab93c9b4b9173b94 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:52.904786 32101 ts_tablet_manager.cc:1367] T 3131a638a8bc4030ab93c9b4b9173b94 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240503 08:12:52.903875 32006 tablet_bootstrap.cc:492] T 2222ff7fb5184d6c86010e1475d6c1f7 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:52.905469 32101 raft_consensus.cc:348] T 3131a638a8bc4030ab93c9b4b9173b94 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:52.905663 32101 raft_consensus.cc:374] T 3131a638a8bc4030ab93c9b4b9173b94 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:52.905735 32101 raft_consensus.cc:729] T 3131a638a8bc4030ab93c9b4b9173b94 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:52.906209 32101 consensus_queue.cc:260] T 3131a638a8bc4030ab93c9b4b9173b94 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:52.906914 32006 tablet_bootstrap.cc:654] T 2222ff7fb5184d6c86010e1475d6c1f7 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:52.907588 32101 ts_tablet_manager.cc:1398] T 3131a638a8bc4030ab93c9b4b9173b94 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:52.919499 27855 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=f8fabbc38d6b4222851841f69771ea30] alter complete (version 2)
I20240503 08:12:52.923552 32006 tablet_bootstrap.cc:492] T 2222ff7fb5184d6c86010e1475d6c1f7 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:52.923699 32006 ts_tablet_manager.cc:1367] T 2222ff7fb5184d6c86010e1475d6c1f7 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.003s
I20240503 08:12:52.924254 32006 raft_consensus.cc:348] T 2222ff7fb5184d6c86010e1475d6c1f7 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:52.924449 32006 raft_consensus.cc:374] T 2222ff7fb5184d6c86010e1475d6c1f7 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:52.924520 32006 raft_consensus.cc:729] T 2222ff7fb5184d6c86010e1475d6c1f7 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:52.924785 32006 consensus_queue.cc:260] T 2222ff7fb5184d6c86010e1475d6c1f7 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:52.925256 32006 ts_tablet_manager.cc:1398] T 2222ff7fb5184d6c86010e1475d6c1f7 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 08:12:52.926048 32006 tablet_bootstrap.cc:492] T 3131a638a8bc4030ab93c9b4b9173b94 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:52.926321 32102 tablet_bootstrap.cc:492] T 3131a638a8bc4030ab93c9b4b9173b94 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:52.926630 32102 ts_tablet_manager.cc:1367] T 3131a638a8bc4030ab93c9b4b9173b94 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.004s
I20240503 08:12:52.927467 32102 raft_consensus.cc:348] T 3131a638a8bc4030ab93c9b4b9173b94 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:52.927739 32102 raft_consensus.cc:374] T 3131a638a8bc4030ab93c9b4b9173b94 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:52.927788 32006 tablet_bootstrap.cc:654] T 3131a638a8bc4030ab93c9b4b9173b94 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:52.927925 32102 raft_consensus.cc:729] T 3131a638a8bc4030ab93c9b4b9173b94 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:52.928222 32102 consensus_queue.cc:260] T 3131a638a8bc4030ab93c9b4b9173b94 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:52.929119 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2222ff7fb5184d6c86010e1475d6c1f7" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:12:52.929358 30761 raft_consensus.cc:2454] T 2222ff7fb5184d6c86010e1475d6c1f7 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:12:52.929757 28198 leader_election.cc:302] T 2222ff7fb5184d6c86010e1475d6c1f7 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, aae860d7ae2b4002bf7794c278f9cc56; no voters: b00f244c69df401997ea99d492218a98
I20240503 08:12:52.929900 32102 ts_tablet_manager.cc:1398] T 3131a638a8bc4030ab93c9b4b9173b94 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:12:52.930215 30652 raft_consensus.cc:2781] T 2222ff7fb5184d6c86010e1475d6c1f7 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:52.930330 30652 raft_consensus.cc:482] T 2222ff7fb5184d6c86010e1475d6c1f7 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:52.930409 30652 raft_consensus.cc:3037] T 2222ff7fb5184d6c86010e1475d6c1f7 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:52.931769 32006 tablet_bootstrap.cc:492] T 3131a638a8bc4030ab93c9b4b9173b94 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:52.931905 32006 ts_tablet_manager.cc:1367] T 3131a638a8bc4030ab93c9b4b9173b94 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20240503 08:12:52.932610 32006 raft_consensus.cc:348] T 3131a638a8bc4030ab93c9b4b9173b94 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:52.932857 32006 raft_consensus.cc:374] T 3131a638a8bc4030ab93c9b4b9173b94 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:52.932955 32006 raft_consensus.cc:729] T 3131a638a8bc4030ab93c9b4b9173b94 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:52.932957 30652 raft_consensus.cc:504] T 2222ff7fb5184d6c86010e1475d6c1f7 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:52.933178 32006 consensus_queue.cc:260] T 3131a638a8bc4030ab93c9b4b9173b94 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:52.933712 32006 ts_tablet_manager.cc:1398] T 3131a638a8bc4030ab93c9b4b9173b94 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:52.934190 30652 leader_election.cc:288] T 2222ff7fb5184d6c86010e1475d6c1f7 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Requested vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:52.934477 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2222ff7fb5184d6c86010e1475d6c1f7" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:12:52.934628 31159 raft_consensus.cc:3037] T 2222ff7fb5184d6c86010e1475d6c1f7 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:52.934623 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2222ff7fb5184d6c86010e1475d6c1f7" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:12:52.934777 30761 raft_consensus.cc:3037] T 2222ff7fb5184d6c86010e1475d6c1f7 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:52.936290 31159 raft_consensus.cc:2454] T 2222ff7fb5184d6c86010e1475d6c1f7 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:52.936689 30761 raft_consensus.cc:2454] T 2222ff7fb5184d6c86010e1475d6c1f7 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:52.936769 28196 leader_election.cc:302] T 2222ff7fb5184d6c86010e1475d6c1f7 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, b00f244c69df401997ea99d492218a98; no voters:
I20240503 08:12:52.937345 31903 raft_consensus.cc:2781] T 2222ff7fb5184d6c86010e1475d6c1f7 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:52.937463 31903 raft_consensus.cc:686] T 2222ff7fb5184d6c86010e1475d6c1f7 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Becoming Leader. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Running, Role: LEADER
I20240503 08:12:52.937698 31903 consensus_queue.cc:237] T 2222ff7fb5184d6c86010e1475d6c1f7 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:52.941203 27855 catalog_manager.cc:5484] T 2222ff7fb5184d6c86010e1475d6c1f7 P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: term changed from 0 to 1, leader changed from <none> to aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67). New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:12:53.023263 27855 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:53.024731 31992 raft_consensus.cc:482] T 3131a638a8bc4030ab93c9b4b9173b94 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:53.024905 31992 raft_consensus.cc:504] T 3131a638a8bc4030ab93c9b4b9173b94 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:53.025676 31992 leader_election.cc:288] T 3131a638a8bc4030ab93c9b4b9173b94 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:53.026131 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3131a638a8bc4030ab93c9b4b9173b94" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:12:53.026258 28392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3131a638a8bc4030ab93c9b4b9173b94" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:53.026352 31159 raft_consensus.cc:2454] T 3131a638a8bc4030ab93c9b4b9173b94 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:12:53.026487 28392 raft_consensus.cc:2454] T 3131a638a8bc4030ab93c9b4b9173b94 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:12:53.026803 30827 leader_election.cc:302] T 3131a638a8bc4030ab93c9b4b9173b94 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:53.027065 31992 raft_consensus.cc:2781] T 3131a638a8bc4030ab93c9b4b9173b94 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:53.027168 31992 raft_consensus.cc:482] T 3131a638a8bc4030ab93c9b4b9173b94 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:53.027244 31992 raft_consensus.cc:3037] T 3131a638a8bc4030ab93c9b4b9173b94 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:53.029711 31992 raft_consensus.cc:504] T 3131a638a8bc4030ab93c9b4b9173b94 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:53.030740 31992 leader_election.cc:288] T 3131a638a8bc4030ab93c9b4b9173b94 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Requested vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:53.031024 31975 tablet.cc:1722] T c1f4ca3e4aa24b3b9af806ea0de85823 P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:53.031450 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3131a638a8bc4030ab93c9b4b9173b94" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:12:53.031656 31159 raft_consensus.cc:3037] T 3131a638a8bc4030ab93c9b4b9173b94 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:53.032106 32088 tablet.cc:1722] T c1f4ca3e4aa24b3b9af806ea0de85823 P 833582ff68c94269be364ebd187a75db: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:53.032510 32045 tablet.cc:1722] T c1f4ca3e4aa24b3b9af806ea0de85823 P b00f244c69df401997ea99d492218a98: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:53.034873 31159 raft_consensus.cc:2454] T 3131a638a8bc4030ab93c9b4b9173b94 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:12:53.035250 30827 leader_election.cc:302] T 3131a638a8bc4030ab93c9b4b9173b94 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:53.038961 32048 raft_consensus.cc:2781] T 3131a638a8bc4030ab93c9b4b9173b94 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:53.041440 28392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3131a638a8bc4030ab93c9b4b9173b94" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:12:53.041651 28392 raft_consensus.cc:3037] T 3131a638a8bc4030ab93c9b4b9173b94 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:53.043581 28392 raft_consensus.cc:2454] T 3131a638a8bc4030ab93c9b4b9173b94 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:12:53.044468 27856 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table_new_name [id=f8fabbc38d6b4222851841f69771ea30] alter complete (version 3)
I20240503 08:12:53.047068 32084 raft_consensus.cc:482] T dcc026d6d9bf40059fd6298c40b04c1a P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:53.047245 32084 raft_consensus.cc:504] T dcc026d6d9bf40059fd6298c40b04c1a P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:53.048002 32084 leader_election.cc:288] T dcc026d6d9bf40059fd6298c40b04c1a P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:53.048828 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc026d6d9bf40059fd6298c40b04c1a" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:53.049010 31027 raft_consensus.cc:2454] T dcc026d6d9bf40059fd6298c40b04c1a P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:53.049407 27937 leader_election.cc:302] T dcc026d6d9bf40059fd6298c40b04c1a P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:53.049644 32084 raft_consensus.cc:2781] T dcc026d6d9bf40059fd6298c40b04c1a P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:53.049782 32084 raft_consensus.cc:482] T dcc026d6d9bf40059fd6298c40b04c1a P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:53.049685 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc026d6d9bf40059fd6298c40b04c1a" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:12:53.049860 32084 raft_consensus.cc:3037] T dcc026d6d9bf40059fd6298c40b04c1a P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:53.049947 31159 raft_consensus.cc:2454] T dcc026d6d9bf40059fd6298c40b04c1a P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:53.050607 32048 raft_consensus.cc:686] T 3131a638a8bc4030ab93c9b4b9173b94 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Becoming Leader. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Running, Role: LEADER
I20240503 08:12:53.051088 32048 consensus_queue.cc:237] T 3131a638a8bc4030ab93c9b4b9173b94 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:53.052390 32084 raft_consensus.cc:504] T dcc026d6d9bf40059fd6298c40b04c1a P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:53.053009 32084 leader_election.cc:288] T dcc026d6d9bf40059fd6298c40b04c1a P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Requested vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:53.053436 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc026d6d9bf40059fd6298c40b04c1a" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:12:53.053661 31027 raft_consensus.cc:3037] T dcc026d6d9bf40059fd6298c40b04c1a P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:53.055619 27855 catalog_manager.cc:5484] T 3131a638a8bc4030ab93c9b4b9173b94 P ce1496a47d084365ad786cd09e472a2c reported cstate change: term changed from 0 to 1, leader changed from <none> to ce1496a47d084365ad786cd09e472a2c (127.20.230.70). New cstate: current_term: 1 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:12:53.056280 31027 raft_consensus.cc:2454] T dcc026d6d9bf40059fd6298c40b04c1a P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:53.056739 27937 leader_election.cc:302] T dcc026d6d9bf40059fd6298c40b04c1a P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:53.057010 32084 raft_consensus.cc:2781] T dcc026d6d9bf40059fd6298c40b04c1a P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:53.057155 32084 raft_consensus.cc:686] T dcc026d6d9bf40059fd6298c40b04c1a P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Becoming Leader. State: Replica: 833582ff68c94269be364ebd187a75db, State: Running, Role: LEADER
I20240503 08:12:53.057397 32084 consensus_queue.cc:237] T dcc026d6d9bf40059fd6298c40b04c1a P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:53.060048 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc026d6d9bf40059fd6298c40b04c1a" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:12:53.060242 31159 raft_consensus.cc:3037] T dcc026d6d9bf40059fd6298c40b04c1a P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:53.063194 31159 raft_consensus.cc:2454] T dcc026d6d9bf40059fd6298c40b04c1a P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:53.063509 27855 catalog_manager.cc:5484] T dcc026d6d9bf40059fd6298c40b04c1a P 833582ff68c94269be364ebd187a75db reported cstate change: term changed from 0 to 1, leader changed from <none> to 833582ff68c94269be364ebd187a75db (127.20.230.65). New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:12:53.151868 27855 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:53.152089 27855 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:53.154824 27855 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet c1f4ca3e4aa24b3b9af806ea0de85823
I20240503 08:12:53.155508 31141 tablet_service.cc:1514] Processing DeleteTablet for tablet c1f4ca3e4aa24b3b9af806ea0de85823 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:53 UTC) from {username='slave'} at 127.0.0.1:45814
I20240503 08:12:53.155666 30741 tablet_service.cc:1514] Processing DeleteTablet for tablet c1f4ca3e4aa24b3b9af806ea0de85823 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:53 UTC) from {username='slave'} at 127.0.0.1:46794
I20240503 08:12:53.156075 27983 tablet_service.cc:1514] Processing DeleteTablet for tablet c1f4ca3e4aa24b3b9af806ea0de85823 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:53 UTC) from {username='slave'} at 127.0.0.1:55610
I20240503 08:12:53.156337 32117 tablet_replica.cc:330] T c1f4ca3e4aa24b3b9af806ea0de85823 P 6d1d7de19459435fbde1999e3dfd226b: stopping tablet replica
I20240503 08:12:53.156590 32117 raft_consensus.cc:2229] T c1f4ca3e4aa24b3b9af806ea0de85823 P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:53.156975 32117 raft_consensus.cc:2258] T c1f4ca3e4aa24b3b9af806ea0de85823 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:53.157788 32118 tablet_replica.cc:330] T c1f4ca3e4aa24b3b9af806ea0de85823 P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:12:53.157829 32117 ts_tablet_manager.cc:1875] T c1f4ca3e4aa24b3b9af806ea0de85823 P 6d1d7de19459435fbde1999e3dfd226b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:53.157868 32116 tablet_replica.cc:330] T c1f4ca3e4aa24b3b9af806ea0de85823 P b00f244c69df401997ea99d492218a98: stopping tablet replica
I20240503 08:12:53.157991 32118 raft_consensus.cc:2229] T c1f4ca3e4aa24b3b9af806ea0de85823 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:53.158082 32116 raft_consensus.cc:2229] T c1f4ca3e4aa24b3b9af806ea0de85823 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:53.158202 32118 raft_consensus.cc:2258] T c1f4ca3e4aa24b3b9af806ea0de85823 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:53.158279 32116 raft_consensus.cc:2258] T c1f4ca3e4aa24b3b9af806ea0de85823 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:53.158996 32116 ts_tablet_manager.cc:1875] T c1f4ca3e4aa24b3b9af806ea0de85823 P b00f244c69df401997ea99d492218a98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:53.159137 32118 ts_tablet_manager.cc:1875] T c1f4ca3e4aa24b3b9af806ea0de85823 P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:53.163242 32117 ts_tablet_manager.cc:1888] T c1f4ca3e4aa24b3b9af806ea0de85823 P 6d1d7de19459435fbde1999e3dfd226b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:53.163410 32117 log.cc:1196] T c1f4ca3e4aa24b3b9af806ea0de85823 P 6d1d7de19459435fbde1999e3dfd226b: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal/wals/c1f4ca3e4aa24b3b9af806ea0de85823
I20240503 08:12:53.163865 32117 ts_tablet_manager.cc:1909] T c1f4ca3e4aa24b3b9af806ea0de85823 P 6d1d7de19459435fbde1999e3dfd226b: Deleting consensus metadata
I20240503 08:12:53.165050 32116 ts_tablet_manager.cc:1888] T c1f4ca3e4aa24b3b9af806ea0de85823 P b00f244c69df401997ea99d492218a98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:53.164908 27848 catalog_manager.cc:4830] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): tablet c1f4ca3e4aa24b3b9af806ea0de85823 (table rebalancer_dynamic_table_new_name [id=f8fabbc38d6b4222851841f69771ea30]) successfully deleted
I20240503 08:12:53.165169 32116 log.cc:1196] T c1f4ca3e4aa24b3b9af806ea0de85823 P b00f244c69df401997ea99d492218a98: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal/wals/c1f4ca3e4aa24b3b9af806ea0de85823
I20240503 08:12:53.165210 32118 ts_tablet_manager.cc:1888] T c1f4ca3e4aa24b3b9af806ea0de85823 P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:53.165316 32118 log.cc:1196] T c1f4ca3e4aa24b3b9af806ea0de85823 P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/c1f4ca3e4aa24b3b9af806ea0de85823
I20240503 08:12:53.165606 32116 ts_tablet_manager.cc:1909] T c1f4ca3e4aa24b3b9af806ea0de85823 P b00f244c69df401997ea99d492218a98: Deleting consensus metadata
I20240503 08:12:53.165789 32118 ts_tablet_manager.cc:1909] T c1f4ca3e4aa24b3b9af806ea0de85823 P 833582ff68c94269be364ebd187a75db: Deleting consensus metadata
I20240503 08:12:53.166489 27846 catalog_manager.cc:4830] TS b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): tablet c1f4ca3e4aa24b3b9af806ea0de85823 (table rebalancer_dynamic_table_new_name [id=f8fabbc38d6b4222851841f69771ea30]) successfully deleted
I20240503 08:12:53.166846 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet c1f4ca3e4aa24b3b9af806ea0de85823 (table rebalancer_dynamic_table_new_name [id=f8fabbc38d6b4222851841f69771ea30]) successfully deleted
I20240503 08:12:53.257077 27855 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:53.263972 28111 tablet_service.cc:1467] Processing CreateTablet for tablet 627a6784140b44dd895ac6b19be92d6b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ca551b32f8a44f27a46af4c2347cb664]), partition=
I20240503 08:12:53.264256 30741 tablet_service.cc:1467] Processing CreateTablet for tablet 627a6784140b44dd895ac6b19be92d6b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ca551b32f8a44f27a46af4c2347cb664]), partition=
I20240503 08:12:53.264309 28111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 627a6784140b44dd895ac6b19be92d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:53.264380 30874 tablet_service.cc:1467] Processing CreateTablet for tablet 627a6784140b44dd895ac6b19be92d6b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ca551b32f8a44f27a46af4c2347cb664]), partition=
I20240503 08:12:53.264638 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 627a6784140b44dd895ac6b19be92d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:53.264695 30874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 627a6784140b44dd895ac6b19be92d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:53.270824 32101 tablet_bootstrap.cc:492] T 627a6784140b44dd895ac6b19be92d6b P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:53.272455 32101 tablet_bootstrap.cc:654] T 627a6784140b44dd895ac6b19be92d6b P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:53.273129 32036 tablet_bootstrap.cc:492] T 627a6784140b44dd895ac6b19be92d6b P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:53.274394 32101 tablet_bootstrap.cc:492] T 627a6784140b44dd895ac6b19be92d6b P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:53.274468 32036 tablet_bootstrap.cc:654] T 627a6784140b44dd895ac6b19be92d6b P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:53.274617 32101 ts_tablet_manager.cc:1367] T 627a6784140b44dd895ac6b19be92d6b P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:12:53.275403 32101 raft_consensus.cc:348] T 627a6784140b44dd895ac6b19be92d6b P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:53.276186 32119 tablet_bootstrap.cc:492] T 627a6784140b44dd895ac6b19be92d6b P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:53.276413 32036 tablet_bootstrap.cc:492] T 627a6784140b44dd895ac6b19be92d6b P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:53.276544 32036 ts_tablet_manager.cc:1367] T 627a6784140b44dd895ac6b19be92d6b P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:53.276799 32101 raft_consensus.cc:374] T 627a6784140b44dd895ac6b19be92d6b P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:53.276896 32101 raft_consensus.cc:729] T 627a6784140b44dd895ac6b19be92d6b P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:53.277130 32101 consensus_queue.cc:260] T 627a6784140b44dd895ac6b19be92d6b P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:53.277312 32036 raft_consensus.cc:348] T 627a6784140b44dd895ac6b19be92d6b P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:53.277485 32036 raft_consensus.cc:374] T 627a6784140b44dd895ac6b19be92d6b P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:53.277804 32119 tablet_bootstrap.cc:654] T 627a6784140b44dd895ac6b19be92d6b P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:53.277947 32101 ts_tablet_manager.cc:1398] T 627a6784140b44dd895ac6b19be92d6b P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:53.278843 32036 raft_consensus.cc:729] T 627a6784140b44dd895ac6b19be92d6b P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:53.279103 32036 consensus_queue.cc:260] T 627a6784140b44dd895ac6b19be92d6b P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:53.279492 32036 ts_tablet_manager.cc:1398] T 627a6784140b44dd895ac6b19be92d6b P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:53.279606 32119 tablet_bootstrap.cc:492] T 627a6784140b44dd895ac6b19be92d6b P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:53.279755 32119 ts_tablet_manager.cc:1367] T 627a6784140b44dd895ac6b19be92d6b P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:12:53.280432 32119 raft_consensus.cc:348] T 627a6784140b44dd895ac6b19be92d6b P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:53.280642 32119 raft_consensus.cc:374] T 627a6784140b44dd895ac6b19be92d6b P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:53.280740 32119 raft_consensus.cc:729] T 627a6784140b44dd895ac6b19be92d6b P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:53.280966 32119 consensus_queue.cc:260] T 627a6784140b44dd895ac6b19be92d6b P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:53.281459 32119 ts_tablet_manager.cc:1398] T 627a6784140b44dd895ac6b19be92d6b P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:12:53.434499 31992 consensus_queue.cc:1035] T 3131a638a8bc4030ab93c9b4b9173b94 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:53.438390 32112 consensus_queue.cc:1035] T 3131a638a8bc4030ab93c9b4b9173b94 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:53.443734 32059 raft_consensus.cc:482] T 627a6784140b44dd895ac6b19be92d6b P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:53.443874 32059 raft_consensus.cc:504] T 627a6784140b44dd895ac6b19be92d6b P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:53.444736 32059 leader_election.cc:288] T 627a6784140b44dd895ac6b19be92d6b P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:12:53.444968 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "627a6784140b44dd895ac6b19be92d6b" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:12:53.445199 30761 raft_consensus.cc:2454] T 627a6784140b44dd895ac6b19be92d6b P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:53.445833 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "627a6784140b44dd895ac6b19be92d6b" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:12:53.446003 28068 leader_election.cc:302] T 627a6784140b44dd895ac6b19be92d6b P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 6d1d7de19459435fbde1999e3dfd226b; no voters:
I20240503 08:12:53.446048 30895 raft_consensus.cc:2454] T 627a6784140b44dd895ac6b19be92d6b P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:53.446424 32059 raft_consensus.cc:2781] T 627a6784140b44dd895ac6b19be92d6b P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:53.446624 32059 raft_consensus.cc:482] T 627a6784140b44dd895ac6b19be92d6b P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:53.446765 32059 raft_consensus.cc:3037] T 627a6784140b44dd895ac6b19be92d6b P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:53.449658 32059 raft_consensus.cc:504] T 627a6784140b44dd895ac6b19be92d6b P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:53.451942 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "627a6784140b44dd895ac6b19be92d6b" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:12:53.452092 30761 raft_consensus.cc:3037] T 627a6784140b44dd895ac6b19be92d6b P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:53.454068 30761 raft_consensus.cc:2454] T 627a6784140b44dd895ac6b19be92d6b P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:12:53.454586 28068 leader_election.cc:302] T 627a6784140b44dd895ac6b19be92d6b P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 6d1d7de19459435fbde1999e3dfd226b; no voters:
I20240503 08:12:53.454825 31803 raft_consensus.cc:2781] T 627a6784140b44dd895ac6b19be92d6b P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:53.455973 32059 leader_election.cc:288] T 627a6784140b44dd895ac6b19be92d6b P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Requested vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:12:53.456418 31803 raft_consensus.cc:686] T 627a6784140b44dd895ac6b19be92d6b P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Becoming Leader. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Running, Role: LEADER
I20240503 08:12:53.456704 31803 consensus_queue.cc:237] T 627a6784140b44dd895ac6b19be92d6b P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:53.457067 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "627a6784140b44dd895ac6b19be92d6b" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:12:53.457258 30895 raft_consensus.cc:3037] T 627a6784140b44dd895ac6b19be92d6b P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:53.459810 30895 raft_consensus.cc:2454] T 627a6784140b44dd895ac6b19be92d6b P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:12:53.460332 27862 catalog_manager.cc:5484] T 627a6784140b44dd895ac6b19be92d6b P 1b411c6bb25549368041773d7b710d48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b411c6bb25549368041773d7b710d48 (127.20.230.66). New cstate: current_term: 1 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:12:53.485383 31903 consensus_queue.cc:1035] T 2222ff7fb5184d6c86010e1475d6c1f7 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:53.491328 30652 consensus_queue.cc:1035] T 2222ff7fb5184d6c86010e1475d6c1f7 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:53.497663 31980 consensus_queue.cc:1035] T dcc026d6d9bf40059fd6298c40b04c1a P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:53.502555 31980 consensus_queue.cc:1035] T dcc026d6d9bf40059fd6298c40b04c1a P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:53.583089 27856 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_extra_table_30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:53.591207 28243 tablet_service.cc:1467] Processing CreateTablet for tablet 1cac39d5831d4b6887250f7a9d63dcd5 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b0813f15eaf04597949cf586c648712c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:53.591439 30741 tablet_service.cc:1467] Processing CreateTablet for tablet 1cac39d5831d4b6887250f7a9d63dcd5 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b0813f15eaf04597949cf586c648712c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:53.591647 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cac39d5831d4b6887250f7a9d63dcd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:53.591917 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cac39d5831d4b6887250f7a9d63dcd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:53.593058 27983 tablet_service.cc:1467] Processing CreateTablet for tablet 6a96965691644db8b9461d146536a026 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b0813f15eaf04597949cf586c648712c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:53.593492 27983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a96965691644db8b9461d146536a026. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:53.596117 31008 tablet_service.cc:1467] Processing CreateTablet for tablet fe4cf85b2a5d46fea0bb09ccda5e6351 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b0813f15eaf04597949cf586c648712c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:53.596562 31008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe4cf85b2a5d46fea0bb09ccda5e6351. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:53.596499 28372 tablet_service.cc:1467] Processing CreateTablet for tablet fe4cf85b2a5d46fea0bb09ccda5e6351 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b0813f15eaf04597949cf586c648712c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:53.596625 30874 tablet_service.cc:1467] Processing CreateTablet for tablet fe4cf85b2a5d46fea0bb09ccda5e6351 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b0813f15eaf04597949cf586c648712c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:53.596688 28242 tablet_service.cc:1467] Processing CreateTablet for tablet 6a96965691644db8b9461d146536a026 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b0813f15eaf04597949cf586c648712c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:53.596892 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe4cf85b2a5d46fea0bb09ccda5e6351. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:53.596987 30874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe4cf85b2a5d46fea0bb09ccda5e6351. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:53.597070 28242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a96965691644db8b9461d146536a026. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:53.602722 31141 tablet_service.cc:1467] Processing CreateTablet for tablet 1cac39d5831d4b6887250f7a9d63dcd5 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b0813f15eaf04597949cf586c648712c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:53.603142 31141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cac39d5831d4b6887250f7a9d63dcd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:53.605894 32036 tablet_bootstrap.cc:492] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:53.606149 31139 tablet_service.cc:1467] Processing CreateTablet for tablet 6a96965691644db8b9461d146536a026 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b0813f15eaf04597949cf586c648712c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:53.606519 31139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a96965691644db8b9461d146536a026. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:53.607481 32036 tablet_bootstrap.cc:654] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:53.611181 27856 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:53.618247 32139 tablet_bootstrap.cc:492] T fe4cf85b2a5d46fea0bb09ccda5e6351 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:53.620908 32140 tablet_bootstrap.cc:492] T 6a96965691644db8b9461d146536a026 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:53.622512 32140 tablet_bootstrap.cc:654] T 6a96965691644db8b9461d146536a026 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:53.632511 32140 tablet_bootstrap.cc:492] T 6a96965691644db8b9461d146536a026 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:53.632733 32140 ts_tablet_manager.cc:1367] T 6a96965691644db8b9461d146536a026 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.009s
I20240503 08:12:53.633569 32140 raft_consensus.cc:348] T 6a96965691644db8b9461d146536a026 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:53.633958 32140 raft_consensus.cc:374] T 6a96965691644db8b9461d146536a026 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:53.634105 32140 raft_consensus.cc:729] T 6a96965691644db8b9461d146536a026 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:53.634385 32140 consensus_queue.cc:260] T 6a96965691644db8b9461d146536a026 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:53.635223 32140 ts_tablet_manager.cc:1398] T 6a96965691644db8b9461d146536a026 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:53.638584 32101 tablet_bootstrap.cc:492] T fe4cf85b2a5d46fea0bb09ccda5e6351 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:53.640130 32101 tablet_bootstrap.cc:654] T fe4cf85b2a5d46fea0bb09ccda5e6351 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:53.640573 32036 tablet_bootstrap.cc:492] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:53.640925 32036 ts_tablet_manager.cc:1367] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.035s user 0.009s sys 0.001s
I20240503 08:12:53.641659 32036 raft_consensus.cc:348] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:53.641870 32036 raft_consensus.cc:374] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:53.641947 32036 raft_consensus.cc:729] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:53.642161 32101 tablet_bootstrap.cc:492] T fe4cf85b2a5d46fea0bb09ccda5e6351 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:53.642294 32101 ts_tablet_manager.cc:1367] T fe4cf85b2a5d46fea0bb09ccda5e6351 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20240503 08:12:53.642223 32036 consensus_queue.cc:260] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:53.642670 32036 ts_tablet_manager.cc:1398] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:53.642974 32101 raft_consensus.cc:348] T fe4cf85b2a5d46fea0bb09ccda5e6351 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:53.643152 32101 raft_consensus.cc:374] T fe4cf85b2a5d46fea0bb09ccda5e6351 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:53.643235 32101 raft_consensus.cc:729] T fe4cf85b2a5d46fea0bb09ccda5e6351 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:53.643425 32101 consensus_queue.cc:260] T fe4cf85b2a5d46fea0bb09ccda5e6351 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:53.645320 32101 ts_tablet_manager.cc:1398] T fe4cf85b2a5d46fea0bb09ccda5e6351 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 08:12:53.643819 32136 tablet_bootstrap.cc:492] T 1cac39d5831d4b6887250f7a9d63dcd5 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:53.648677 30895 raft_consensus.cc:1261] T 627a6784140b44dd895ac6b19be92d6b P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:53.649238 31803 consensus_queue.cc:1035] T 627a6784140b44dd895ac6b19be92d6b P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:53.651705 32136 tablet_bootstrap.cc:654] T 1cac39d5831d4b6887250f7a9d63dcd5 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:53.655864 32143 tablet_bootstrap.cc:492] T 1cac39d5831d4b6887250f7a9d63dcd5 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:53.656478 32139 tablet_bootstrap.cc:654] T fe4cf85b2a5d46fea0bb09ccda5e6351 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:53.656719 30761 raft_consensus.cc:1261] T 627a6784140b44dd895ac6b19be92d6b P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:53.657292 31803 consensus_queue.cc:1035] T 627a6784140b44dd895ac6b19be92d6b P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:53.661537 27856 catalog_manager.cc:5576] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725) does not have the latest schema for tablet 627a6784140b44dd895ac6b19be92d6b (table rebalancer_dynamic_table [id=ca551b32f8a44f27a46af4c2347cb664]). Expected version 1 got 0
I20240503 08:12:53.661612 32143 tablet_bootstrap.cc:654] T 1cac39d5831d4b6887250f7a9d63dcd5 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:53.662710 32147 tablet.cc:1722] T 627a6784140b44dd895ac6b19be92d6b P 1b411c6bb25549368041773d7b710d48: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:53.663445 32147 tablet.cc:1620] T 627a6784140b44dd895ac6b19be92d6b P 1b411c6bb25549368041773d7b710d48: MemRowSet was empty: no flush needed.
I20240503 08:12:53.663381 32148 tablet.cc:1722] T 627a6784140b44dd895ac6b19be92d6b P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:53.663894 32148 tablet.cc:1620] T 627a6784140b44dd895ac6b19be92d6b P 6d1d7de19459435fbde1999e3dfd226b: MemRowSet was empty: no flush needed.
I20240503 08:12:53.666042 27855 catalog_manager.cc:5576] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725) does not have the latest schema for tablet 627a6784140b44dd895ac6b19be92d6b (table rebalancer_dynamic_table [id=ca551b32f8a44f27a46af4c2347cb664]). Expected version 1 got 0
I20240503 08:12:53.670415 32136 tablet_bootstrap.cc:492] T 1cac39d5831d4b6887250f7a9d63dcd5 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:53.670625 32136 ts_tablet_manager.cc:1367] T 1cac39d5831d4b6887250f7a9d63dcd5 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.002s
I20240503 08:12:53.671412 32136 raft_consensus.cc:348] T 1cac39d5831d4b6887250f7a9d63dcd5 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:53.671684 32136 raft_consensus.cc:374] T 1cac39d5831d4b6887250f7a9d63dcd5 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:53.671814 32136 raft_consensus.cc:729] T 1cac39d5831d4b6887250f7a9d63dcd5 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:53.672101 32136 consensus_queue.cc:260] T 1cac39d5831d4b6887250f7a9d63dcd5 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:53.672750 32136 ts_tablet_manager.cc:1398] T 1cac39d5831d4b6887250f7a9d63dcd5 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:53.673302 32136 tablet_bootstrap.cc:492] T 6a96965691644db8b9461d146536a026 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:53.675853 27862 catalog_manager.cc:5576] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747) does not have the latest schema for tablet 627a6784140b44dd895ac6b19be92d6b (table rebalancer_dynamic_table [id=ca551b32f8a44f27a46af4c2347cb664]). Expected version 1 got 0
I20240503 08:12:53.676009 32137 tablet_bootstrap.cc:492] T fe4cf85b2a5d46fea0bb09ccda5e6351 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:53.677548 32137 tablet_bootstrap.cc:654] T fe4cf85b2a5d46fea0bb09ccda5e6351 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:53.678619 27855 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=ca551b32f8a44f27a46af4c2347cb664] alter complete (version 1)
I20240503 08:12:53.678886 32136 tablet_bootstrap.cc:654] T 6a96965691644db8b9461d146536a026 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:53.679677 32139 tablet_bootstrap.cc:492] T fe4cf85b2a5d46fea0bb09ccda5e6351 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:53.679811 32139 ts_tablet_manager.cc:1367] T fe4cf85b2a5d46fea0bb09ccda5e6351 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.062s user 0.003s sys 0.000s
I20240503 08:12:53.680508 32139 raft_consensus.cc:348] T fe4cf85b2a5d46fea0bb09ccda5e6351 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:53.680722 32139 raft_consensus.cc:374] T fe4cf85b2a5d46fea0bb09ccda5e6351 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:53.680817 32139 raft_consensus.cc:729] T fe4cf85b2a5d46fea0bb09ccda5e6351 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:53.681064 32139 consensus_queue.cc:260] T fe4cf85b2a5d46fea0bb09ccda5e6351 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:53.682183 32139 ts_tablet_manager.cc:1398] T fe4cf85b2a5d46fea0bb09ccda5e6351 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:53.684329 32112 raft_consensus.cc:482] T fe4cf85b2a5d46fea0bb09ccda5e6351 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:53.684518 32143 tablet_bootstrap.cc:492] T 1cac39d5831d4b6887250f7a9d63dcd5 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:53.684490 32112 raft_consensus.cc:504] T fe4cf85b2a5d46fea0bb09ccda5e6351 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:53.684660 32143 ts_tablet_manager.cc:1367] T 1cac39d5831d4b6887250f7a9d63dcd5 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.000s
I20240503 08:12:53.685135 32112 leader_election.cc:288] T fe4cf85b2a5d46fea0bb09ccda5e6351 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:53.685313 32143 raft_consensus.cc:348] T 1cac39d5831d4b6887250f7a9d63dcd5 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:53.685530 32143 raft_consensus.cc:374] T 1cac39d5831d4b6887250f7a9d63dcd5 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:53.685478 31028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4cf85b2a5d46fea0bb09ccda5e6351" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:53.685613 32143 raft_consensus.cc:729] T 1cac39d5831d4b6887250f7a9d63dcd5 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:53.685871 32143 consensus_queue.cc:260] T 1cac39d5831d4b6887250f7a9d63dcd5 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:53.686278 28392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4cf85b2a5d46fea0bb09ccda5e6351" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:53.686462 28392 raft_consensus.cc:2454] T fe4cf85b2a5d46fea0bb09ccda5e6351 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:12:53.686506 32143 ts_tablet_manager.cc:1398] T 1cac39d5831d4b6887250f7a9d63dcd5 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20240503 08:12:53.686959 30828 leader_election.cc:341] T fe4cf85b2a5d46fea0bb09ccda5e6351 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:53.687186 32143 tablet_bootstrap.cc:492] T 6a96965691644db8b9461d146536a026 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:53.687286 31601 raft_consensus.cc:482] T fe4cf85b2a5d46fea0bb09ccda5e6351 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:53.687394 31601 raft_consensus.cc:504] T fe4cf85b2a5d46fea0bb09ccda5e6351 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:53.687964 31601 leader_election.cc:288] T fe4cf85b2a5d46fea0bb09ccda5e6351 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:12:53.688603 30830 leader_election.cc:302] T fe4cf85b2a5d46fea0bb09ccda5e6351 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c, d09a11a03989421b9f28019dc1c14cc2; no voters: ced21ca983b749d0b4d4f6a0e2426bf8
I20240503 08:12:53.688719 32143 tablet_bootstrap.cc:654] T 6a96965691644db8b9461d146536a026 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:53.689007 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4cf85b2a5d46fea0bb09ccda5e6351" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:53.689324 30894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4cf85b2a5d46fea0bb09ccda5e6351" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:12:53.689517 30894 raft_consensus.cc:2454] T fe4cf85b2a5d46fea0bb09ccda5e6351 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:12:53.689939 32112 raft_consensus.cc:2781] T fe4cf85b2a5d46fea0bb09ccda5e6351 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:53.690022 32112 raft_consensus.cc:482] T fe4cf85b2a5d46fea0bb09ccda5e6351 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:53.690119 32112 raft_consensus.cc:3037] T fe4cf85b2a5d46fea0bb09ccda5e6351 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
W20240503 08:12:53.690610 28327 leader_election.cc:341] T fe4cf85b2a5d46fea0bb09ccda5e6351 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:53.700546 32137 tablet_bootstrap.cc:492] T fe4cf85b2a5d46fea0bb09ccda5e6351 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:53.700706 32137 ts_tablet_manager.cc:1367] T fe4cf85b2a5d46fea0bb09ccda5e6351 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.006s
I20240503 08:12:53.702337 32143 tablet_bootstrap.cc:492] T 6a96965691644db8b9461d146536a026 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:53.702507 32143 ts_tablet_manager.cc:1367] T 6a96965691644db8b9461d146536a026 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.005s
I20240503 08:12:53.702919 32137 raft_consensus.cc:348] T fe4cf85b2a5d46fea0bb09ccda5e6351 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:53.703186 32137 raft_consensus.cc:374] T fe4cf85b2a5d46fea0bb09ccda5e6351 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:53.703299 32137 raft_consensus.cc:729] T fe4cf85b2a5d46fea0bb09ccda5e6351 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:53.703330 32143 raft_consensus.cc:348] T 6a96965691644db8b9461d146536a026 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:53.703548 32143 raft_consensus.cc:374] T 6a96965691644db8b9461d146536a026 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:53.703631 32143 raft_consensus.cc:729] T 6a96965691644db8b9461d146536a026 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:53.703586 32137 consensus_queue.cc:260] T fe4cf85b2a5d46fea0bb09ccda5e6351 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:53.703860 32143 consensus_queue.cc:260] T 6a96965691644db8b9461d146536a026 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:53.706218 32137 ts_tablet_manager.cc:1398] T fe4cf85b2a5d46fea0bb09ccda5e6351 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240503 08:12:53.706558 28326 leader_election.cc:302] T fe4cf85b2a5d46fea0bb09ccda5e6351 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c, d09a11a03989421b9f28019dc1c14cc2; no voters: ced21ca983b749d0b4d4f6a0e2426bf8
I20240503 08:12:53.706995 32115 raft_consensus.cc:2781] T fe4cf85b2a5d46fea0bb09ccda5e6351 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:53.707093 32115 raft_consensus.cc:482] T fe4cf85b2a5d46fea0bb09ccda5e6351 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:53.707203 32115 raft_consensus.cc:3037] T fe4cf85b2a5d46fea0bb09ccda5e6351 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:53.709316 32143 ts_tablet_manager.cc:1398] T 6a96965691644db8b9461d146536a026 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20240503 08:12:53.715842 32115 raft_consensus.cc:504] T fe4cf85b2a5d46fea0bb09ccda5e6351 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:53.716780 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4cf85b2a5d46fea0bb09ccda5e6351" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:12:53.716972 31027 raft_consensus.cc:3037] T fe4cf85b2a5d46fea0bb09ccda5e6351 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:53.717165 32115 leader_election.cc:288] T fe4cf85b2a5d46fea0bb09ccda5e6351 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Requested vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:12:53.717726 30894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4cf85b2a5d46fea0bb09ccda5e6351" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:12:53.718217 32112 raft_consensus.cc:504] T fe4cf85b2a5d46fea0bb09ccda5e6351 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:53.718657 30894 raft_consensus.cc:2379] T fe4cf85b2a5d46fea0bb09ccda5e6351 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d09a11a03989421b9f28019dc1c14cc2 in current term 1: Already voted for candidate ce1496a47d084365ad786cd09e472a2c in this term.
I20240503 08:12:53.719875 31027 raft_consensus.cc:2454] T fe4cf85b2a5d46fea0bb09ccda5e6351 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:12:53.719949 28392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4cf85b2a5d46fea0bb09ccda5e6351" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:12:53.720185 28392 raft_consensus.cc:2379] T fe4cf85b2a5d46fea0bb09ccda5e6351 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ce1496a47d084365ad786cd09e472a2c in current term 1: Already voted for candidate d09a11a03989421b9f28019dc1c14cc2 in this term.
I20240503 08:12:53.720551 31028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4cf85b2a5d46fea0bb09ccda5e6351" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:12:53.720641 32112 leader_election.cc:288] T fe4cf85b2a5d46fea0bb09ccda5e6351 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Requested vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:53.720798 31028 raft_consensus.cc:2379] T fe4cf85b2a5d46fea0bb09ccda5e6351 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ce1496a47d084365ad786cd09e472a2c in current term 1: Already voted for candidate d09a11a03989421b9f28019dc1c14cc2 in this term.
I20240503 08:12:53.721419 30828 leader_election.cc:302] T fe4cf85b2a5d46fea0bb09ccda5e6351 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c; no voters: ced21ca983b749d0b4d4f6a0e2426bf8, d09a11a03989421b9f28019dc1c14cc2
I20240503 08:12:53.722008 28327 leader_election.cc:302] T fe4cf85b2a5d46fea0bb09ccda5e6351 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8, d09a11a03989421b9f28019dc1c14cc2; no voters: ce1496a47d084365ad786cd09e472a2c
I20240503 08:12:53.722514 32115 raft_consensus.cc:2781] T fe4cf85b2a5d46fea0bb09ccda5e6351 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:53.722815 32115 raft_consensus.cc:686] T fe4cf85b2a5d46fea0bb09ccda5e6351 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Becoming Leader. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Running, Role: LEADER
I20240503 08:12:53.723121 32115 consensus_queue.cc:237] T fe4cf85b2a5d46fea0bb09ccda5e6351 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:53.723799 32112 raft_consensus.cc:2726] T fe4cf85b2a5d46fea0bb09ccda5e6351 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 08:12:53.733426 27855 catalog_manager.cc:5484] T fe4cf85b2a5d46fea0bb09ccda5e6351 P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68). New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:12:53.737356 32154 tablet.cc:1722] T 627a6784140b44dd895ac6b19be92d6b P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:53.737459 27856 catalog_manager.cc:5576] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747) does not have the latest schema for tablet 627a6784140b44dd895ac6b19be92d6b (table rebalancer_dynamic_table [id=ca551b32f8a44f27a46af4c2347cb664]). Expected version 1 got 0
I20240503 08:12:53.737973 32154 tablet.cc:1620] T 627a6784140b44dd895ac6b19be92d6b P ce1496a47d084365ad786cd09e472a2c: MemRowSet was empty: no flush needed.
I20240503 08:12:53.739573 32136 tablet_bootstrap.cc:492] T 6a96965691644db8b9461d146536a026 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:53.739746 32136 ts_tablet_manager.cc:1367] T 6a96965691644db8b9461d146536a026 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.066s user 0.010s sys 0.009s
I20240503 08:12:53.740495 32136 raft_consensus.cc:348] T 6a96965691644db8b9461d146536a026 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:53.740722 32136 raft_consensus.cc:374] T 6a96965691644db8b9461d146536a026 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:53.740792 32136 raft_consensus.cc:729] T 6a96965691644db8b9461d146536a026 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:53.741007 32136 consensus_queue.cc:260] T 6a96965691644db8b9461d146536a026 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:53.742439 32136 ts_tablet_manager.cc:1398] T 6a96965691644db8b9461d146536a026 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 08:12:53.812906 27855 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:53.819797 32147 tablet.cc:1722] T 627a6784140b44dd895ac6b19be92d6b P 1b411c6bb25549368041773d7b710d48: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:53.820297 32147 tablet.cc:1620] T 627a6784140b44dd895ac6b19be92d6b P 1b411c6bb25549368041773d7b710d48: MemRowSet was empty: no flush needed.
I20240503 08:12:53.821033 32154 tablet.cc:1722] T 627a6784140b44dd895ac6b19be92d6b P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:53.821137 32148 tablet.cc:1722] T 627a6784140b44dd895ac6b19be92d6b P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:53.821488 32154 tablet.cc:1620] T 627a6784140b44dd895ac6b19be92d6b P ce1496a47d084365ad786cd09e472a2c: MemRowSet was empty: no flush needed.
I20240503 08:12:53.821513 32148 tablet.cc:1620] T 627a6784140b44dd895ac6b19be92d6b P 6d1d7de19459435fbde1999e3dfd226b: MemRowSet was empty: no flush needed.
I20240503 08:12:53.830474 27855 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=ca551b32f8a44f27a46af4c2347cb664] alter complete (version 2)
I20240503 08:12:53.869721 31903 raft_consensus.cc:482] T 1cac39d5831d4b6887250f7a9d63dcd5 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:53.869911 31903 raft_consensus.cc:504] T 1cac39d5831d4b6887250f7a9d63dcd5 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:53.871022 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cac39d5831d4b6887250f7a9d63dcd5" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:12:53.871246 30761 raft_consensus.cc:2454] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:12:53.871255 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cac39d5831d4b6887250f7a9d63dcd5" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:12:53.871506 31159 raft_consensus.cc:2454] T 1cac39d5831d4b6887250f7a9d63dcd5 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:12:53.871606 28198 leader_election.cc:302] T 1cac39d5831d4b6887250f7a9d63dcd5 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:53.871865 30652 raft_consensus.cc:2781] T 1cac39d5831d4b6887250f7a9d63dcd5 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:53.871987 30652 raft_consensus.cc:482] T 1cac39d5831d4b6887250f7a9d63dcd5 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:53.872071 30652 raft_consensus.cc:3037] T 1cac39d5831d4b6887250f7a9d63dcd5 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:53.872412 31903 leader_election.cc:288] T 1cac39d5831d4b6887250f7a9d63dcd5 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:53.874286 30652 raft_consensus.cc:504] T 1cac39d5831d4b6887250f7a9d63dcd5 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:53.874820 30652 leader_election.cc:288] T 1cac39d5831d4b6887250f7a9d63dcd5 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Requested vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:53.875170 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cac39d5831d4b6887250f7a9d63dcd5" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:12:53.875223 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cac39d5831d4b6887250f7a9d63dcd5" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:12:53.875334 30761 raft_consensus.cc:3037] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:53.875422 31159 raft_consensus.cc:3037] T 1cac39d5831d4b6887250f7a9d63dcd5 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:53.877473 31159 raft_consensus.cc:2454] T 1cac39d5831d4b6887250f7a9d63dcd5 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:53.877596 30761 raft_consensus.cc:2454] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:53.877861 28196 leader_election.cc:302] T 1cac39d5831d4b6887250f7a9d63dcd5 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, b00f244c69df401997ea99d492218a98; no voters:
I20240503 08:12:53.878131 30652 raft_consensus.cc:2781] T 1cac39d5831d4b6887250f7a9d63dcd5 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:53.878280 30652 raft_consensus.cc:686] T 1cac39d5831d4b6887250f7a9d63dcd5 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Becoming Leader. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Running, Role: LEADER
I20240503 08:12:53.878572 30652 consensus_queue.cc:237] T 1cac39d5831d4b6887250f7a9d63dcd5 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:53.881685 27855 catalog_manager.cc:5484] T 1cac39d5831d4b6887250f7a9d63dcd5 P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: term changed from 0 to 1, leader changed from <none> to aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67). New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:12:53.935288 27855 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:53.942063 32147 tablet.cc:1722] T 627a6784140b44dd895ac6b19be92d6b P 1b411c6bb25549368041773d7b710d48: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:53.942814 32154 tablet.cc:1722] T 627a6784140b44dd895ac6b19be92d6b P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:53.943816 32148 tablet.cc:1722] T 627a6784140b44dd895ac6b19be92d6b P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:53.951498 27855 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table_new_name [id=ca551b32f8a44f27a46af4c2347cb664] alter complete (version 3)
I20240503 08:12:54.053949 27856 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:54.054169 27856 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:54.057286 27856 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet 627a6784140b44dd895ac6b19be92d6b
I20240503 08:12:54.058197 30874 tablet_service.cc:1514] Processing DeleteTablet for tablet 627a6784140b44dd895ac6b19be92d6b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:54 UTC) from {username='slave'} at 127.0.0.1:34426
I20240503 08:12:54.058485 30741 tablet_service.cc:1514] Processing DeleteTablet for tablet 627a6784140b44dd895ac6b19be92d6b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:54 UTC) from {username='slave'} at 127.0.0.1:46794
I20240503 08:12:54.058777 32160 tablet_replica.cc:330] T 627a6784140b44dd895ac6b19be92d6b P ce1496a47d084365ad786cd09e472a2c: stopping tablet replica
I20240503 08:12:54.058979 32160 raft_consensus.cc:2229] T 627a6784140b44dd895ac6b19be92d6b P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:54.059278 32160 raft_consensus.cc:2258] T 627a6784140b44dd895ac6b19be92d6b P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:54.059948 32160 ts_tablet_manager.cc:1875] T 627a6784140b44dd895ac6b19be92d6b P ce1496a47d084365ad786cd09e472a2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:54.061600 28111 tablet_service.cc:1514] Processing DeleteTablet for tablet 627a6784140b44dd895ac6b19be92d6b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:54 UTC) from {username='slave'} at 127.0.0.1:57912
I20240503 08:12:54.065573 32161 tablet_replica.cc:330] T 627a6784140b44dd895ac6b19be92d6b P 6d1d7de19459435fbde1999e3dfd226b: stopping tablet replica
I20240503 08:12:54.065806 32161 raft_consensus.cc:2229] T 627a6784140b44dd895ac6b19be92d6b P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:54.066076 32160 ts_tablet_manager.cc:1888] T 627a6784140b44dd895ac6b19be92d6b P ce1496a47d084365ad786cd09e472a2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:54.066093 32161 raft_consensus.cc:2258] T 627a6784140b44dd895ac6b19be92d6b P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:54.066131 32162 tablet_replica.cc:330] T 627a6784140b44dd895ac6b19be92d6b P 1b411c6bb25549368041773d7b710d48: stopping tablet replica
I20240503 08:12:54.066319 32162 raft_consensus.cc:2229] T 627a6784140b44dd895ac6b19be92d6b P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:54.066473 32160 log.cc:1196] T 627a6784140b44dd895ac6b19be92d6b P ce1496a47d084365ad786cd09e472a2c: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal/wals/627a6784140b44dd895ac6b19be92d6b
I20240503 08:12:54.066730 32162 raft_consensus.cc:2258] T 627a6784140b44dd895ac6b19be92d6b P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:54.066998 32160 ts_tablet_manager.cc:1909] T 627a6784140b44dd895ac6b19be92d6b P ce1496a47d084365ad786cd09e472a2c: Deleting consensus metadata
I20240503 08:12:54.067191 32161 ts_tablet_manager.cc:1875] T 627a6784140b44dd895ac6b19be92d6b P 6d1d7de19459435fbde1999e3dfd226b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:54.067960 32162 ts_tablet_manager.cc:1875] T 627a6784140b44dd895ac6b19be92d6b P 1b411c6bb25549368041773d7b710d48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:54.067960 27846 catalog_manager.cc:4830] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): tablet 627a6784140b44dd895ac6b19be92d6b (table rebalancer_dynamic_table_new_name [id=ca551b32f8a44f27a46af4c2347cb664]) successfully deleted
I20240503 08:12:54.073065 32161 ts_tablet_manager.cc:1888] T 627a6784140b44dd895ac6b19be92d6b P 6d1d7de19459435fbde1999e3dfd226b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:54.073189 32161 log.cc:1196] T 627a6784140b44dd895ac6b19be92d6b P 6d1d7de19459435fbde1999e3dfd226b: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal/wals/627a6784140b44dd895ac6b19be92d6b
I20240503 08:12:54.073601 32161 ts_tablet_manager.cc:1909] T 627a6784140b44dd895ac6b19be92d6b P 6d1d7de19459435fbde1999e3dfd226b: Deleting consensus metadata
I20240503 08:12:54.075039 27848 catalog_manager.cc:4830] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): tablet 627a6784140b44dd895ac6b19be92d6b (table rebalancer_dynamic_table_new_name [id=ca551b32f8a44f27a46af4c2347cb664]) successfully deleted
I20240503 08:12:54.075459 32162 ts_tablet_manager.cc:1888] T 627a6784140b44dd895ac6b19be92d6b P 1b411c6bb25549368041773d7b710d48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:54.075546 32162 log.cc:1196] T 627a6784140b44dd895ac6b19be92d6b P 1b411c6bb25549368041773d7b710d48: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/627a6784140b44dd895ac6b19be92d6b
I20240503 08:12:54.075888 32162 ts_tablet_manager.cc:1909] T 627a6784140b44dd895ac6b19be92d6b P 1b411c6bb25549368041773d7b710d48: Deleting consensus metadata
I20240503 08:12:54.076674 27848 catalog_manager.cc:4830] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): tablet 627a6784140b44dd895ac6b19be92d6b (table rebalancer_dynamic_table_new_name [id=ca551b32f8a44f27a46af4c2347cb664]) successfully deleted
I20240503 08:12:54.084820 30652 raft_consensus.cc:482] T 6a96965691644db8b9461d146536a026 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:54.084964 30652 raft_consensus.cc:504] T 6a96965691644db8b9461d146536a026 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:54.085645 30652 leader_election.cc:288] T 6a96965691644db8b9461d146536a026 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:12:54.086236 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a96965691644db8b9461d146536a026" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:12:54.086323 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a96965691644db8b9461d146536a026" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:12:54.086455 31159 raft_consensus.cc:2454] T 6a96965691644db8b9461d146536a026 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:12:54.086544 28002 raft_consensus.cc:2454] T 6a96965691644db8b9461d146536a026 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:12:54.086848 28196 leader_election.cc:302] T 6a96965691644db8b9461d146536a026 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, b00f244c69df401997ea99d492218a98; no voters:
I20240503 08:12:54.087124 30652 raft_consensus.cc:2781] T 6a96965691644db8b9461d146536a026 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:54.087211 30652 raft_consensus.cc:482] T 6a96965691644db8b9461d146536a026 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:54.087321 30652 raft_consensus.cc:3037] T 6a96965691644db8b9461d146536a026 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:54.089735 30652 raft_consensus.cc:504] T 6a96965691644db8b9461d146536a026 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:54.090327 30652 leader_election.cc:288] T 6a96965691644db8b9461d146536a026 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Requested vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:12:54.090660 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a96965691644db8b9461d146536a026" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:12:54.090721 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a96965691644db8b9461d146536a026" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:12:54.090844 31159 raft_consensus.cc:3037] T 6a96965691644db8b9461d146536a026 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:54.090911 28002 raft_consensus.cc:3037] T 6a96965691644db8b9461d146536a026 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:54.093457 28002 raft_consensus.cc:2454] T 6a96965691644db8b9461d146536a026 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:54.093468 31159 raft_consensus.cc:2454] T 6a96965691644db8b9461d146536a026 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:12:54.093914 28196 leader_election.cc:302] T 6a96965691644db8b9461d146536a026 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, b00f244c69df401997ea99d492218a98; no voters:
I20240503 08:12:54.094141 30652 raft_consensus.cc:2781] T 6a96965691644db8b9461d146536a026 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:54.094288 30652 raft_consensus.cc:686] T 6a96965691644db8b9461d146536a026 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Becoming Leader. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Running, Role: LEADER
I20240503 08:12:54.094532 30652 consensus_queue.cc:237] T 6a96965691644db8b9461d146536a026 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:54.099853 27856 catalog_manager.cc:5484] T 6a96965691644db8b9461d146536a026 P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: term changed from 0 to 1, leader changed from <none> to aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67). New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:12:54.160418 27856 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:54.167249 31141 tablet_service.cc:1467] Processing CreateTablet for tablet fb6bd3d9d0b2480e8a91546e38cd72c7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3f7b1d0309e74be0b23aff6bcd1776e5]), partition=
I20240503 08:12:54.167511 30874 tablet_service.cc:1467] Processing CreateTablet for tablet fb6bd3d9d0b2480e8a91546e38cd72c7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3f7b1d0309e74be0b23aff6bcd1776e5]), partition=
I20240503 08:12:54.167817 31141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb6bd3d9d0b2480e8a91546e38cd72c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:54.167888 30874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb6bd3d9d0b2480e8a91546e38cd72c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:54.168295 30741 tablet_service.cc:1467] Processing CreateTablet for tablet fb6bd3d9d0b2480e8a91546e38cd72c7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3f7b1d0309e74be0b23aff6bcd1776e5]), partition=
I20240503 08:12:54.168642 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb6bd3d9d0b2480e8a91546e38cd72c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:54.173269 32143 tablet_bootstrap.cc:492] T fb6bd3d9d0b2480e8a91546e38cd72c7 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:54.174681 32143 tablet_bootstrap.cc:654] T fb6bd3d9d0b2480e8a91546e38cd72c7 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:54.176726 32165 tablet_bootstrap.cc:492] T fb6bd3d9d0b2480e8a91546e38cd72c7 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:54.176887 32143 tablet_bootstrap.cc:492] T fb6bd3d9d0b2480e8a91546e38cd72c7 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:54.177019 32143 ts_tablet_manager.cc:1367] T fb6bd3d9d0b2480e8a91546e38cd72c7 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:12:54.177733 32143 raft_consensus.cc:348] T fb6bd3d9d0b2480e8a91546e38cd72c7 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:54.177944 32143 raft_consensus.cc:374] T fb6bd3d9d0b2480e8a91546e38cd72c7 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:54.178025 32143 raft_consensus.cc:729] T fb6bd3d9d0b2480e8a91546e38cd72c7 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:54.178258 32165 tablet_bootstrap.cc:654] T fb6bd3d9d0b2480e8a91546e38cd72c7 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:54.178237 32143 consensus_queue.cc:260] T fb6bd3d9d0b2480e8a91546e38cd72c7 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:54.178782 32143 ts_tablet_manager.cc:1398] T fb6bd3d9d0b2480e8a91546e38cd72c7 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:54.180774 32165 tablet_bootstrap.cc:492] T fb6bd3d9d0b2480e8a91546e38cd72c7 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:54.180907 32165 ts_tablet_manager.cc:1367] T fb6bd3d9d0b2480e8a91546e38cd72c7 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:12:54.181473 32165 raft_consensus.cc:348] T fb6bd3d9d0b2480e8a91546e38cd72c7 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:54.181749 32165 raft_consensus.cc:374] T fb6bd3d9d0b2480e8a91546e38cd72c7 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:54.181908 32165 raft_consensus.cc:729] T fb6bd3d9d0b2480e8a91546e38cd72c7 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:54.182168 32165 consensus_queue.cc:260] T fb6bd3d9d0b2480e8a91546e38cd72c7 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:54.182740 32165 ts_tablet_manager.cc:1398] T fb6bd3d9d0b2480e8a91546e38cd72c7 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:54.184396 31601 consensus_queue.cc:1035] T fe4cf85b2a5d46fea0bb09ccda5e6351 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:54.198344 32164 tablet_bootstrap.cc:492] T fb6bd3d9d0b2480e8a91546e38cd72c7 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:54.201485 32164 tablet_bootstrap.cc:654] T fb6bd3d9d0b2480e8a91546e38cd72c7 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:54.201776 32168 consensus_queue.cc:1035] T fe4cf85b2a5d46fea0bb09ccda5e6351 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:54.232736 32164 tablet_bootstrap.cc:492] T fb6bd3d9d0b2480e8a91546e38cd72c7 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:54.232898 32164 ts_tablet_manager.cc:1367] T fb6bd3d9d0b2480e8a91546e38cd72c7 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.035s user 0.008s sys 0.022s
I20240503 08:12:54.233637 32164 raft_consensus.cc:348] T fb6bd3d9d0b2480e8a91546e38cd72c7 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:54.233883 32164 raft_consensus.cc:374] T fb6bd3d9d0b2480e8a91546e38cd72c7 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:54.233969 32164 raft_consensus.cc:729] T fb6bd3d9d0b2480e8a91546e38cd72c7 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:54.234179 32164 consensus_queue.cc:260] T fb6bd3d9d0b2480e8a91546e38cd72c7 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:54.234900 32164 ts_tablet_manager.cc:1398] T fb6bd3d9d0b2480e8a91546e38cd72c7 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:54.266170 32112 raft_consensus.cc:482] T fb6bd3d9d0b2480e8a91546e38cd72c7 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:54.266317 32112 raft_consensus.cc:504] T fb6bd3d9d0b2480e8a91546e38cd72c7 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:54.266891 32112 leader_election.cc:288] T fb6bd3d9d0b2480e8a91546e38cd72c7 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:12:54.267204 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6bd3d9d0b2480e8a91546e38cd72c7" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:12:54.267279 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6bd3d9d0b2480e8a91546e38cd72c7" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:12:54.267424 31159 raft_consensus.cc:2454] T fb6bd3d9d0b2480e8a91546e38cd72c7 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:12:54.267679 30761 raft_consensus.cc:2454] T fb6bd3d9d0b2480e8a91546e38cd72c7 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:12:54.267897 30827 leader_election.cc:302] T fb6bd3d9d0b2480e8a91546e38cd72c7 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:54.268122 32112 raft_consensus.cc:2781] T fb6bd3d9d0b2480e8a91546e38cd72c7 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:54.268252 32112 raft_consensus.cc:482] T fb6bd3d9d0b2480e8a91546e38cd72c7 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:54.268353 32112 raft_consensus.cc:3037] T fb6bd3d9d0b2480e8a91546e38cd72c7 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:54.270661 32112 raft_consensus.cc:504] T fb6bd3d9d0b2480e8a91546e38cd72c7 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:54.271139 32112 leader_election.cc:288] T fb6bd3d9d0b2480e8a91546e38cd72c7 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Requested vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:12:54.271497 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6bd3d9d0b2480e8a91546e38cd72c7" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:12:54.271658 31159 raft_consensus.cc:3037] T fb6bd3d9d0b2480e8a91546e38cd72c7 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:54.271572 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6bd3d9d0b2480e8a91546e38cd72c7" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:12:54.271842 30761 raft_consensus.cc:3037] T fb6bd3d9d0b2480e8a91546e38cd72c7 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:54.273819 31159 raft_consensus.cc:2454] T fb6bd3d9d0b2480e8a91546e38cd72c7 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:12:54.274179 30827 leader_election.cc:302] T fb6bd3d9d0b2480e8a91546e38cd72c7 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:54.274345 30761 raft_consensus.cc:2454] T fb6bd3d9d0b2480e8a91546e38cd72c7 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:12:54.274483 32112 raft_consensus.cc:2781] T fb6bd3d9d0b2480e8a91546e38cd72c7 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:54.274883 32112 raft_consensus.cc:686] T fb6bd3d9d0b2480e8a91546e38cd72c7 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Becoming Leader. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Running, Role: LEADER
I20240503 08:12:54.275171 32112 consensus_queue.cc:237] T fb6bd3d9d0b2480e8a91546e38cd72c7 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:54.278648 27856 catalog_manager.cc:5484] T fb6bd3d9d0b2480e8a91546e38cd72c7 P ce1496a47d084365ad786cd09e472a2c reported cstate change: term changed from 0 to 1, leader changed from <none> to ce1496a47d084365ad786cd09e472a2c (127.20.230.70). New cstate: current_term: 1 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:12:54.404736 27856 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:54.410892 31161 raft_consensus.cc:1261] T fb6bd3d9d0b2480e8a91546e38cd72c7 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:54.411037 30761 raft_consensus.cc:1261] T fb6bd3d9d0b2480e8a91546e38cd72c7 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:54.411639 31992 consensus_queue.cc:1035] T fb6bd3d9d0b2480e8a91546e38cd72c7 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:54.412074 32112 consensus_queue.cc:1035] T fb6bd3d9d0b2480e8a91546e38cd72c7 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:54.416296 32154 tablet.cc:1722] T fb6bd3d9d0b2480e8a91546e38cd72c7 P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:54.416776 32154 tablet.cc:1620] T fb6bd3d9d0b2480e8a91546e38cd72c7 P ce1496a47d084365ad786cd09e472a2c: MemRowSet was empty: no flush needed.
I20240503 08:12:54.418152 27855 catalog_manager.cc:5576] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725) does not have the latest schema for tablet fb6bd3d9d0b2480e8a91546e38cd72c7 (table rebalancer_dynamic_table [id=3f7b1d0309e74be0b23aff6bcd1776e5]). Expected version 1 got 0
I20240503 08:12:54.419443 32148 tablet.cc:1722] T fb6bd3d9d0b2480e8a91546e38cd72c7 P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:54.419950 32148 tablet.cc:1620] T fb6bd3d9d0b2480e8a91546e38cd72c7 P 6d1d7de19459435fbde1999e3dfd226b: MemRowSet was empty: no flush needed.
I20240503 08:12:54.420044 27856 catalog_manager.cc:5576] TS b00f244c69df401997ea99d492218a98 (127.20.230.72:40893) does not have the latest schema for tablet fb6bd3d9d0b2480e8a91546e38cd72c7 (table rebalancer_dynamic_table [id=3f7b1d0309e74be0b23aff6bcd1776e5]). Expected version 1 got 0
I20240503 08:12:54.422770 32179 tablet.cc:1722] T fb6bd3d9d0b2480e8a91546e38cd72c7 P b00f244c69df401997ea99d492218a98: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:54.423794 32179 tablet.cc:1620] T fb6bd3d9d0b2480e8a91546e38cd72c7 P b00f244c69df401997ea99d492218a98: MemRowSet was empty: no flush needed.
I20240503 08:12:54.425460 27855 catalog_manager.cc:5576] TS b00f244c69df401997ea99d492218a98 (127.20.230.72:40893) does not have the latest schema for tablet fb6bd3d9d0b2480e8a91546e38cd72c7 (table rebalancer_dynamic_table [id=3f7b1d0309e74be0b23aff6bcd1776e5]). Expected version 1 got 0
I20240503 08:12:54.426569 27859 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=3f7b1d0309e74be0b23aff6bcd1776e5] alter complete (version 1)
I20240503 08:12:54.430102 31903 consensus_queue.cc:1035] T 1cac39d5831d4b6887250f7a9d63dcd5 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:54.434391 30652 consensus_queue.cc:1035] T 1cac39d5831d4b6887250f7a9d63dcd5 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:54.531005 27859 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:54.539393 32154 tablet.cc:1722] T fb6bd3d9d0b2480e8a91546e38cd72c7 P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:54.540031 32154 tablet.cc:1620] T fb6bd3d9d0b2480e8a91546e38cd72c7 P ce1496a47d084365ad786cd09e472a2c: MemRowSet was empty: no flush needed.
I20240503 08:12:54.540156 32148 tablet.cc:1722] T fb6bd3d9d0b2480e8a91546e38cd72c7 P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:54.540642 32148 tablet.cc:1620] T fb6bd3d9d0b2480e8a91546e38cd72c7 P 6d1d7de19459435fbde1999e3dfd226b: MemRowSet was empty: no flush needed.
I20240503 08:12:54.540570 32179 tablet.cc:1722] T fb6bd3d9d0b2480e8a91546e38cd72c7 P b00f244c69df401997ea99d492218a98: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:54.540992 32179 tablet.cc:1620] T fb6bd3d9d0b2480e8a91546e38cd72c7 P b00f244c69df401997ea99d492218a98: MemRowSet was empty: no flush needed.
I20240503 08:12:54.549456 27859 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=3f7b1d0309e74be0b23aff6bcd1776e5] alter complete (version 2)
I20240503 08:12:54.588735 30652 consensus_queue.cc:1035] T 6a96965691644db8b9461d146536a026 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:54.594746 32187 consensus_queue.cc:1035] T 6a96965691644db8b9461d146536a026 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:54.653463 27856 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:54.661283 32154 tablet.cc:1722] T fb6bd3d9d0b2480e8a91546e38cd72c7 P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:54.661908 32179 tablet.cc:1722] T fb6bd3d9d0b2480e8a91546e38cd72c7 P b00f244c69df401997ea99d492218a98: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:54.663445 32148 tablet.cc:1722] T fb6bd3d9d0b2480e8a91546e38cd72c7 P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:54.670773 27859 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table_new_name [id=3f7b1d0309e74be0b23aff6bcd1776e5] alter complete (version 3)
I20240503 08:12:54.704730 27856 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_extra_table_32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:54.713464 30874 tablet_service.cc:1467] Processing CreateTablet for tablet 1fcc779a649b461db4d14b5cd8b17744 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6e8613a79fc542daa3011dfac68e972f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:54.713935 30874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fcc779a649b461db4d14b5cd8b17744. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:54.714880 30741 tablet_service.cc:1467] Processing CreateTablet for tablet 1fcc779a649b461db4d14b5cd8b17744 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6e8613a79fc542daa3011dfac68e972f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:54.715292 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fcc779a649b461db4d14b5cd8b17744. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:54.717432 28111 tablet_service.cc:1467] Processing CreateTablet for tablet 1fcc779a649b461db4d14b5cd8b17744 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6e8613a79fc542daa3011dfac68e972f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:54.717876 28111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fcc779a649b461db4d14b5cd8b17744. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:54.719066 31141 tablet_service.cc:1467] Processing CreateTablet for tablet 82591d648f3f4a55b32ed8e09490953d (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6e8613a79fc542daa3011dfac68e972f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:54.719449 31141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82591d648f3f4a55b32ed8e09490953d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:54.722465 28243 tablet_service.cc:1467] Processing CreateTablet for tablet f9b4f775b9914bdc98b9b582072cb1ab (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6e8613a79fc542daa3011dfac68e972f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:54.722759 30742 tablet_service.cc:1467] Processing CreateTablet for tablet f9b4f775b9914bdc98b9b582072cb1ab (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6e8613a79fc542daa3011dfac68e972f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:54.722885 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9b4f775b9914bdc98b9b582072cb1ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:54.722932 31007 tablet_service.cc:1467] Processing CreateTablet for tablet f9b4f775b9914bdc98b9b582072cb1ab (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6e8613a79fc542daa3011dfac68e972f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:54.723114 30742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9b4f775b9914bdc98b9b582072cb1ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:54.723330 31007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9b4f775b9914bdc98b9b582072cb1ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:54.726123 31008 tablet_service.cc:1467] Processing CreateTablet for tablet 82591d648f3f4a55b32ed8e09490953d (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6e8613a79fc542daa3011dfac68e972f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:54.726480 31008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82591d648f3f4a55b32ed8e09490953d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:54.727288 32164 tablet_bootstrap.cc:492] T 1fcc779a649b461db4d14b5cd8b17744 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:54.728835 32164 tablet_bootstrap.cc:654] T 1fcc779a649b461db4d14b5cd8b17744 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:54.730368 28372 tablet_service.cc:1467] Processing CreateTablet for tablet 82591d648f3f4a55b32ed8e09490953d (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6e8613a79fc542daa3011dfac68e972f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:54.730778 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82591d648f3f4a55b32ed8e09490953d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:54.741194 32189 tablet_bootstrap.cc:492] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:54.743436 32189 tablet_bootstrap.cc:654] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:54.747467 32193 tablet_bootstrap.cc:492] T 82591d648f3f4a55b32ed8e09490953d P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:54.749042 32193 tablet_bootstrap.cc:654] T 82591d648f3f4a55b32ed8e09490953d P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:54.749621 32195 tablet_bootstrap.cc:492] T f9b4f775b9914bdc98b9b582072cb1ab P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:54.749845 32164 tablet_bootstrap.cc:492] T 1fcc779a649b461db4d14b5cd8b17744 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:54.750028 32164 ts_tablet_manager.cc:1367] T 1fcc779a649b461db4d14b5cd8b17744 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.004s
I20240503 08:12:54.750743 32164 raft_consensus.cc:348] T 1fcc779a649b461db4d14b5cd8b17744 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:54.750984 32164 raft_consensus.cc:374] T 1fcc779a649b461db4d14b5cd8b17744 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:54.751052 32164 raft_consensus.cc:729] T 1fcc779a649b461db4d14b5cd8b17744 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:54.751327 32164 consensus_queue.cc:260] T 1fcc779a649b461db4d14b5cd8b17744 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:54.753437 32194 tablet_bootstrap.cc:492] T 82591d648f3f4a55b32ed8e09490953d P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:54.754034 32192 tablet_bootstrap.cc:492] T f9b4f775b9914bdc98b9b582072cb1ab P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:54.756906 32195 tablet_bootstrap.cc:654] T f9b4f775b9914bdc98b9b582072cb1ab P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:54.757898 32193 tablet_bootstrap.cc:492] T 82591d648f3f4a55b32ed8e09490953d P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:54.757958 32164 ts_tablet_manager.cc:1398] T 1fcc779a649b461db4d14b5cd8b17744 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 08:12:54.758144 32193 ts_tablet_manager.cc:1367] T 82591d648f3f4a55b32ed8e09490953d P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.001s
I20240503 08:12:54.759624 32193 raft_consensus.cc:348] T 82591d648f3f4a55b32ed8e09490953d P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:54.759913 32193 raft_consensus.cc:374] T 82591d648f3f4a55b32ed8e09490953d P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:54.760064 32193 raft_consensus.cc:729] T 82591d648f3f4a55b32ed8e09490953d P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:54.760653 32193 consensus_queue.cc:260] T 82591d648f3f4a55b32ed8e09490953d P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:54.760985 32192 tablet_bootstrap.cc:654] T f9b4f775b9914bdc98b9b582072cb1ab P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:54.762543 32193 ts_tablet_manager.cc:1398] T 82591d648f3f4a55b32ed8e09490953d P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 08:12:54.763746 32194 tablet_bootstrap.cc:654] T 82591d648f3f4a55b32ed8e09490953d P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:54.764122 32189 tablet_bootstrap.cc:492] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:54.764317 32189 ts_tablet_manager.cc:1367] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.011s
I20240503 08:12:54.764439 32190 tablet_bootstrap.cc:492] T 1fcc779a649b461db4d14b5cd8b17744 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:54.765130 32189 raft_consensus.cc:348] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:54.765385 32189 raft_consensus.cc:374] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:54.765496 32189 raft_consensus.cc:729] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:54.767037 32189 consensus_queue.cc:260] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:54.767652 32189 ts_tablet_manager.cc:1398] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 08:12:54.766817 32190 tablet_bootstrap.cc:654] T 1fcc779a649b461db4d14b5cd8b17744 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:54.768524 32189 tablet_bootstrap.cc:492] T 1fcc779a649b461db4d14b5cd8b17744 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:54.769030 32195 tablet_bootstrap.cc:492] T f9b4f775b9914bdc98b9b582072cb1ab P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:54.769228 32195 ts_tablet_manager.cc:1367] T f9b4f775b9914bdc98b9b582072cb1ab P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.002s
I20240503 08:12:54.770112 32195 raft_consensus.cc:348] T f9b4f775b9914bdc98b9b582072cb1ab P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:54.770401 32195 raft_consensus.cc:374] T f9b4f775b9914bdc98b9b582072cb1ab P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:54.770522 32195 raft_consensus.cc:729] T f9b4f775b9914bdc98b9b582072cb1ab P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:54.770800 32195 consensus_queue.cc:260] T f9b4f775b9914bdc98b9b582072cb1ab P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:54.772148 27859 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:54.772351 27859 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:54.774623 32195 ts_tablet_manager.cc:1398] T f9b4f775b9914bdc98b9b582072cb1ab P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:12:54.776775 32189 tablet_bootstrap.cc:654] T 1fcc779a649b461db4d14b5cd8b17744 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:54.778864 32190 tablet_bootstrap.cc:492] T 1fcc779a649b461db4d14b5cd8b17744 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:54.779070 32190 ts_tablet_manager.cc:1367] T 1fcc779a649b461db4d14b5cd8b17744 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.000s
I20240503 08:12:54.779553 27859 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet fb6bd3d9d0b2480e8a91546e38cd72c7
I20240503 08:12:54.780052 32190 raft_consensus.cc:348] T 1fcc779a649b461db4d14b5cd8b17744 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:54.780218 31141 tablet_service.cc:1514] Processing DeleteTablet for tablet fb6bd3d9d0b2480e8a91546e38cd72c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:54 UTC) from {username='slave'} at 127.0.0.1:45814
I20240503 08:12:54.780373 32190 raft_consensus.cc:374] T 1fcc779a649b461db4d14b5cd8b17744 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:54.780539 32192 tablet_bootstrap.cc:492] T f9b4f775b9914bdc98b9b582072cb1ab P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:54.780547 32190 raft_consensus.cc:729] T 1fcc779a649b461db4d14b5cd8b17744 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:54.780722 32192 ts_tablet_manager.cc:1367] T f9b4f775b9914bdc98b9b582072cb1ab P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.000s
I20240503 08:12:54.780726 30742 tablet_service.cc:1514] Processing DeleteTablet for tablet fb6bd3d9d0b2480e8a91546e38cd72c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:54 UTC) from {username='slave'} at 127.0.0.1:46794
I20240503 08:12:54.780908 32190 consensus_queue.cc:260] T 1fcc779a649b461db4d14b5cd8b17744 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:54.781059 30874 tablet_service.cc:1514] Processing DeleteTablet for tablet fb6bd3d9d0b2480e8a91546e38cd72c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:54 UTC) from {username='slave'} at 127.0.0.1:34426
I20240503 08:12:54.781558 32192 raft_consensus.cc:348] T f9b4f775b9914bdc98b9b582072cb1ab P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:54.781816 32192 raft_consensus.cc:374] T f9b4f775b9914bdc98b9b582072cb1ab P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:54.781926 32192 raft_consensus.cc:729] T f9b4f775b9914bdc98b9b582072cb1ab P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:54.782199 32192 consensus_queue.cc:260] T f9b4f775b9914bdc98b9b582072cb1ab P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:54.783694 32190 ts_tablet_manager.cc:1398] T 1fcc779a649b461db4d14b5cd8b17744 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:54.784231 32203 tablet_replica.cc:330] T fb6bd3d9d0b2480e8a91546e38cd72c7 P b00f244c69df401997ea99d492218a98: stopping tablet replica
I20240503 08:12:54.784473 32203 raft_consensus.cc:2229] T fb6bd3d9d0b2480e8a91546e38cd72c7 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:54.784741 32203 raft_consensus.cc:2258] T fb6bd3d9d0b2480e8a91546e38cd72c7 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:54.786301 32203 ts_tablet_manager.cc:1875] T fb6bd3d9d0b2480e8a91546e38cd72c7 P b00f244c69df401997ea99d492218a98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:54.788077 32192 ts_tablet_manager.cc:1398] T f9b4f775b9914bdc98b9b582072cb1ab P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:12:54.789002 32192 tablet_bootstrap.cc:492] T 82591d648f3f4a55b32ed8e09490953d P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:54.791041 32192 tablet_bootstrap.cc:654] T 82591d648f3f4a55b32ed8e09490953d P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:54.791612 32204 tablet_replica.cc:330] T fb6bd3d9d0b2480e8a91546e38cd72c7 P 6d1d7de19459435fbde1999e3dfd226b: stopping tablet replica
I20240503 08:12:54.791896 32194 tablet_bootstrap.cc:492] T 82591d648f3f4a55b32ed8e09490953d P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:54.791957 32204 raft_consensus.cc:2229] T fb6bd3d9d0b2480e8a91546e38cd72c7 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:54.792105 32194 ts_tablet_manager.cc:1367] T 82591d648f3f4a55b32ed8e09490953d P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.039s user 0.005s sys 0.000s
I20240503 08:12:54.792258 32204 raft_consensus.cc:2258] T fb6bd3d9d0b2480e8a91546e38cd72c7 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:54.793136 32204 ts_tablet_manager.cc:1875] T fb6bd3d9d0b2480e8a91546e38cd72c7 P 6d1d7de19459435fbde1999e3dfd226b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:54.793299 32205 tablet_replica.cc:330] T fb6bd3d9d0b2480e8a91546e38cd72c7 P ce1496a47d084365ad786cd09e472a2c: stopping tablet replica
I20240503 08:12:54.794687 32194 raft_consensus.cc:348] T 82591d648f3f4a55b32ed8e09490953d P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:54.794957 32205 raft_consensus.cc:2229] T fb6bd3d9d0b2480e8a91546e38cd72c7 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:54.795040 32194 raft_consensus.cc:374] T 82591d648f3f4a55b32ed8e09490953d P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:54.795188 32194 raft_consensus.cc:729] T 82591d648f3f4a55b32ed8e09490953d P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:54.795527 32194 consensus_queue.cc:260] T 82591d648f3f4a55b32ed8e09490953d P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:54.795861 32205 raft_consensus.cc:2258] T fb6bd3d9d0b2480e8a91546e38cd72c7 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:54.797541 32205 ts_tablet_manager.cc:1875] T fb6bd3d9d0b2480e8a91546e38cd72c7 P ce1496a47d084365ad786cd09e472a2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:54.797597 32194 ts_tablet_manager.cc:1398] T 82591d648f3f4a55b32ed8e09490953d P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:12:54.798825 32157 raft_consensus.cc:482] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:54.798997 32157 raft_consensus.cc:504] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:54.799633 32204 ts_tablet_manager.cc:1888] T fb6bd3d9d0b2480e8a91546e38cd72c7 P 6d1d7de19459435fbde1999e3dfd226b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:54.799765 32204 log.cc:1196] T fb6bd3d9d0b2480e8a91546e38cd72c7 P 6d1d7de19459435fbde1999e3dfd226b: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal/wals/fb6bd3d9d0b2480e8a91546e38cd72c7
I20240503 08:12:54.799860 32157 leader_election.cc:288] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:54.800015 32189 tablet_bootstrap.cc:492] T 1fcc779a649b461db4d14b5cd8b17744 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:54.800159 32189 ts_tablet_manager.cc:1367] T 1fcc779a649b461db4d14b5cd8b17744 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.000s
I20240503 08:12:54.800185 32204 ts_tablet_manager.cc:1909] T fb6bd3d9d0b2480e8a91546e38cd72c7 P 6d1d7de19459435fbde1999e3dfd226b: Deleting consensus metadata
I20240503 08:12:54.800920 32189 raft_consensus.cc:348] T 1fcc779a649b461db4d14b5cd8b17744 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:54.801098 27848 catalog_manager.cc:4830] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): tablet fb6bd3d9d0b2480e8a91546e38cd72c7 (table rebalancer_dynamic_table_new_name [id=3f7b1d0309e74be0b23aff6bcd1776e5]) successfully deleted
I20240503 08:12:54.803858 32203 ts_tablet_manager.cc:1888] T fb6bd3d9d0b2480e8a91546e38cd72c7 P b00f244c69df401997ea99d492218a98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:54.804045 32203 log.cc:1196] T fb6bd3d9d0b2480e8a91546e38cd72c7 P b00f244c69df401997ea99d492218a98: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal/wals/fb6bd3d9d0b2480e8a91546e38cd72c7
I20240503 08:12:54.805338 32203 ts_tablet_manager.cc:1909] T fb6bd3d9d0b2480e8a91546e38cd72c7 P b00f244c69df401997ea99d492218a98: Deleting consensus metadata
I20240503 08:12:54.805848 32189 raft_consensus.cc:374] T 1fcc779a649b461db4d14b5cd8b17744 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:54.805989 32189 raft_consensus.cc:729] T 1fcc779a649b461db4d14b5cd8b17744 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:54.806191 32205 ts_tablet_manager.cc:1888] T fb6bd3d9d0b2480e8a91546e38cd72c7 P ce1496a47d084365ad786cd09e472a2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:54.806253 32189 consensus_queue.cc:260] T 1fcc779a649b461db4d14b5cd8b17744 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:54.806375 32205 log.cc:1196] T fb6bd3d9d0b2480e8a91546e38cd72c7 P ce1496a47d084365ad786cd09e472a2c: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal/wals/fb6bd3d9d0b2480e8a91546e38cd72c7
I20240503 08:12:54.806527 27846 catalog_manager.cc:4830] TS b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): tablet fb6bd3d9d0b2480e8a91546e38cd72c7 (table rebalancer_dynamic_table_new_name [id=3f7b1d0309e74be0b23aff6bcd1776e5]) successfully deleted
I20240503 08:12:54.806968 32189 ts_tablet_manager.cc:1398] T 1fcc779a649b461db4d14b5cd8b17744 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240503 08:12:54.807540 32205 ts_tablet_manager.cc:1909] T fb6bd3d9d0b2480e8a91546e38cd72c7 P ce1496a47d084365ad786cd09e472a2c: Deleting consensus metadata
I20240503 08:12:54.808368 27846 catalog_manager.cc:4830] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): tablet fb6bd3d9d0b2480e8a91546e38cd72c7 (table rebalancer_dynamic_table_new_name [id=3f7b1d0309e74be0b23aff6bcd1776e5]) successfully deleted
I20240503 08:12:54.809521 32192 tablet_bootstrap.cc:492] T 82591d648f3f4a55b32ed8e09490953d P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:54.809789 32192 ts_tablet_manager.cc:1367] T 82591d648f3f4a55b32ed8e09490953d P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.000s
I20240503 08:12:54.809787 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9b4f775b9914bdc98b9b582072cb1ab" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:54.810084 31027 raft_consensus.cc:2454] T f9b4f775b9914bdc98b9b582072cb1ab P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:12:54.810678 32192 raft_consensus.cc:348] T 82591d648f3f4a55b32ed8e09490953d P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:54.810912 32192 raft_consensus.cc:374] T 82591d648f3f4a55b32ed8e09490953d P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:54.811003 32192 raft_consensus.cc:729] T 82591d648f3f4a55b32ed8e09490953d P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:54.811249 32192 consensus_queue.cc:260] T 82591d648f3f4a55b32ed8e09490953d P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:54.811820 32192 ts_tablet_manager.cc:1398] T 82591d648f3f4a55b32ed8e09490953d P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:12:54.812222 30695 leader_election.cc:302] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:54.812475 32157 raft_consensus.cc:2781] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:54.812616 32157 raft_consensus.cc:482] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:54.812714 32157 raft_consensus.cc:3037] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:54.815562 32157 raft_consensus.cc:504] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:54.816272 32157 leader_election.cc:288] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:54.816639 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9b4f775b9914bdc98b9b582072cb1ab" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:12:54.816804 31027 raft_consensus.cc:3037] T f9b4f775b9914bdc98b9b582072cb1ab P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:54.817520 28261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9b4f775b9914bdc98b9b582072cb1ab" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:12:54.817739 28261 raft_consensus.cc:2454] T f9b4f775b9914bdc98b9b582072cb1ab P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:12:54.818480 28261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9b4f775b9914bdc98b9b582072cb1ab" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:12:54.818657 28261 raft_consensus.cc:3037] T f9b4f775b9914bdc98b9b582072cb1ab P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:54.819111 31027 raft_consensus.cc:2454] T f9b4f775b9914bdc98b9b582072cb1ab P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:12:54.819428 30695 leader_election.cc:302] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:54.819698 32157 raft_consensus.cc:2781] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:54.820079 32157 raft_consensus.cc:686] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Becoming Leader. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Running, Role: LEADER
I20240503 08:12:54.820861 28261 raft_consensus.cc:2454] T f9b4f775b9914bdc98b9b582072cb1ab P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:12:54.820356 32157 consensus_queue.cc:237] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:54.825301 27855 catalog_manager.cc:5484] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69). New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:12:54.889026 32157 raft_consensus.cc:482] T 1fcc779a649b461db4d14b5cd8b17744 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:54.889209 32157 raft_consensus.cc:504] T 1fcc779a649b461db4d14b5cd8b17744 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:54.889950 27855 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:54.890172 32157 leader_election.cc:288] T 1fcc779a649b461db4d14b5cd8b17744 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:54.890583 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fcc779a649b461db4d14b5cd8b17744" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:12:54.890818 30895 raft_consensus.cc:2454] T 1fcc779a649b461db4d14b5cd8b17744 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:12:54.891207 30694 leader_election.cc:302] T 1fcc779a649b461db4d14b5cd8b17744 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:54.891484 32157 raft_consensus.cc:2781] T 1fcc779a649b461db4d14b5cd8b17744 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:54.891571 32157 raft_consensus.cc:482] T 1fcc779a649b461db4d14b5cd8b17744 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:54.891662 32157 raft_consensus.cc:3037] T 1fcc779a649b461db4d14b5cd8b17744 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:54.894042 32157 raft_consensus.cc:504] T 1fcc779a649b461db4d14b5cd8b17744 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:54.894707 32157 leader_election.cc:288] T 1fcc779a649b461db4d14b5cd8b17744 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Requested vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:54.895395 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fcc779a649b461db4d14b5cd8b17744" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:12:54.895610 30895 raft_consensus.cc:3037] T 1fcc779a649b461db4d14b5cd8b17744 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:54.898167 30895 raft_consensus.cc:2454] T 1fcc779a649b461db4d14b5cd8b17744 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:12:54.898613 30694 leader_election.cc:302] T 1fcc779a649b461db4d14b5cd8b17744 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:54.898896 32157 raft_consensus.cc:2781] T 1fcc779a649b461db4d14b5cd8b17744 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:54.899083 32157 raft_consensus.cc:686] T 1fcc779a649b461db4d14b5cd8b17744 P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Becoming Leader. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Running, Role: LEADER
I20240503 08:12:54.899320 31007 tablet_service.cc:1467] Processing CreateTablet for tablet 2319c1f136744cbcb9f1277f1fdf3214 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7475cf6b4b9a418ea69389abe7c9922b]), partition=
I20240503 08:12:54.899340 32157 consensus_queue.cc:237] T 1fcc779a649b461db4d14b5cd8b17744 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:54.899686 31007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2319c1f136744cbcb9f1277f1fdf3214. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:54.899808 30742 tablet_service.cc:1467] Processing CreateTablet for tablet 2319c1f136744cbcb9f1277f1fdf3214 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7475cf6b4b9a418ea69389abe7c9922b]), partition=
I20240503 08:12:54.900147 30742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2319c1f136744cbcb9f1277f1fdf3214. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:54.903406 30874 tablet_service.cc:1467] Processing CreateTablet for tablet 2319c1f136744cbcb9f1277f1fdf3214 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7475cf6b4b9a418ea69389abe7c9922b]), partition=
I20240503 08:12:54.903713 30874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2319c1f136744cbcb9f1277f1fdf3214. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:54.903556 27855 catalog_manager.cc:5484] T 1fcc779a649b461db4d14b5cd8b17744 P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69). New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:12:54.906114 32189 tablet_bootstrap.cc:492] T 2319c1f136744cbcb9f1277f1fdf3214 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:54.907534 32189 tablet_bootstrap.cc:654] T 2319c1f136744cbcb9f1277f1fdf3214 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:54.907467 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fcc779a649b461db4d14b5cd8b17744" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:12:54.907544 28131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fcc779a649b461db4d14b5cd8b17744" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:12:54.907682 28132 raft_consensus.cc:2454] T 1fcc779a649b461db4d14b5cd8b17744 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:12:54.910719 32189 tablet_bootstrap.cc:492] T 2319c1f136744cbcb9f1277f1fdf3214 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:54.910859 32189 ts_tablet_manager.cc:1367] T 2319c1f136744cbcb9f1277f1fdf3214 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240503 08:12:54.912355 32189 raft_consensus.cc:348] T 2319c1f136744cbcb9f1277f1fdf3214 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:54.912621 32189 raft_consensus.cc:374] T 2319c1f136744cbcb9f1277f1fdf3214 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:54.912721 32189 raft_consensus.cc:729] T 2319c1f136744cbcb9f1277f1fdf3214 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:54.912917 32189 consensus_queue.cc:260] T 2319c1f136744cbcb9f1277f1fdf3214 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:54.913436 32164 tablet_bootstrap.cc:492] T 2319c1f136744cbcb9f1277f1fdf3214 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:54.914507 32189 ts_tablet_manager.cc:1398] T 2319c1f136744cbcb9f1277f1fdf3214 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 08:12:54.914848 32164 tablet_bootstrap.cc:654] T 2319c1f136744cbcb9f1277f1fdf3214 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:54.916296 32192 tablet_bootstrap.cc:492] T 2319c1f136744cbcb9f1277f1fdf3214 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:54.917019 32164 tablet_bootstrap.cc:492] T 2319c1f136744cbcb9f1277f1fdf3214 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:54.917140 32164 ts_tablet_manager.cc:1367] T 2319c1f136744cbcb9f1277f1fdf3214 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:12:54.917510 32192 tablet_bootstrap.cc:654] T 2319c1f136744cbcb9f1277f1fdf3214 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:54.917716 32164 raft_consensus.cc:348] T 2319c1f136744cbcb9f1277f1fdf3214 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:54.918184 32164 raft_consensus.cc:374] T 2319c1f136744cbcb9f1277f1fdf3214 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:54.918279 32164 raft_consensus.cc:729] T 2319c1f136744cbcb9f1277f1fdf3214 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:54.918489 32164 consensus_queue.cc:260] T 2319c1f136744cbcb9f1277f1fdf3214 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:54.919013 32164 ts_tablet_manager.cc:1398] T 2319c1f136744cbcb9f1277f1fdf3214 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:54.921212 32192 tablet_bootstrap.cc:492] T 2319c1f136744cbcb9f1277f1fdf3214 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:54.922325 32192 ts_tablet_manager.cc:1367] T 2319c1f136744cbcb9f1277f1fdf3214 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240503 08:12:54.923103 32192 raft_consensus.cc:348] T 2319c1f136744cbcb9f1277f1fdf3214 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:54.923245 32192 raft_consensus.cc:374] T 2319c1f136744cbcb9f1277f1fdf3214 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:54.923329 32192 raft_consensus.cc:729] T 2319c1f136744cbcb9f1277f1fdf3214 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:54.923502 32192 consensus_queue.cc:260] T 2319c1f136744cbcb9f1277f1fdf3214 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:54.924700 32192 ts_tablet_manager.cc:1398] T 2319c1f136744cbcb9f1277f1fdf3214 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:55.024722 31999 raft_consensus.cc:482] T 82591d648f3f4a55b32ed8e09490953d P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:55.024935 31999 raft_consensus.cc:504] T 82591d648f3f4a55b32ed8e09490953d P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:55.025107 32218 raft_consensus.cc:482] T 2319c1f136744cbcb9f1277f1fdf3214 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:55.025277 32218 raft_consensus.cc:504] T 2319c1f136744cbcb9f1277f1fdf3214 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:55.025985 31999 leader_election.cc:288] T 82591d648f3f4a55b32ed8e09490953d P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:55.026186 32218 leader_election.cc:288] T 2319c1f136744cbcb9f1277f1fdf3214 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:12:55.026399 31161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82591d648f3f4a55b32ed8e09490953d" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:12:55.026640 31161 raft_consensus.cc:2454] T 82591d648f3f4a55b32ed8e09490953d P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 0.
I20240503 08:12:55.026798 28392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82591d648f3f4a55b32ed8e09490953d" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:55.027000 28392 raft_consensus.cc:2454] T 82591d648f3f4a55b32ed8e09490953d P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 0.
I20240503 08:12:55.027487 30959 leader_election.cc:302] T 82591d648f3f4a55b32ed8e09490953d P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:55.027740 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2319c1f136744cbcb9f1277f1fdf3214" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:12:55.027951 30895 raft_consensus.cc:2454] T 2319c1f136744cbcb9f1277f1fdf3214 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 0.
I20240503 08:12:55.028240 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2319c1f136744cbcb9f1277f1fdf3214" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:12:55.028338 30959 leader_election.cc:302] T 2319c1f136744cbcb9f1277f1fdf3214 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:55.028475 32218 raft_consensus.cc:2781] T 82591d648f3f4a55b32ed8e09490953d P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:55.028468 30761 raft_consensus.cc:2454] T 2319c1f136744cbcb9f1277f1fdf3214 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 0.
I20240503 08:12:55.028592 32218 raft_consensus.cc:482] T 82591d648f3f4a55b32ed8e09490953d P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:55.028596 31999 raft_consensus.cc:2781] T 2319c1f136744cbcb9f1277f1fdf3214 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:55.028695 32218 raft_consensus.cc:3037] T 82591d648f3f4a55b32ed8e09490953d P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:55.028757 31999 raft_consensus.cc:482] T 2319c1f136744cbcb9f1277f1fdf3214 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:55.028856 31999 raft_consensus.cc:3037] T 2319c1f136744cbcb9f1277f1fdf3214 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:55.031211 31999 raft_consensus.cc:504] T 2319c1f136744cbcb9f1277f1fdf3214 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:55.031548 32218 raft_consensus.cc:504] T 82591d648f3f4a55b32ed8e09490953d P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:55.032114 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2319c1f136744cbcb9f1277f1fdf3214" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:12:55.032199 32218 leader_election.cc:288] T 82591d648f3f4a55b32ed8e09490953d P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 election: Requested vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:55.032320 30761 raft_consensus.cc:3037] T 2319c1f136744cbcb9f1277f1fdf3214 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:55.032419 31999 leader_election.cc:288] T 2319c1f136744cbcb9f1277f1fdf3214 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 election: Requested vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:12:55.032452 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2319c1f136744cbcb9f1277f1fdf3214" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:12:55.032624 30895 raft_consensus.cc:3037] T 2319c1f136744cbcb9f1277f1fdf3214 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:55.032718 31160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82591d648f3f4a55b32ed8e09490953d" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:12:55.032946 31160 raft_consensus.cc:3037] T 82591d648f3f4a55b32ed8e09490953d P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:55.034204 28392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82591d648f3f4a55b32ed8e09490953d" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:12:55.034343 28392 raft_consensus.cc:3037] T 82591d648f3f4a55b32ed8e09490953d P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:55.035228 30761 raft_consensus.cc:2454] T 2319c1f136744cbcb9f1277f1fdf3214 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 1.
I20240503 08:12:55.035260 30895 raft_consensus.cc:2454] T 2319c1f136744cbcb9f1277f1fdf3214 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 1.
I20240503 08:12:55.035761 30959 leader_election.cc:302] T 2319c1f136744cbcb9f1277f1fdf3214 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:55.036548 31999 raft_consensus.cc:2781] T 2319c1f136744cbcb9f1277f1fdf3214 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:55.036597 28392 raft_consensus.cc:2454] T 82591d648f3f4a55b32ed8e09490953d P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 1.
I20240503 08:12:55.036690 31999 raft_consensus.cc:686] T 2319c1f136744cbcb9f1277f1fdf3214 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 LEADER]: Becoming Leader. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Running, Role: LEADER
I20240503 08:12:55.037040 30962 leader_election.cc:302] T 82591d648f3f4a55b32ed8e09490953d P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:55.037293 32218 raft_consensus.cc:2781] T 82591d648f3f4a55b32ed8e09490953d P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:55.037395 32218 raft_consensus.cc:686] T 82591d648f3f4a55b32ed8e09490953d P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 LEADER]: Becoming Leader. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Running, Role: LEADER
I20240503 08:12:55.037665 32218 consensus_queue.cc:237] T 82591d648f3f4a55b32ed8e09490953d P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:55.040391 31160 raft_consensus.cc:2454] T 82591d648f3f4a55b32ed8e09490953d P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 1.
I20240503 08:12:55.036986 31999 consensus_queue.cc:237] T 2319c1f136744cbcb9f1277f1fdf3214 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:55.052042 27862 catalog_manager.cc:5484] T 2319c1f136744cbcb9f1277f1fdf3214 P ced21ca983b749d0b4d4f6a0e2426bf8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71). New cstate: current_term: 1 leader_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:12:55.052433 27862 catalog_manager.cc:5484] T 82591d648f3f4a55b32ed8e09490953d P ced21ca983b749d0b4d4f6a0e2426bf8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71). New cstate: current_term: 1 leader_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:12:55.173177 27862 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:55.178671 30895 raft_consensus.cc:1261] T 2319c1f136744cbcb9f1277f1fdf3214 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer ced21ca983b749d0b4d4f6a0e2426bf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:55.178791 30761 raft_consensus.cc:1261] T 2319c1f136744cbcb9f1277f1fdf3214 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer ced21ca983b749d0b4d4f6a0e2426bf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:55.179272 31999 consensus_queue.cc:1035] T 2319c1f136744cbcb9f1277f1fdf3214 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:55.179530 32219 consensus_queue.cc:1035] T 2319c1f136744cbcb9f1277f1fdf3214 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:55.183503 27862 catalog_manager.cc:5576] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725) does not have the latest schema for tablet 2319c1f136744cbcb9f1277f1fdf3214 (table rebalancer_dynamic_table [id=7475cf6b4b9a418ea69389abe7c9922b]). Expected version 1 got 0
I20240503 08:12:55.184679 27855 catalog_manager.cc:5576] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747) does not have the latest schema for tablet 2319c1f136744cbcb9f1277f1fdf3214 (table rebalancer_dynamic_table [id=7475cf6b4b9a418ea69389abe7c9922b]). Expected version 1 got 0
I20240503 08:12:55.184793 32227 tablet.cc:1722] T 2319c1f136744cbcb9f1277f1fdf3214 P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:55.185461 32227 tablet.cc:1620] T 2319c1f136744cbcb9f1277f1fdf3214 P 6d1d7de19459435fbde1999e3dfd226b: MemRowSet was empty: no flush needed.
I20240503 08:12:55.185511 32228 tablet.cc:1722] T 2319c1f136744cbcb9f1277f1fdf3214 P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:55.186415 32228 tablet.cc:1620] T 2319c1f136744cbcb9f1277f1fdf3214 P ced21ca983b749d0b4d4f6a0e2426bf8: MemRowSet was empty: no flush needed.
I20240503 08:12:55.187453 27855 catalog_manager.cc:5576] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725) does not have the latest schema for tablet 2319c1f136744cbcb9f1277f1fdf3214 (table rebalancer_dynamic_table [id=7475cf6b4b9a418ea69389abe7c9922b]). Expected version 1 got 0
I20240503 08:12:55.189615 32229 tablet.cc:1722] T 2319c1f136744cbcb9f1277f1fdf3214 P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:55.190687 27855 catalog_manager.cc:5576] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747) does not have the latest schema for tablet 2319c1f136744cbcb9f1277f1fdf3214 (table rebalancer_dynamic_table [id=7475cf6b4b9a418ea69389abe7c9922b]). Expected version 1 got 0
I20240503 08:12:55.191305 32229 tablet.cc:1620] T 2319c1f136744cbcb9f1277f1fdf3214 P ce1496a47d084365ad786cd09e472a2c: MemRowSet was empty: no flush needed.
I20240503 08:12:55.193830 27862 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=7475cf6b4b9a418ea69389abe7c9922b] alter complete (version 1)
I20240503 08:12:55.213949 32157 consensus_queue.cc:1035] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:55.219748 32157 consensus_queue.cc:1035] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:55.298218 27862 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:55.300364 28131 raft_consensus.cc:3037] T 1fcc779a649b461db4d14b5cd8b17744 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:55.303422 32157 consensus_queue.cc:1035] T 1fcc779a649b461db4d14b5cd8b17744 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:55.305950 32228 tablet.cc:1722] T 2319c1f136744cbcb9f1277f1fdf3214 P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:55.306406 32228 tablet.cc:1620] T 2319c1f136744cbcb9f1277f1fdf3214 P ced21ca983b749d0b4d4f6a0e2426bf8: MemRowSet was empty: no flush needed.
I20240503 08:12:55.308022 32229 tablet.cc:1722] T 2319c1f136744cbcb9f1277f1fdf3214 P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:55.308454 32229 tablet.cc:1620] T 2319c1f136744cbcb9f1277f1fdf3214 P ce1496a47d084365ad786cd09e472a2c: MemRowSet was empty: no flush needed.
I20240503 08:12:55.308624 32227 tablet.cc:1722] T 2319c1f136744cbcb9f1277f1fdf3214 P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:55.308991 32227 tablet.cc:1620] T 2319c1f136744cbcb9f1277f1fdf3214 P 6d1d7de19459435fbde1999e3dfd226b: MemRowSet was empty: no flush needed.
I20240503 08:12:55.318521 32157 consensus_queue.cc:1035] T 1fcc779a649b461db4d14b5cd8b17744 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:55.319538 27855 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=7475cf6b4b9a418ea69389abe7c9922b] alter complete (version 2)
I20240503 08:12:55.422871 27862 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:55.429812 32228 tablet.cc:1722] T 2319c1f136744cbcb9f1277f1fdf3214 P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:55.430666 32219 consensus_queue.cc:1035] T 82591d648f3f4a55b32ed8e09490953d P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:55.436017 32227 tablet.cc:1722] T 2319c1f136744cbcb9f1277f1fdf3214 P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:55.437824 32229 tablet.cc:1722] T 2319c1f136744cbcb9f1277f1fdf3214 P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:55.438489 31999 consensus_queue.cc:1035] T 82591d648f3f4a55b32ed8e09490953d P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:55.442188 27855 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table_new_name [id=7475cf6b4b9a418ea69389abe7c9922b] alter complete (version 3)
I20240503 08:12:55.552003 27862 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:55.552237 27862 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:55.555213 27862 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet 2319c1f136744cbcb9f1277f1fdf3214
I20240503 08:12:55.556030 30742 tablet_service.cc:1514] Processing DeleteTablet for tablet 2319c1f136744cbcb9f1277f1fdf3214 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:55 UTC) from {username='slave'} at 127.0.0.1:46794
I20240503 08:12:55.556030 31007 tablet_service.cc:1514] Processing DeleteTablet for tablet 2319c1f136744cbcb9f1277f1fdf3214 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:55 UTC) from {username='slave'} at 127.0.0.1:47020
I20240503 08:12:55.556593 30874 tablet_service.cc:1514] Processing DeleteTablet for tablet 2319c1f136744cbcb9f1277f1fdf3214 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:55 UTC) from {username='slave'} at 127.0.0.1:34426
I20240503 08:12:55.556674 32241 tablet_replica.cc:330] T 2319c1f136744cbcb9f1277f1fdf3214 P ced21ca983b749d0b4d4f6a0e2426bf8: stopping tablet replica
I20240503 08:12:55.556933 32241 raft_consensus.cc:2229] T 2319c1f136744cbcb9f1277f1fdf3214 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:55.557106 32243 tablet_replica.cc:330] T 2319c1f136744cbcb9f1277f1fdf3214 P ce1496a47d084365ad786cd09e472a2c: stopping tablet replica
I20240503 08:12:55.557294 32243 raft_consensus.cc:2229] T 2319c1f136744cbcb9f1277f1fdf3214 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:55.557307 32241 raft_consensus.cc:2258] T 2319c1f136744cbcb9f1277f1fdf3214 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:55.557482 32243 raft_consensus.cc:2258] T 2319c1f136744cbcb9f1277f1fdf3214 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:55.557816 32242 tablet_replica.cc:330] T 2319c1f136744cbcb9f1277f1fdf3214 P 6d1d7de19459435fbde1999e3dfd226b: stopping tablet replica
I20240503 08:12:55.558064 32242 raft_consensus.cc:2229] T 2319c1f136744cbcb9f1277f1fdf3214 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:55.558203 32241 ts_tablet_manager.cc:1875] T 2319c1f136744cbcb9f1277f1fdf3214 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:55.558328 32242 raft_consensus.cc:2258] T 2319c1f136744cbcb9f1277f1fdf3214 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:55.558343 32243 ts_tablet_manager.cc:1875] T 2319c1f136744cbcb9f1277f1fdf3214 P ce1496a47d084365ad786cd09e472a2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:55.559013 32242 ts_tablet_manager.cc:1875] T 2319c1f136744cbcb9f1277f1fdf3214 P 6d1d7de19459435fbde1999e3dfd226b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:55.563920 32242 ts_tablet_manager.cc:1888] T 2319c1f136744cbcb9f1277f1fdf3214 P 6d1d7de19459435fbde1999e3dfd226b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:55.564023 32242 log.cc:1196] T 2319c1f136744cbcb9f1277f1fdf3214 P 6d1d7de19459435fbde1999e3dfd226b: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal/wals/2319c1f136744cbcb9f1277f1fdf3214
I20240503 08:12:55.564232 32241 ts_tablet_manager.cc:1888] T 2319c1f136744cbcb9f1277f1fdf3214 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:55.564232 32243 ts_tablet_manager.cc:1888] T 2319c1f136744cbcb9f1277f1fdf3214 P ce1496a47d084365ad786cd09e472a2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:55.564352 32243 log.cc:1196] T 2319c1f136744cbcb9f1277f1fdf3214 P ce1496a47d084365ad786cd09e472a2c: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal/wals/2319c1f136744cbcb9f1277f1fdf3214
I20240503 08:12:55.564359 32241 log.cc:1196] T 2319c1f136744cbcb9f1277f1fdf3214 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal/wals/2319c1f136744cbcb9f1277f1fdf3214
I20240503 08:12:55.564450 32242 ts_tablet_manager.cc:1909] T 2319c1f136744cbcb9f1277f1fdf3214 P 6d1d7de19459435fbde1999e3dfd226b: Deleting consensus metadata
I20240503 08:12:55.564806 32241 ts_tablet_manager.cc:1909] T 2319c1f136744cbcb9f1277f1fdf3214 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting consensus metadata
I20240503 08:12:55.564817 32243 ts_tablet_manager.cc:1909] T 2319c1f136744cbcb9f1277f1fdf3214 P ce1496a47d084365ad786cd09e472a2c: Deleting consensus metadata
I20240503 08:12:55.565346 27848 catalog_manager.cc:4830] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): tablet 2319c1f136744cbcb9f1277f1fdf3214 (table rebalancer_dynamic_table_new_name [id=7475cf6b4b9a418ea69389abe7c9922b]) successfully deleted
I20240503 08:12:55.565699 27846 catalog_manager.cc:4830] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): tablet 2319c1f136744cbcb9f1277f1fdf3214 (table rebalancer_dynamic_table_new_name [id=7475cf6b4b9a418ea69389abe7c9922b]) successfully deleted
I20240503 08:12:55.565836 27847 catalog_manager.cc:4830] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): tablet 2319c1f136744cbcb9f1277f1fdf3214 (table rebalancer_dynamic_table_new_name [id=7475cf6b4b9a418ea69389abe7c9922b]) successfully deleted
I20240503 08:12:55.621685 27862 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_extra_table_34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:55.632629 27983 tablet_service.cc:1467] Processing CreateTablet for tablet 417cedc3fa5c4a85a7a130005291d64b (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=505d74ce858645248ed95157f710daf1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:55.632615 31007 tablet_service.cc:1467] Processing CreateTablet for tablet 417cedc3fa5c4a85a7a130005291d64b (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=505d74ce858645248ed95157f710daf1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:55.633054 31007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 417cedc3fa5c4a85a7a130005291d64b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:55.633064 27983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 417cedc3fa5c4a85a7a130005291d64b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:55.637840 30742 tablet_service.cc:1467] Processing CreateTablet for tablet 97149ad1d03847a085b2142519135a17 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=505d74ce858645248ed95157f710daf1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:55.638293 30742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97149ad1d03847a085b2142519135a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:55.641412 30741 tablet_service.cc:1467] Processing CreateTablet for tablet fd6f4b81101b46588c3c7711f6bafcfa (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=505d74ce858645248ed95157f710daf1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:55.641786 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd6f4b81101b46588c3c7711f6bafcfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:55.642650 31141 tablet_service.cc:1467] Processing CreateTablet for tablet 97149ad1d03847a085b2142519135a17 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=505d74ce858645248ed95157f710daf1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:55.642778 30874 tablet_service.cc:1467] Processing CreateTablet for tablet 97149ad1d03847a085b2142519135a17 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=505d74ce858645248ed95157f710daf1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:55.643064 31141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97149ad1d03847a085b2142519135a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:55.643157 30874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97149ad1d03847a085b2142519135a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:55.648036 31139 tablet_service.cc:1467] Processing CreateTablet for tablet 417cedc3fa5c4a85a7a130005291d64b (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=505d74ce858645248ed95157f710daf1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:55.648417 31139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 417cedc3fa5c4a85a7a130005291d64b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:55.649947 31140 tablet_service.cc:1467] Processing CreateTablet for tablet fd6f4b81101b46588c3c7711f6bafcfa (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=505d74ce858645248ed95157f710daf1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:55.650329 31140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd6f4b81101b46588c3c7711f6bafcfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:55.653566 32244 tablet_bootstrap.cc:492] T 417cedc3fa5c4a85a7a130005291d64b P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:55.654445 30873 tablet_service.cc:1467] Processing CreateTablet for tablet fd6f4b81101b46588c3c7711f6bafcfa (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=505d74ce858645248ed95157f710daf1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:55.654817 30873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd6f4b81101b46588c3c7711f6bafcfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:55.655475 32244 tablet_bootstrap.cc:654] T 417cedc3fa5c4a85a7a130005291d64b P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:55.657369 27862 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:55.664445 32249 tablet_bootstrap.cc:492] T 97149ad1d03847a085b2142519135a17 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:55.667804 32249 tablet_bootstrap.cc:654] T 97149ad1d03847a085b2142519135a17 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:55.668912 30740 tablet_service.cc:1467] Processing CreateTablet for tablet ff7be440a855440d8a655877d9553d8a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=844951c455764b679f4e60c4b22738ae]), partition=
I20240503 08:12:55.669032 27981 tablet_service.cc:1467] Processing CreateTablet for tablet ff7be440a855440d8a655877d9553d8a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=844951c455764b679f4e60c4b22738ae]), partition=
I20240503 08:12:55.669260 30740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff7be440a855440d8a655877d9553d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:55.669394 27981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff7be440a855440d8a655877d9553d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:55.673025 31007 tablet_service.cc:1467] Processing CreateTablet for tablet ff7be440a855440d8a655877d9553d8a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=844951c455764b679f4e60c4b22738ae]), partition=
I20240503 08:12:55.673344 31007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff7be440a855440d8a655877d9553d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:55.677121 32247 tablet_bootstrap.cc:492] T fd6f4b81101b46588c3c7711f6bafcfa P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:55.678973 32247 tablet_bootstrap.cc:654] T fd6f4b81101b46588c3c7711f6bafcfa P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:55.679530 32246 tablet_bootstrap.cc:492] T 417cedc3fa5c4a85a7a130005291d64b P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:55.682615 32246 tablet_bootstrap.cc:654] T 417cedc3fa5c4a85a7a130005291d64b P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:55.684839 32245 tablet_bootstrap.cc:492] T fd6f4b81101b46588c3c7711f6bafcfa P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:55.686376 32245 tablet_bootstrap.cc:654] T fd6f4b81101b46588c3c7711f6bafcfa P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:55.688542 32249 tablet_bootstrap.cc:492] T 97149ad1d03847a085b2142519135a17 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:55.691769 32247 tablet_bootstrap.cc:492] T fd6f4b81101b46588c3c7711f6bafcfa P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:55.691871 32249 ts_tablet_manager.cc:1367] T 97149ad1d03847a085b2142519135a17 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.004s
I20240503 08:12:55.691978 32247 ts_tablet_manager.cc:1367] T fd6f4b81101b46588c3c7711f6bafcfa P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.000s
I20240503 08:12:55.692310 32245 tablet_bootstrap.cc:492] T fd6f4b81101b46588c3c7711f6bafcfa P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:55.692535 32245 ts_tablet_manager.cc:1367] T fd6f4b81101b46588c3c7711f6bafcfa P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240503 08:12:55.692765 32247 raft_consensus.cc:348] T fd6f4b81101b46588c3c7711f6bafcfa P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:55.692799 32249 raft_consensus.cc:348] T 97149ad1d03847a085b2142519135a17 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:55.692971 32249 raft_consensus.cc:374] T 97149ad1d03847a085b2142519135a17 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:55.693653 32246 tablet_bootstrap.cc:492] T 417cedc3fa5c4a85a7a130005291d64b P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:55.693809 32247 raft_consensus.cc:374] T fd6f4b81101b46588c3c7711f6bafcfa P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:55.693920 32249 raft_consensus.cc:729] T 97149ad1d03847a085b2142519135a17 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:55.693972 32247 raft_consensus.cc:729] T fd6f4b81101b46588c3c7711f6bafcfa P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:55.694201 32247 consensus_queue.cc:260] T fd6f4b81101b46588c3c7711f6bafcfa P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:55.694536 32246 ts_tablet_manager.cc:1367] T 417cedc3fa5c4a85a7a130005291d64b P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.000s
I20240503 08:12:55.694350 32249 consensus_queue.cc:260] T 97149ad1d03847a085b2142519135a17 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:55.693290 32245 raft_consensus.cc:348] T fd6f4b81101b46588c3c7711f6bafcfa P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:55.695014 32245 raft_consensus.cc:374] T fd6f4b81101b46588c3c7711f6bafcfa P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:55.695155 32245 raft_consensus.cc:729] T fd6f4b81101b46588c3c7711f6bafcfa P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:55.695302 32246 raft_consensus.cc:348] T 417cedc3fa5c4a85a7a130005291d64b P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:55.695462 32245 consensus_queue.cc:260] T fd6f4b81101b46588c3c7711f6bafcfa P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:55.696657 32249 ts_tablet_manager.cc:1398] T 97149ad1d03847a085b2142519135a17 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 08:12:55.696848 32247 ts_tablet_manager.cc:1398] T fd6f4b81101b46588c3c7711f6bafcfa P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 08:12:55.697124 32247 tablet_bootstrap.cc:492] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:55.698653 32247 tablet_bootstrap.cc:654] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:55.699841 32244 tablet_bootstrap.cc:492] T 417cedc3fa5c4a85a7a130005291d64b P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:55.700071 32246 raft_consensus.cc:374] T 417cedc3fa5c4a85a7a130005291d64b P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:55.699978 32244 ts_tablet_manager.cc:1367] T 417cedc3fa5c4a85a7a130005291d64b P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.047s user 0.007s sys 0.005s
I20240503 08:12:55.700217 32246 raft_consensus.cc:729] T 417cedc3fa5c4a85a7a130005291d64b P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:55.700505 32246 consensus_queue.cc:260] T 417cedc3fa5c4a85a7a130005291d64b P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:55.701023 32244 raft_consensus.cc:348] T 417cedc3fa5c4a85a7a130005291d64b P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:55.701215 32246 ts_tablet_manager.cc:1398] T 417cedc3fa5c4a85a7a130005291d64b P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240503 08:12:55.701280 32244 raft_consensus.cc:374] T 417cedc3fa5c4a85a7a130005291d64b P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:55.701627 31980 raft_consensus.cc:482] T 417cedc3fa5c4a85a7a130005291d64b P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:55.701812 31980 raft_consensus.cc:504] T 417cedc3fa5c4a85a7a130005291d64b P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:55.702520 31980 leader_election.cc:288] T 417cedc3fa5c4a85a7a130005291d64b P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:55.701442 32244 raft_consensus.cc:729] T 417cedc3fa5c4a85a7a130005291d64b P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:55.703814 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417cedc3fa5c4a85a7a130005291d64b" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:55.701540 32246 tablet_bootstrap.cc:492] T ff7be440a855440d8a655877d9553d8a P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:55.704409 32244 consensus_queue.cc:260] T 417cedc3fa5c4a85a7a130005291d64b P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:55.704721 31027 raft_consensus.cc:2454] T 417cedc3fa5c4a85a7a130005291d64b P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:55.703063 31160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417cedc3fa5c4a85a7a130005291d64b" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:12:55.705164 27937 leader_election.cc:302] T 417cedc3fa5c4a85a7a130005291d64b P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:55.705644 31980 raft_consensus.cc:2781] T 417cedc3fa5c4a85a7a130005291d64b P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:55.705756 31980 raft_consensus.cc:482] T 417cedc3fa5c4a85a7a130005291d64b P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240503 08:12:55.705718 27936 leader_election.cc:341] T 417cedc3fa5c4a85a7a130005291d64b P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:55.705858 31980 raft_consensus.cc:3037] T 417cedc3fa5c4a85a7a130005291d64b P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:55.707058 32249 tablet_bootstrap.cc:492] T 417cedc3fa5c4a85a7a130005291d64b P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:55.707553 32246 tablet_bootstrap.cc:654] T ff7be440a855440d8a655877d9553d8a P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:55.709276 31980 raft_consensus.cc:504] T 417cedc3fa5c4a85a7a130005291d64b P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:55.710790 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417cedc3fa5c4a85a7a130005291d64b" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:12:55.710983 31027 raft_consensus.cc:3037] T 417cedc3fa5c4a85a7a130005291d64b P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:55.711061 32245 ts_tablet_manager.cc:1398] T fd6f4b81101b46588c3c7711f6bafcfa P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.018s user 0.001s sys 0.000s
I20240503 08:12:55.712388 32245 tablet_bootstrap.cc:492] T 97149ad1d03847a085b2142519135a17 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:55.713654 31027 raft_consensus.cc:2454] T 417cedc3fa5c4a85a7a130005291d64b P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:55.714059 32245 tablet_bootstrap.cc:654] T 97149ad1d03847a085b2142519135a17 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:55.714708 27937 leader_election.cc:302] T 417cedc3fa5c4a85a7a130005291d64b P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:55.715371 32210 raft_consensus.cc:2781] T 417cedc3fa5c4a85a7a130005291d64b P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:55.711135 31160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417cedc3fa5c4a85a7a130005291d64b" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:12:55.708648 32249 tablet_bootstrap.cc:654] T 417cedc3fa5c4a85a7a130005291d64b P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
W20240503 08:12:55.716657 27936 leader_election.cc:341] T 417cedc3fa5c4a85a7a130005291d64b P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:55.717789 32210 raft_consensus.cc:686] T 417cedc3fa5c4a85a7a130005291d64b P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Becoming Leader. State: Replica: 833582ff68c94269be364ebd187a75db, State: Running, Role: LEADER
I20240503 08:12:55.718150 32210 consensus_queue.cc:237] T 417cedc3fa5c4a85a7a130005291d64b P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:55.720918 32247 tablet_bootstrap.cc:492] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:55.721150 32247 ts_tablet_manager.cc:1367] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.003s
I20240503 08:12:55.722080 31980 leader_election.cc:288] T 417cedc3fa5c4a85a7a130005291d64b P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Requested vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:55.722271 32247 raft_consensus.cc:348] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:55.722589 32247 raft_consensus.cc:374] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:55.722702 32247 raft_consensus.cc:729] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:55.722985 32247 consensus_queue.cc:260] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:55.724016 32249 tablet_bootstrap.cc:492] T 417cedc3fa5c4a85a7a130005291d64b P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:55.724206 32249 ts_tablet_manager.cc:1367] T 417cedc3fa5c4a85a7a130005291d64b P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.006s
I20240503 08:12:55.724750 32247 ts_tablet_manager.cc:1398] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:55.725183 32249 raft_consensus.cc:348] T 417cedc3fa5c4a85a7a130005291d64b P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:55.725553 32249 raft_consensus.cc:374] T 417cedc3fa5c4a85a7a130005291d64b P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:55.725675 32249 raft_consensus.cc:729] T 417cedc3fa5c4a85a7a130005291d64b P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:55.726121 32249 consensus_queue.cc:260] T 417cedc3fa5c4a85a7a130005291d64b P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:55.729324 32249 ts_tablet_manager.cc:1398] T 417cedc3fa5c4a85a7a130005291d64b P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 08:12:55.729842 32249 tablet_bootstrap.cc:492] T fd6f4b81101b46588c3c7711f6bafcfa P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:55.735373 32244 ts_tablet_manager.cc:1398] T 417cedc3fa5c4a85a7a130005291d64b P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.035s user 0.001s sys 0.001s
I20240503 08:12:55.736670 32249 tablet_bootstrap.cc:654] T fd6f4b81101b46588c3c7711f6bafcfa P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:55.739595 32244 tablet_bootstrap.cc:492] T ff7be440a855440d8a655877d9553d8a P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:55.740986 32249 tablet_bootstrap.cc:492] T fd6f4b81101b46588c3c7711f6bafcfa P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:55.740986 32246 tablet_bootstrap.cc:492] T ff7be440a855440d8a655877d9553d8a P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:55.741148 32246 ts_tablet_manager.cc:1367] T ff7be440a855440d8a655877d9553d8a P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.040s user 0.013s sys 0.008s
I20240503 08:12:55.741178 32249 ts_tablet_manager.cc:1367] T fd6f4b81101b46588c3c7711f6bafcfa P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.001s
I20240503 08:12:55.741931 32246 raft_consensus.cc:348] T ff7be440a855440d8a655877d9553d8a P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:55.741947 32249 raft_consensus.cc:348] T fd6f4b81101b46588c3c7711f6bafcfa P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:55.742156 32246 raft_consensus.cc:374] T ff7be440a855440d8a655877d9553d8a P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:55.742235 32246 raft_consensus.cc:729] T ff7be440a855440d8a655877d9553d8a P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:55.742247 32249 raft_consensus.cc:374] T fd6f4b81101b46588c3c7711f6bafcfa P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:55.742416 32249 raft_consensus.cc:729] T fd6f4b81101b46588c3c7711f6bafcfa P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:55.742467 32246 consensus_queue.cc:260] T ff7be440a855440d8a655877d9553d8a P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:55.742749 32249 consensus_queue.cc:260] T fd6f4b81101b46588c3c7711f6bafcfa P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:55.743002 32246 ts_tablet_manager.cc:1398] T ff7be440a855440d8a655877d9553d8a P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:55.744210 32249 ts_tablet_manager.cc:1398] T fd6f4b81101b46588c3c7711f6bafcfa P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:12:55.749384 32244 tablet_bootstrap.cc:654] T ff7be440a855440d8a655877d9553d8a P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:55.758782 32244 tablet_bootstrap.cc:492] T ff7be440a855440d8a655877d9553d8a P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:55.759166 32244 ts_tablet_manager.cc:1367] T ff7be440a855440d8a655877d9553d8a P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.001s
I20240503 08:12:55.759881 27855 catalog_manager.cc:5484] T 417cedc3fa5c4a85a7a130005291d64b P 833582ff68c94269be364ebd187a75db reported cstate change: term changed from 0 to 1, leader changed from <none> to 833582ff68c94269be364ebd187a75db (127.20.230.65). New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:12:55.760637 32244 raft_consensus.cc:348] T ff7be440a855440d8a655877d9553d8a P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:55.760910 32244 raft_consensus.cc:374] T ff7be440a855440d8a655877d9553d8a P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:55.761031 32244 raft_consensus.cc:729] T ff7be440a855440d8a655877d9553d8a P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:55.761567 32244 consensus_queue.cc:260] T ff7be440a855440d8a655877d9553d8a P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:55.762363 32245 tablet_bootstrap.cc:492] T 97149ad1d03847a085b2142519135a17 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:55.762554 32245 ts_tablet_manager.cc:1367] T 97149ad1d03847a085b2142519135a17 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.050s user 0.012s sys 0.008s
I20240503 08:12:55.763289 32245 raft_consensus.cc:348] T 97149ad1d03847a085b2142519135a17 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:55.764613 32244 ts_tablet_manager.cc:1398] T ff7be440a855440d8a655877d9553d8a P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:12:55.764689 32245 raft_consensus.cc:374] T 97149ad1d03847a085b2142519135a17 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:55.764770 32245 raft_consensus.cc:729] T 97149ad1d03847a085b2142519135a17 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:55.765051 32245 consensus_queue.cc:260] T 97149ad1d03847a085b2142519135a17 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:55.765570 32245 ts_tablet_manager.cc:1398] T 97149ad1d03847a085b2142519135a17 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:12:55.767422 32245 tablet_bootstrap.cc:492] T ff7be440a855440d8a655877d9553d8a P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:55.769250 32245 tablet_bootstrap.cc:654] T ff7be440a855440d8a655877d9553d8a P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:55.773326 32245 tablet_bootstrap.cc:492] T ff7be440a855440d8a655877d9553d8a P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:55.773473 32245 ts_tablet_manager.cc:1367] T ff7be440a855440d8a655877d9553d8a P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240503 08:12:55.774158 32245 raft_consensus.cc:348] T ff7be440a855440d8a655877d9553d8a P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:55.774351 32245 raft_consensus.cc:374] T ff7be440a855440d8a655877d9553d8a P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:55.774487 32245 raft_consensus.cc:729] T ff7be440a855440d8a655877d9553d8a P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:55.774706 32245 consensus_queue.cc:260] T ff7be440a855440d8a655877d9553d8a P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:55.775213 32245 ts_tablet_manager.cc:1398] T ff7be440a855440d8a655877d9553d8a P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:55.800132 32046 raft_consensus.cc:482] T fd6f4b81101b46588c3c7711f6bafcfa P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:55.800284 32046 raft_consensus.cc:504] T fd6f4b81101b46588c3c7711f6bafcfa P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:55.800923 32046 leader_election.cc:288] T fd6f4b81101b46588c3c7711f6bafcfa P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:12:55.801290 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd6f4b81101b46588c3c7711f6bafcfa" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:12:55.801354 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd6f4b81101b46588c3c7711f6bafcfa" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:12:55.801520 30895 raft_consensus.cc:2454] T fd6f4b81101b46588c3c7711f6bafcfa P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 0.
I20240503 08:12:55.801541 30761 raft_consensus.cc:2454] T fd6f4b81101b46588c3c7711f6bafcfa P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 0.
I20240503 08:12:55.801923 31096 leader_election.cc:302] T fd6f4b81101b46588c3c7711f6bafcfa P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, b00f244c69df401997ea99d492218a98, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:55.802202 32046 raft_consensus.cc:2781] T fd6f4b81101b46588c3c7711f6bafcfa P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:55.802314 32046 raft_consensus.cc:482] T fd6f4b81101b46588c3c7711f6bafcfa P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:55.802407 32046 raft_consensus.cc:3037] T fd6f4b81101b46588c3c7711f6bafcfa P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:55.804514 32046 raft_consensus.cc:504] T fd6f4b81101b46588c3c7711f6bafcfa P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:55.805027 32046 leader_election.cc:288] T fd6f4b81101b46588c3c7711f6bafcfa P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 election: Requested vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:12:55.805397 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd6f4b81101b46588c3c7711f6bafcfa" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:12:55.805546 30895 raft_consensus.cc:3037] T fd6f4b81101b46588c3c7711f6bafcfa P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:55.805545 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd6f4b81101b46588c3c7711f6bafcfa" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:12:55.805796 30761 raft_consensus.cc:3037] T fd6f4b81101b46588c3c7711f6bafcfa P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:55.807519 30895 raft_consensus.cc:2454] T fd6f4b81101b46588c3c7711f6bafcfa P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 1.
I20240503 08:12:55.807864 30761 raft_consensus.cc:2454] T fd6f4b81101b46588c3c7711f6bafcfa P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 1.
I20240503 08:12:55.807870 31094 leader_election.cc:302] T fd6f4b81101b46588c3c7711f6bafcfa P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:55.808109 32046 raft_consensus.cc:2781] T fd6f4b81101b46588c3c7711f6bafcfa P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:55.808547 32046 raft_consensus.cc:686] T fd6f4b81101b46588c3c7711f6bafcfa P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Becoming Leader. State: Replica: b00f244c69df401997ea99d492218a98, State: Running, Role: LEADER
I20240503 08:12:55.808871 32046 consensus_queue.cc:237] T fd6f4b81101b46588c3c7711f6bafcfa P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:55.812368 27856 catalog_manager.cc:5484] T fd6f4b81101b46588c3c7711f6bafcfa P b00f244c69df401997ea99d492218a98 reported cstate change: term changed from 0 to 1, leader changed from <none> to b00f244c69df401997ea99d492218a98 (127.20.230.72). New cstate: current_term: 1 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:12:55.824476 32046 raft_consensus.cc:482] T 417cedc3fa5c4a85a7a130005291d64b P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:55.824654 32046 raft_consensus.cc:504] T 417cedc3fa5c4a85a7a130005291d64b P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:55.825361 32046 leader_election.cc:288] T 417cedc3fa5c4a85a7a130005291d64b P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:55.826419 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417cedc3fa5c4a85a7a130005291d64b" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:55.826711 31027 raft_consensus.cc:2379] T 417cedc3fa5c4a85a7a130005291d64b P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b00f244c69df401997ea99d492218a98 in current term 1: Already voted for candidate 833582ff68c94269be364ebd187a75db in this term.
I20240503 08:12:55.832136 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417cedc3fa5c4a85a7a130005291d64b" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:12:55.832660 31097 leader_election.cc:302] T 417cedc3fa5c4a85a7a130005291d64b P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b00f244c69df401997ea99d492218a98; no voters: 833582ff68c94269be364ebd187a75db, ced21ca983b749d0b4d4f6a0e2426bf8
I20240503 08:12:55.832943 32046 raft_consensus.cc:3037] T 417cedc3fa5c4a85a7a130005291d64b P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:55.834949 32046 raft_consensus.cc:2726] T 417cedc3fa5c4a85a7a130005291d64b P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:12:55.851092 32238 raft_consensus.cc:482] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:55.851269 32238 raft_consensus.cc:504] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:55.852296 32238 leader_election.cc:288] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:55.853266 31160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97149ad1d03847a085b2142519135a17" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:12:55.853523 31160 raft_consensus.cc:2454] T 97149ad1d03847a085b2142519135a17 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:12:55.854898 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97149ad1d03847a085b2142519135a17" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:12:55.855087 30761 raft_consensus.cc:2454] T 97149ad1d03847a085b2142519135a17 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:12:55.855631 30829 leader_election.cc:302] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:55.855911 32238 raft_consensus.cc:2781] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:55.856041 32238 raft_consensus.cc:482] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:55.856127 32238 raft_consensus.cc:3037] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:55.858902 32238 raft_consensus.cc:504] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:55.859885 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97149ad1d03847a085b2142519135a17" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:12:55.860113 30761 raft_consensus.cc:3037] T 97149ad1d03847a085b2142519135a17 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:55.862123 30761 raft_consensus.cc:2454] T 97149ad1d03847a085b2142519135a17 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:12:55.862442 30829 leader_election.cc:302] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:55.862712 32112 raft_consensus.cc:2781] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:55.862851 32238 leader_election.cc:288] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Requested vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:55.862932 31160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97149ad1d03847a085b2142519135a17" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:12:55.863116 31160 raft_consensus.cc:3037] T 97149ad1d03847a085b2142519135a17 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:55.863145 32112 raft_consensus.cc:686] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Becoming Leader. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Running, Role: LEADER
I20240503 08:12:55.863425 32112 consensus_queue.cc:237] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:55.865389 31160 raft_consensus.cc:2454] T 97149ad1d03847a085b2142519135a17 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:12:55.867107 27856 catalog_manager.cc:5484] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c reported cstate change: term changed from 0 to 1, leader changed from <none> to ce1496a47d084365ad786cd09e472a2c (127.20.230.70). New cstate: current_term: 1 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:12:55.939296 32218 raft_consensus.cc:482] T ff7be440a855440d8a655877d9553d8a P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:55.939466 32218 raft_consensus.cc:504] T ff7be440a855440d8a655877d9553d8a P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:55.940112 32218 leader_election.cc:288] T ff7be440a855440d8a655877d9553d8a P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:12:55.940621 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff7be440a855440d8a655877d9553d8a" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:12:55.940769 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff7be440a855440d8a655877d9553d8a" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:12:55.940850 28002 raft_consensus.cc:2454] T ff7be440a855440d8a655877d9553d8a P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 0.
I20240503 08:12:55.941022 30761 raft_consensus.cc:2454] T ff7be440a855440d8a655877d9553d8a P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 0.
I20240503 08:12:55.941229 30962 leader_election.cc:302] T ff7be440a855440d8a655877d9553d8a P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:55.941823 32218 raft_consensus.cc:2781] T ff7be440a855440d8a655877d9553d8a P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:55.941970 32218 raft_consensus.cc:482] T ff7be440a855440d8a655877d9553d8a P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:55.942088 32218 raft_consensus.cc:3037] T ff7be440a855440d8a655877d9553d8a P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:55.944422 32218 raft_consensus.cc:504] T ff7be440a855440d8a655877d9553d8a P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:55.945077 32218 leader_election.cc:288] T ff7be440a855440d8a655877d9553d8a P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 election: Requested vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:12:55.945263 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff7be440a855440d8a655877d9553d8a" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:12:55.945477 28002 raft_consensus.cc:3037] T ff7be440a855440d8a655877d9553d8a P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:55.945506 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff7be440a855440d8a655877d9553d8a" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:12:55.945672 30761 raft_consensus.cc:3037] T ff7be440a855440d8a655877d9553d8a P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:55.947355 28002 raft_consensus.cc:2454] T ff7be440a855440d8a655877d9553d8a P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 1.
I20240503 08:12:55.947639 30761 raft_consensus.cc:2454] T ff7be440a855440d8a655877d9553d8a P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 1.
I20240503 08:12:55.947849 30962 leader_election.cc:302] T ff7be440a855440d8a655877d9553d8a P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:55.948208 32218 raft_consensus.cc:2781] T ff7be440a855440d8a655877d9553d8a P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:55.948369 32218 raft_consensus.cc:686] T ff7be440a855440d8a655877d9553d8a P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 LEADER]: Becoming Leader. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Running, Role: LEADER
I20240503 08:12:55.948807 32218 consensus_queue.cc:237] T ff7be440a855440d8a655877d9553d8a P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:55.951967 27856 catalog_manager.cc:5484] T ff7be440a855440d8a655877d9553d8a P ced21ca983b749d0b4d4f6a0e2426bf8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71). New cstate: current_term: 1 leader_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:12:56.074630 27856 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:56.081243 30761 raft_consensus.cc:1261] T ff7be440a855440d8a655877d9553d8a P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer ced21ca983b749d0b4d4f6a0e2426bf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:56.081872 28002 raft_consensus.cc:1261] T ff7be440a855440d8a655877d9553d8a P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer ced21ca983b749d0b4d4f6a0e2426bf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:56.082158 32218 consensus_queue.cc:1035] T ff7be440a855440d8a655877d9553d8a P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:56.083549 32218 consensus_queue.cc:1035] T ff7be440a855440d8a655877d9553d8a P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:56.087117 27856 catalog_manager.cc:5576] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725) does not have the latest schema for tablet ff7be440a855440d8a655877d9553d8a (table rebalancer_dynamic_table [id=844951c455764b679f4e60c4b22738ae]). Expected version 1 got 0
I20240503 08:12:56.088764 32275 tablet.cc:1722] T ff7be440a855440d8a655877d9553d8a P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:56.092389 32275 tablet.cc:1620] T ff7be440a855440d8a655877d9553d8a P ced21ca983b749d0b4d4f6a0e2426bf8: MemRowSet was empty: no flush needed.
I20240503 08:12:56.095568 32276 tablet.cc:1722] T ff7be440a855440d8a655877d9553d8a P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:56.096129 32276 tablet.cc:1620] T ff7be440a855440d8a655877d9553d8a P 6d1d7de19459435fbde1999e3dfd226b: MemRowSet was empty: no flush needed.
I20240503 08:12:56.098825 27856 catalog_manager.cc:5576] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725) does not have the latest schema for tablet ff7be440a855440d8a655877d9553d8a (table rebalancer_dynamic_table [id=844951c455764b679f4e60c4b22738ae]). Expected version 1 got 0
I20240503 08:12:56.107009 27855 catalog_manager.cc:5576] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081) does not have the latest schema for tablet ff7be440a855440d8a655877d9553d8a (table rebalancer_dynamic_table [id=844951c455764b679f4e60c4b22738ae]). Expected version 1 got 0
I20240503 08:12:56.111672 27856 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=844951c455764b679f4e60c4b22738ae] alter complete (version 1)
I20240503 08:12:56.114193 32279 tablet.cc:1722] T ff7be440a855440d8a655877d9553d8a P 833582ff68c94269be364ebd187a75db: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:56.115362 32279 tablet.cc:1620] T ff7be440a855440d8a655877d9553d8a P 833582ff68c94269be364ebd187a75db: MemRowSet was empty: no flush needed.
I20240503 08:12:56.116607 27856 catalog_manager.cc:5576] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081) does not have the latest schema for tablet ff7be440a855440d8a655877d9553d8a (table rebalancer_dynamic_table [id=844951c455764b679f4e60c4b22738ae]). Expected version 1 got 0
I20240503 08:12:56.214478 32210 consensus_queue.cc:1035] T 417cedc3fa5c4a85a7a130005291d64b P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:56.219326 32256 consensus_queue.cc:1035] T 417cedc3fa5c4a85a7a130005291d64b P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:56.222615 27856 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:56.234500 32275 tablet.cc:1722] T ff7be440a855440d8a655877d9553d8a P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:56.234995 32275 tablet.cc:1620] T ff7be440a855440d8a655877d9553d8a P ced21ca983b749d0b4d4f6a0e2426bf8: MemRowSet was empty: no flush needed.
I20240503 08:12:56.236290 32276 tablet.cc:1722] T ff7be440a855440d8a655877d9553d8a P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:56.236526 32279 tablet.cc:1722] T ff7be440a855440d8a655877d9553d8a P 833582ff68c94269be364ebd187a75db: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:56.236748 32276 tablet.cc:1620] T ff7be440a855440d8a655877d9553d8a P 6d1d7de19459435fbde1999e3dfd226b: MemRowSet was empty: no flush needed.
I20240503 08:12:56.236966 32279 tablet.cc:1620] T ff7be440a855440d8a655877d9553d8a P 833582ff68c94269be364ebd187a75db: MemRowSet was empty: no flush needed.
I20240503 08:12:56.245934 27855 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=844951c455764b679f4e60c4b22738ae] alter complete (version 2)
I20240503 08:12:56.254696 32046 consensus_queue.cc:1035] T fd6f4b81101b46588c3c7711f6bafcfa P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:56.258854 32046 consensus_queue.cc:1035] T fd6f4b81101b46588c3c7711f6bafcfa P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:56.348753 27856 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:56.355458 32275 tablet.cc:1722] T ff7be440a855440d8a655877d9553d8a P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:56.356165 32276 tablet.cc:1722] T ff7be440a855440d8a655877d9553d8a P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:56.356137 32279 tablet.cc:1722] T ff7be440a855440d8a655877d9553d8a P 833582ff68c94269be364ebd187a75db: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:56.361505 32238 consensus_queue.cc:1035] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:56.365571 27856 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table_new_name [id=844951c455764b679f4e60c4b22738ae] alter complete (version 3)
I20240503 08:12:56.367805 32238 consensus_queue.cc:1035] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:56.396185 27859 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_extra_table_36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:56.404029 30741 tablet_service.cc:1467] Processing CreateTablet for tablet f9746769e2514c65ac8d48ca7df8bca7 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=ae8d4f3f001646efbeb5154b3ce01aac]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:56.404317 31140 tablet_service.cc:1467] Processing CreateTablet for tablet f9746769e2514c65ac8d48ca7df8bca7 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=ae8d4f3f001646efbeb5154b3ce01aac]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:56.404501 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9746769e2514c65ac8d48ca7df8bca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:56.404773 31140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9746769e2514c65ac8d48ca7df8bca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:56.404801 28111 tablet_service.cc:1467] Processing CreateTablet for tablet 362590d3d0834d0b8a672ab552f8c209 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=ae8d4f3f001646efbeb5154b3ce01aac]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:56.405247 28111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 362590d3d0834d0b8a672ab552f8c209. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:56.407004 30873 tablet_service.cc:1467] Processing CreateTablet for tablet 362590d3d0834d0b8a672ab552f8c209 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=ae8d4f3f001646efbeb5154b3ce01aac]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:56.407392 30873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 362590d3d0834d0b8a672ab552f8c209. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:56.408260 30740 tablet_service.cc:1467] Processing CreateTablet for tablet ebe1a0b781404070a5326a80f8f9bfb4 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=ae8d4f3f001646efbeb5154b3ce01aac]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:56.408633 30740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebe1a0b781404070a5326a80f8f9bfb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:56.410012 30872 tablet_service.cc:1467] Processing CreateTablet for tablet ebe1a0b781404070a5326a80f8f9bfb4 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=ae8d4f3f001646efbeb5154b3ce01aac]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:56.410115 30874 tablet_service.cc:1467] Processing CreateTablet for tablet f9746769e2514c65ac8d48ca7df8bca7 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=ae8d4f3f001646efbeb5154b3ce01aac]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:56.410398 30872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebe1a0b781404070a5326a80f8f9bfb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:56.410821 30874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9746769e2514c65ac8d48ca7df8bca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:56.412792 31007 tablet_service.cc:1467] Processing CreateTablet for tablet ebe1a0b781404070a5326a80f8f9bfb4 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=ae8d4f3f001646efbeb5154b3ce01aac]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:56.413189 31007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebe1a0b781404070a5326a80f8f9bfb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:56.423735 28372 tablet_service.cc:1467] Processing CreateTablet for tablet 362590d3d0834d0b8a672ab552f8c209 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=ae8d4f3f001646efbeb5154b3ce01aac]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:56.424193 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 362590d3d0834d0b8a672ab552f8c209. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:56.427789 32291 tablet_bootstrap.cc:492] T f9746769e2514c65ac8d48ca7df8bca7 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:56.428037 32293 tablet_bootstrap.cc:492] T ebe1a0b781404070a5326a80f8f9bfb4 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:56.428843 32290 tablet_bootstrap.cc:492] T ebe1a0b781404070a5326a80f8f9bfb4 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:56.429961 32291 tablet_bootstrap.cc:654] T f9746769e2514c65ac8d48ca7df8bca7 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:56.430639 32290 tablet_bootstrap.cc:654] T ebe1a0b781404070a5326a80f8f9bfb4 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:56.433641 32293 tablet_bootstrap.cc:654] T ebe1a0b781404070a5326a80f8f9bfb4 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:56.436848 32291 tablet_bootstrap.cc:492] T f9746769e2514c65ac8d48ca7df8bca7 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:56.437448 32291 ts_tablet_manager.cc:1367] T f9746769e2514c65ac8d48ca7df8bca7 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s
I20240503 08:12:56.438424 32291 raft_consensus.cc:348] T f9746769e2514c65ac8d48ca7df8bca7 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:56.439720 32291 raft_consensus.cc:374] T f9746769e2514c65ac8d48ca7df8bca7 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:56.440052 32291 raft_consensus.cc:729] T f9746769e2514c65ac8d48ca7df8bca7 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:56.440974 32291 consensus_queue.cc:260] T f9746769e2514c65ac8d48ca7df8bca7 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:56.441994 32291 ts_tablet_manager.cc:1398] T f9746769e2514c65ac8d48ca7df8bca7 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:56.442662 32293 tablet_bootstrap.cc:492] T ebe1a0b781404070a5326a80f8f9bfb4 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:56.442864 32293 ts_tablet_manager.cc:1367] T ebe1a0b781404070a5326a80f8f9bfb4 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20240503 08:12:56.445084 32290 tablet_bootstrap.cc:492] T ebe1a0b781404070a5326a80f8f9bfb4 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:56.445269 32290 ts_tablet_manager.cc:1367] T ebe1a0b781404070a5326a80f8f9bfb4 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.007s
I20240503 08:12:56.445286 32293 raft_consensus.cc:348] T ebe1a0b781404070a5326a80f8f9bfb4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:56.445583 32293 raft_consensus.cc:374] T ebe1a0b781404070a5326a80f8f9bfb4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:56.445717 32293 raft_consensus.cc:729] T ebe1a0b781404070a5326a80f8f9bfb4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:56.446142 32290 raft_consensus.cc:348] T ebe1a0b781404070a5326a80f8f9bfb4 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:56.446384 32290 raft_consensus.cc:374] T ebe1a0b781404070a5326a80f8f9bfb4 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:56.447172 32290 raft_consensus.cc:729] T ebe1a0b781404070a5326a80f8f9bfb4 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:56.447564 32290 consensus_queue.cc:260] T ebe1a0b781404070a5326a80f8f9bfb4 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:56.446048 32293 consensus_queue.cc:260] T ebe1a0b781404070a5326a80f8f9bfb4 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:56.448375 32293 ts_tablet_manager.cc:1398] T ebe1a0b781404070a5326a80f8f9bfb4 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:56.450749 32295 tablet_bootstrap.cc:492] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:56.452168 32290 ts_tablet_manager.cc:1398] T ebe1a0b781404070a5326a80f8f9bfb4 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240503 08:12:56.452663 32295 tablet_bootstrap.cc:654] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:56.453195 32294 tablet_bootstrap.cc:492] T 362590d3d0834d0b8a672ab552f8c209 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:56.453850 32290 tablet_bootstrap.cc:492] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:56.454520 32292 tablet_bootstrap.cc:492] T ebe1a0b781404070a5326a80f8f9bfb4 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:56.455696 32294 tablet_bootstrap.cc:654] T 362590d3d0834d0b8a672ab552f8c209 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:56.457245 32292 tablet_bootstrap.cc:654] T ebe1a0b781404070a5326a80f8f9bfb4 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:56.457962 32290 tablet_bootstrap.cc:654] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:56.460465 32290 tablet_bootstrap.cc:492] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:56.460608 32290 ts_tablet_manager.cc:1367] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20240503 08:12:56.460891 32294 tablet_bootstrap.cc:492] T 362590d3d0834d0b8a672ab552f8c209 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:56.460891 32295 tablet_bootstrap.cc:492] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:56.461030 32294 ts_tablet_manager.cc:1367] T 362590d3d0834d0b8a672ab552f8c209 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240503 08:12:56.461261 32295 ts_tablet_manager.cc:1367] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240503 08:12:56.461220 32290 raft_consensus.cc:348] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:56.461508 32290 raft_consensus.cc:374] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:56.461632 32290 raft_consensus.cc:729] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:56.461796 32294 raft_consensus.cc:348] T 362590d3d0834d0b8a672ab552f8c209 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:56.462042 32294 raft_consensus.cc:374] T 362590d3d0834d0b8a672ab552f8c209 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:56.461957 32290 consensus_queue.cc:260] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:56.462155 32294 raft_consensus.cc:729] T 362590d3d0834d0b8a672ab552f8c209 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:56.462029 32295 raft_consensus.cc:348] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:56.462332 32295 raft_consensus.cc:374] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:56.462469 32295 raft_consensus.cc:729] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:56.462381 32294 consensus_queue.cc:260] T 362590d3d0834d0b8a672ab552f8c209 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:56.462693 32290 ts_tablet_manager.cc:1398] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:56.462739 32295 consensus_queue.cc:260] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:56.462999 32290 tablet_bootstrap.cc:492] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:56.463418 32295 ts_tablet_manager.cc:1398] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:56.464346 32292 tablet_bootstrap.cc:492] T ebe1a0b781404070a5326a80f8f9bfb4 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:56.464506 32292 ts_tablet_manager.cc:1367] T ebe1a0b781404070a5326a80f8f9bfb4 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240503 08:12:56.464496 32294 ts_tablet_manager.cc:1398] T 362590d3d0834d0b8a672ab552f8c209 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:56.465102 32292 raft_consensus.cc:348] T ebe1a0b781404070a5326a80f8f9bfb4 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:56.465798 32292 raft_consensus.cc:374] T ebe1a0b781404070a5326a80f8f9bfb4 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:56.466019 32292 raft_consensus.cc:729] T ebe1a0b781404070a5326a80f8f9bfb4 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:56.466275 32292 consensus_queue.cc:260] T ebe1a0b781404070a5326a80f8f9bfb4 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:56.467765 32292 ts_tablet_manager.cc:1398] T ebe1a0b781404070a5326a80f8f9bfb4 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:56.468142 32292 tablet_bootstrap.cc:492] T f9746769e2514c65ac8d48ca7df8bca7 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:56.469640 32292 tablet_bootstrap.cc:654] T f9746769e2514c65ac8d48ca7df8bca7 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:56.472393 27856 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:56.472589 27856 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:56.474254 32290 tablet_bootstrap.cc:654] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:56.476311 32290 tablet_bootstrap.cc:492] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:56.476409 32292 tablet_bootstrap.cc:492] T f9746769e2514c65ac8d48ca7df8bca7 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:56.476454 32290 ts_tablet_manager.cc:1367] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20240503 08:12:56.476662 32292 ts_tablet_manager.cc:1367] T f9746769e2514c65ac8d48ca7df8bca7 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240503 08:12:56.477207 32290 raft_consensus.cc:348] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:56.477350 32292 raft_consensus.cc:348] T f9746769e2514c65ac8d48ca7df8bca7 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:56.477521 32290 raft_consensus.cc:374] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:56.477640 32290 raft_consensus.cc:729] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:56.477826 32290 consensus_queue.cc:260] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:56.477955 32292 raft_consensus.cc:374] T f9746769e2514c65ac8d48ca7df8bca7 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:56.478061 32292 raft_consensus.cc:729] T f9746769e2514c65ac8d48ca7df8bca7 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:56.478338 32292 consensus_queue.cc:260] T f9746769e2514c65ac8d48ca7df8bca7 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:56.478863 32292 ts_tablet_manager.cc:1398] T f9746769e2514c65ac8d48ca7df8bca7 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:56.479447 32290 ts_tablet_manager.cc:1398] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:56.485271 27856 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet ff7be440a855440d8a655877d9553d8a
I20240503 08:12:56.485998 27983 tablet_service.cc:1514] Processing DeleteTablet for tablet ff7be440a855440d8a655877d9553d8a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:56 UTC) from {username='slave'} at 127.0.0.1:55610
I20240503 08:12:56.486325 30740 tablet_service.cc:1514] Processing DeleteTablet for tablet ff7be440a855440d8a655877d9553d8a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:56 UTC) from {username='slave'} at 127.0.0.1:46794
I20240503 08:12:56.486773 31007 tablet_service.cc:1514] Processing DeleteTablet for tablet ff7be440a855440d8a655877d9553d8a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:56 UTC) from {username='slave'} at 127.0.0.1:47020
I20240503 08:12:56.501139 32307 tablet_replica.cc:330] T ff7be440a855440d8a655877d9553d8a P ced21ca983b749d0b4d4f6a0e2426bf8: stopping tablet replica
I20240503 08:12:56.501603 32307 raft_consensus.cc:2229] T ff7be440a855440d8a655877d9553d8a P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:56.502012 32307 raft_consensus.cc:2258] T ff7be440a855440d8a655877d9553d8a P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:56.505964 32307 ts_tablet_manager.cc:1875] T ff7be440a855440d8a655877d9553d8a P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:56.508203 32112 raft_consensus.cc:482] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:56.508801 32112 raft_consensus.cc:504] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:56.516662 28131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "362590d3d0834d0b8a672ab552f8c209" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:12:56.516878 28131 raft_consensus.cc:2454] T 362590d3d0834d0b8a672ab552f8c209 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:12:56.517454 30829 leader_election.cc:302] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:56.517791 32238 raft_consensus.cc:2781] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:56.517877 32238 raft_consensus.cc:482] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:56.517963 32238 raft_consensus.cc:3037] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:56.521310 32238 raft_consensus.cc:504] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:56.523979 28393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "362590d3d0834d0b8a672ab552f8c209" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:12:56.524124 32238 leader_election.cc:288] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Requested vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:56.524189 28393 raft_consensus.cc:3037] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:56.524647 28131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "362590d3d0834d0b8a672ab552f8c209" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:12:56.524825 28131 raft_consensus.cc:3037] T 362590d3d0834d0b8a672ab552f8c209 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:56.526404 32238 raft_consensus.cc:482] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:56.526579 32238 raft_consensus.cc:504] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:56.526916 28393 raft_consensus.cc:2454] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:12:56.527382 32238 leader_election.cc:288] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:56.527707 31160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9746769e2514c65ac8d48ca7df8bca7" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:12:56.527927 31160 raft_consensus.cc:2454] T f9746769e2514c65ac8d48ca7df8bca7 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:12:56.528472 30827 leader_election.cc:302] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:56.529191 30830 leader_election.cc:302] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:56.529731 28131 raft_consensus.cc:2454] T 362590d3d0834d0b8a672ab552f8c209 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:12:56.530531 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9746769e2514c65ac8d48ca7df8bca7" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:12:56.530810 30762 raft_consensus.cc:2454] T f9746769e2514c65ac8d48ca7df8bca7 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:12:56.531388 32112 leader_election.cc:288] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:12:56.531544 32112 raft_consensus.cc:2781] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:56.531628 32112 raft_consensus.cc:482] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:56.531716 32112 raft_consensus.cc:3037] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:56.537145 32308 raft_consensus.cc:2781] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:56.537351 32308 raft_consensus.cc:686] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Becoming Leader. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Running, Role: LEADER
I20240503 08:12:56.537914 28393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "362590d3d0834d0b8a672ab552f8c209" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:12:56.538106 28393 raft_consensus.cc:2362] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1. Re-sending same reply.
I20240503 08:12:56.538292 32112 raft_consensus.cc:504] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:56.539569 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9746769e2514c65ac8d48ca7df8bca7" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:12:56.539788 30762 raft_consensus.cc:3037] T f9746769e2514c65ac8d48ca7df8bca7 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:56.541975 31160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9746769e2514c65ac8d48ca7df8bca7" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:12:56.542171 31160 raft_consensus.cc:3037] T f9746769e2514c65ac8d48ca7df8bca7 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:56.542586 30762 raft_consensus.cc:2454] T f9746769e2514c65ac8d48ca7df8bca7 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:12:56.542898 32308 consensus_queue.cc:237] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:56.544862 31160 raft_consensus.cc:2454] T f9746769e2514c65ac8d48ca7df8bca7 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:12:56.545944 30829 leader_election.cc:302] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:56.547040 32238 raft_consensus.cc:2781] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:56.555011 32112 leader_election.cc:288] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Requested vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:56.560882 32306 tablet_replica.cc:330] T ff7be440a855440d8a655877d9553d8a P 6d1d7de19459435fbde1999e3dfd226b: stopping tablet replica
I20240503 08:12:56.561113 32306 raft_consensus.cc:2229] T ff7be440a855440d8a655877d9553d8a P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:56.561336 32306 raft_consensus.cc:2258] T ff7be440a855440d8a655877d9553d8a P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:56.562178 32305 tablet_replica.cc:330] T ff7be440a855440d8a655877d9553d8a P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:12:56.562398 32305 raft_consensus.cc:2229] T ff7be440a855440d8a655877d9553d8a P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:56.562611 32305 raft_consensus.cc:2258] T ff7be440a855440d8a655877d9553d8a P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:56.563025 32238 raft_consensus.cc:686] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Becoming Leader. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Running, Role: LEADER
I20240503 08:12:56.563390 32305 ts_tablet_manager.cc:1875] T ff7be440a855440d8a655877d9553d8a P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:56.563359 32238 consensus_queue.cc:237] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:56.568231 32305 ts_tablet_manager.cc:1888] T ff7be440a855440d8a655877d9553d8a P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:56.568358 32305 log.cc:1196] T ff7be440a855440d8a655877d9553d8a P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/ff7be440a855440d8a655877d9553d8a
I20240503 08:12:56.568747 32305 ts_tablet_manager.cc:1909] T ff7be440a855440d8a655877d9553d8a P 833582ff68c94269be364ebd187a75db: Deleting consensus metadata
I20240503 08:12:56.569983 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet ff7be440a855440d8a655877d9553d8a (table rebalancer_dynamic_table_new_name [id=844951c455764b679f4e60c4b22738ae]) successfully deleted
I20240503 08:12:56.570417 27856 catalog_manager.cc:5484] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c reported cstate change: term changed from 0 to 1, leader changed from <none> to ce1496a47d084365ad786cd09e472a2c (127.20.230.70). New cstate: current_term: 1 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:12:56.574194 32306 ts_tablet_manager.cc:1875] T ff7be440a855440d8a655877d9553d8a P 6d1d7de19459435fbde1999e3dfd226b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:56.575749 32307 ts_tablet_manager.cc:1888] T ff7be440a855440d8a655877d9553d8a P ced21ca983b749d0b4d4f6a0e2426bf8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:56.575913 32307 log.cc:1196] T ff7be440a855440d8a655877d9553d8a P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal/wals/ff7be440a855440d8a655877d9553d8a
W20240503 08:12:56.575974 28069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.68:46211, user_credentials={real_user=slave}} blocked reactor thread for 43784.5us
I20240503 08:12:56.576522 27856 catalog_manager.cc:5484] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c reported cstate change: term changed from 0 to 1, leader changed from <none> to ce1496a47d084365ad786cd09e472a2c (127.20.230.70). New cstate: current_term: 1 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:12:56.577085 32307 ts_tablet_manager.cc:1909] T ff7be440a855440d8a655877d9553d8a P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting consensus metadata
I20240503 08:12:56.578816 27847 catalog_manager.cc:4830] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): tablet ff7be440a855440d8a655877d9553d8a (table rebalancer_dynamic_table_new_name [id=844951c455764b679f4e60c4b22738ae]) successfully deleted
I20240503 08:12:56.580603 32306 ts_tablet_manager.cc:1888] T ff7be440a855440d8a655877d9553d8a P 6d1d7de19459435fbde1999e3dfd226b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:56.580710 32306 log.cc:1196] T ff7be440a855440d8a655877d9553d8a P 6d1d7de19459435fbde1999e3dfd226b: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal/wals/ff7be440a855440d8a655877d9553d8a
I20240503 08:12:56.581566 32306 ts_tablet_manager.cc:1909] T ff7be440a855440d8a655877d9553d8a P 6d1d7de19459435fbde1999e3dfd226b: Deleting consensus metadata
I20240503 08:12:56.582476 27848 catalog_manager.cc:4830] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): tablet ff7be440a855440d8a655877d9553d8a (table rebalancer_dynamic_table_new_name [id=844951c455764b679f4e60c4b22738ae]) successfully deleted
I20240503 08:12:56.587903 27855 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:56.596645 31007 tablet_service.cc:1467] Processing CreateTablet for tablet ac2b98ace3534e8c8656f91b3e044261 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e400d7b778084ee8b001d93eda4fa2ab]), partition=
I20240503 08:12:56.596648 30872 tablet_service.cc:1467] Processing CreateTablet for tablet ac2b98ace3534e8c8656f91b3e044261 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e400d7b778084ee8b001d93eda4fa2ab]), partition=
I20240503 08:12:56.596973 30872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac2b98ace3534e8c8656f91b3e044261. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:56.597035 31007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac2b98ace3534e8c8656f91b3e044261. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:56.601505 32293 tablet_bootstrap.cc:492] T ac2b98ace3534e8c8656f91b3e044261 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:56.602618 30740 tablet_service.cc:1467] Processing CreateTablet for tablet ac2b98ace3534e8c8656f91b3e044261 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e400d7b778084ee8b001d93eda4fa2ab]), partition=
I20240503 08:12:56.602967 30740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac2b98ace3534e8c8656f91b3e044261. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:56.603129 32293 tablet_bootstrap.cc:654] T ac2b98ace3534e8c8656f91b3e044261 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:56.605232 32293 tablet_bootstrap.cc:492] T ac2b98ace3534e8c8656f91b3e044261 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:56.605473 32293 ts_tablet_manager.cc:1367] T ac2b98ace3534e8c8656f91b3e044261 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 08:12:56.607762 32290 tablet_bootstrap.cc:492] T ac2b98ace3534e8c8656f91b3e044261 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:56.608433 32293 raft_consensus.cc:348] T ac2b98ace3534e8c8656f91b3e044261 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:56.608644 32293 raft_consensus.cc:374] T ac2b98ace3534e8c8656f91b3e044261 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:56.608943 32292 tablet_bootstrap.cc:492] T ac2b98ace3534e8c8656f91b3e044261 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:56.609231 32290 tablet_bootstrap.cc:654] T ac2b98ace3534e8c8656f91b3e044261 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:56.610364 32292 tablet_bootstrap.cc:654] T ac2b98ace3534e8c8656f91b3e044261 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:56.608707 32293 raft_consensus.cc:729] T ac2b98ace3534e8c8656f91b3e044261 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:56.611474 32293 consensus_queue.cc:260] T ac2b98ace3534e8c8656f91b3e044261 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:56.612324 32290 tablet_bootstrap.cc:492] T ac2b98ace3534e8c8656f91b3e044261 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:56.612563 32290 ts_tablet_manager.cc:1367] T ac2b98ace3534e8c8656f91b3e044261 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 08:12:56.613063 32293 ts_tablet_manager.cc:1398] T ac2b98ace3534e8c8656f91b3e044261 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240503 08:12:56.613255 32290 raft_consensus.cc:348] T ac2b98ace3534e8c8656f91b3e044261 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:56.613865 32290 raft_consensus.cc:374] T ac2b98ace3534e8c8656f91b3e044261 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:56.614357 32290 raft_consensus.cc:729] T ac2b98ace3534e8c8656f91b3e044261 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:56.614689 32290 consensus_queue.cc:260] T ac2b98ace3534e8c8656f91b3e044261 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:56.617362 32290 ts_tablet_manager.cc:1398] T ac2b98ace3534e8c8656f91b3e044261 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:12:56.625658 32292 tablet_bootstrap.cc:492] T ac2b98ace3534e8c8656f91b3e044261 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:56.625944 32292 ts_tablet_manager.cc:1367] T ac2b98ace3534e8c8656f91b3e044261 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.010s
I20240503 08:12:56.626919 32292 raft_consensus.cc:348] T ac2b98ace3534e8c8656f91b3e044261 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:56.627169 32292 raft_consensus.cc:374] T ac2b98ace3534e8c8656f91b3e044261 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:56.627321 32292 raft_consensus.cc:729] T ac2b98ace3534e8c8656f91b3e044261 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:56.627605 32292 consensus_queue.cc:260] T ac2b98ace3534e8c8656f91b3e044261 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:56.629355 32292 ts_tablet_manager.cc:1398] T ac2b98ace3534e8c8656f91b3e044261 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:12:56.669533 32238 raft_consensus.cc:482] T ebe1a0b781404070a5326a80f8f9bfb4 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:56.669725 32238 raft_consensus.cc:504] T ebe1a0b781404070a5326a80f8f9bfb4 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:56.670480 32238 leader_election.cc:288] T ebe1a0b781404070a5326a80f8f9bfb4 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:12:56.670851 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe1a0b781404070a5326a80f8f9bfb4" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:56.671088 31027 raft_consensus.cc:2454] T ebe1a0b781404070a5326a80f8f9bfb4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:12:56.671473 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe1a0b781404070a5326a80f8f9bfb4" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:12:56.671662 30762 raft_consensus.cc:2454] T ebe1a0b781404070a5326a80f8f9bfb4 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:12:56.671636 30828 leader_election.cc:302] T ebe1a0b781404070a5326a80f8f9bfb4 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:56.671932 32238 raft_consensus.cc:2781] T ebe1a0b781404070a5326a80f8f9bfb4 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:56.672048 32238 raft_consensus.cc:482] T ebe1a0b781404070a5326a80f8f9bfb4 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:56.672143 32238 raft_consensus.cc:3037] T ebe1a0b781404070a5326a80f8f9bfb4 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:56.674096 32238 raft_consensus.cc:504] T ebe1a0b781404070a5326a80f8f9bfb4 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:56.674688 32238 leader_election.cc:288] T ebe1a0b781404070a5326a80f8f9bfb4 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Requested vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:12:56.675084 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe1a0b781404070a5326a80f8f9bfb4" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:12:56.675194 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe1a0b781404070a5326a80f8f9bfb4" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:12:56.675259 30762 raft_consensus.cc:3037] T ebe1a0b781404070a5326a80f8f9bfb4 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:56.675386 31027 raft_consensus.cc:3037] T ebe1a0b781404070a5326a80f8f9bfb4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:56.677945 31027 raft_consensus.cc:2454] T ebe1a0b781404070a5326a80f8f9bfb4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:12:56.677958 30762 raft_consensus.cc:2454] T ebe1a0b781404070a5326a80f8f9bfb4 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:12:56.678318 30828 leader_election.cc:302] T ebe1a0b781404070a5326a80f8f9bfb4 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:56.678678 32238 raft_consensus.cc:2781] T ebe1a0b781404070a5326a80f8f9bfb4 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:56.678870 32238 raft_consensus.cc:686] T ebe1a0b781404070a5326a80f8f9bfb4 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Becoming Leader. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Running, Role: LEADER
I20240503 08:12:56.679149 32238 consensus_queue.cc:237] T ebe1a0b781404070a5326a80f8f9bfb4 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:56.682190 32280 raft_consensus.cc:482] T ac2b98ace3534e8c8656f91b3e044261 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:56.682315 32280 raft_consensus.cc:504] T ac2b98ace3534e8c8656f91b3e044261 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:56.682603 27856 catalog_manager.cc:5484] T ebe1a0b781404070a5326a80f8f9bfb4 P ce1496a47d084365ad786cd09e472a2c reported cstate change: term changed from 0 to 1, leader changed from <none> to ce1496a47d084365ad786cd09e472a2c (127.20.230.70). New cstate: current_term: 1 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:12:56.683025 32280 leader_election.cc:288] T ac2b98ace3534e8c8656f91b3e044261 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:56.683370 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac2b98ace3534e8c8656f91b3e044261" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:12:56.683477 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac2b98ace3534e8c8656f91b3e044261" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:56.683588 30895 raft_consensus.cc:2454] T ac2b98ace3534e8c8656f91b3e044261 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:12:56.683693 31027 raft_consensus.cc:2454] T ac2b98ace3534e8c8656f91b3e044261 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:12:56.683984 30694 leader_election.cc:302] T ac2b98ace3534e8c8656f91b3e044261 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:56.684229 32280 raft_consensus.cc:2781] T ac2b98ace3534e8c8656f91b3e044261 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:56.684345 32280 raft_consensus.cc:482] T ac2b98ace3534e8c8656f91b3e044261 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:56.684434 32280 raft_consensus.cc:3037] T ac2b98ace3534e8c8656f91b3e044261 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:56.686398 32280 raft_consensus.cc:504] T ac2b98ace3534e8c8656f91b3e044261 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:56.686823 32280 leader_election.cc:288] T ac2b98ace3534e8c8656f91b3e044261 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Requested vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:56.687155 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac2b98ace3534e8c8656f91b3e044261" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:12:56.687328 30895 raft_consensus.cc:3037] T ac2b98ace3534e8c8656f91b3e044261 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:56.689462 30895 raft_consensus.cc:2454] T ac2b98ace3534e8c8656f91b3e044261 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:12:56.689797 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac2b98ace3534e8c8656f91b3e044261" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:12:56.689855 30694 leader_election.cc:302] T ac2b98ace3534e8c8656f91b3e044261 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:56.689957 31027 raft_consensus.cc:3037] T ac2b98ace3534e8c8656f91b3e044261 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:56.690084 32280 raft_consensus.cc:2781] T ac2b98ace3534e8c8656f91b3e044261 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:56.690222 32280 raft_consensus.cc:686] T ac2b98ace3534e8c8656f91b3e044261 P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Becoming Leader. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Running, Role: LEADER
I20240503 08:12:56.690537 32280 consensus_queue.cc:237] T ac2b98ace3534e8c8656f91b3e044261 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:56.692000 31027 raft_consensus.cc:2454] T ac2b98ace3534e8c8656f91b3e044261 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:12:56.694017 27855 catalog_manager.cc:5484] T ac2b98ace3534e8c8656f91b3e044261 P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69). New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:12:56.810077 27855 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:56.816027 30895 raft_consensus.cc:1261] T ac2b98ace3534e8c8656f91b3e044261 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:56.816092 31027 raft_consensus.cc:1261] T ac2b98ace3534e8c8656f91b3e044261 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:56.816578 32157 consensus_queue.cc:1035] T ac2b98ace3534e8c8656f91b3e044261 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:56.816800 32280 consensus_queue.cc:1035] T ac2b98ace3534e8c8656f91b3e044261 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:56.820479 32276 tablet.cc:1722] T ac2b98ace3534e8c8656f91b3e044261 P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:56.820952 32276 tablet.cc:1620] T ac2b98ace3534e8c8656f91b3e044261 P 6d1d7de19459435fbde1999e3dfd226b: MemRowSet was empty: no flush needed.
I20240503 08:12:56.823340 27856 catalog_manager.cc:5576] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915) does not have the latest schema for tablet ac2b98ace3534e8c8656f91b3e044261 (table rebalancer_dynamic_table [id=e400d7b778084ee8b001d93eda4fa2ab]). Expected version 1 got 0
I20240503 08:12:56.823598 32275 tablet.cc:1722] T ac2b98ace3534e8c8656f91b3e044261 P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:56.824050 32275 tablet.cc:1620] T ac2b98ace3534e8c8656f91b3e044261 P ced21ca983b749d0b4d4f6a0e2426bf8: MemRowSet was empty: no flush needed.
I20240503 08:12:56.826972 27855 catalog_manager.cc:5576] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747) does not have the latest schema for tablet ac2b98ace3534e8c8656f91b3e044261 (table rebalancer_dynamic_table [id=e400d7b778084ee8b001d93eda4fa2ab]). Expected version 1 got 0
I20240503 08:12:56.828750 32323 tablet.cc:1722] T ac2b98ace3534e8c8656f91b3e044261 P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:56.829401 32323 tablet.cc:1620] T ac2b98ace3534e8c8656f91b3e044261 P ce1496a47d084365ad786cd09e472a2c: MemRowSet was empty: no flush needed.
I20240503 08:12:56.832067 27862 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=e400d7b778084ee8b001d93eda4fa2ab] alter complete (version 1)
I20240503 08:12:56.934355 32238 consensus_queue.cc:1035] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:56.937220 27862 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:56.938149 32308 consensus_queue.cc:1035] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:56.946316 32276 tablet.cc:1722] T ac2b98ace3534e8c8656f91b3e044261 P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:56.946878 32276 tablet.cc:1620] T ac2b98ace3534e8c8656f91b3e044261 P 6d1d7de19459435fbde1999e3dfd226b: MemRowSet was empty: no flush needed.
I20240503 08:12:56.949422 32275 tablet.cc:1722] T ac2b98ace3534e8c8656f91b3e044261 P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:56.949931 32275 tablet.cc:1620] T ac2b98ace3534e8c8656f91b3e044261 P ced21ca983b749d0b4d4f6a0e2426bf8: MemRowSet was empty: no flush needed.
I20240503 08:12:56.950359 32323 tablet.cc:1722] T ac2b98ace3534e8c8656f91b3e044261 P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:56.950791 32323 tablet.cc:1620] T ac2b98ace3534e8c8656f91b3e044261 P ce1496a47d084365ad786cd09e472a2c: MemRowSet was empty: no flush needed.
I20240503 08:12:56.959148 27856 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=e400d7b778084ee8b001d93eda4fa2ab] alter complete (version 2)
I20240503 08:12:56.978117 32308 consensus_queue.cc:1035] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:56.983098 32308 consensus_queue.cc:1035] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:57.063725 27855 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:57.071308 32276 tablet.cc:1722] T ac2b98ace3534e8c8656f91b3e044261 P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:57.071959 32275 tablet.cc:1722] T ac2b98ace3534e8c8656f91b3e044261 P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:57.077574 32323 tablet.cc:1722] T ac2b98ace3534e8c8656f91b3e044261 P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:57.083169 27855 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table_new_name [id=e400d7b778084ee8b001d93eda4fa2ab] alter complete (version 3)
I20240503 08:12:57.185483 27855 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:57.185981 27855 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:57.188906 27855 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet ac2b98ace3534e8c8656f91b3e044261
I20240503 08:12:57.189723 31007 tablet_service.cc:1514] Processing DeleteTablet for tablet ac2b98ace3534e8c8656f91b3e044261 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:57 UTC) from {username='slave'} at 127.0.0.1:47020
I20240503 08:12:57.189751 30872 tablet_service.cc:1514] Processing DeleteTablet for tablet ac2b98ace3534e8c8656f91b3e044261 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:57 UTC) from {username='slave'} at 127.0.0.1:34426
I20240503 08:12:57.189837 30740 tablet_service.cc:1514] Processing DeleteTablet for tablet ac2b98ace3534e8c8656f91b3e044261 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:57 UTC) from {username='slave'} at 127.0.0.1:46794
I20240503 08:12:57.190304 32333 tablet_replica.cc:330] T ac2b98ace3534e8c8656f91b3e044261 P ced21ca983b749d0b4d4f6a0e2426bf8: stopping tablet replica
I20240503 08:12:57.190510 32333 raft_consensus.cc:2229] T ac2b98ace3534e8c8656f91b3e044261 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:57.190544 32334 tablet_replica.cc:330] T ac2b98ace3534e8c8656f91b3e044261 P 6d1d7de19459435fbde1999e3dfd226b: stopping tablet replica
I20240503 08:12:57.190707 32333 raft_consensus.cc:2258] T ac2b98ace3534e8c8656f91b3e044261 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:57.190752 32334 raft_consensus.cc:2229] T ac2b98ace3534e8c8656f91b3e044261 P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:57.191113 32334 raft_consensus.cc:2258] T ac2b98ace3534e8c8656f91b3e044261 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:57.191166 32332 tablet_replica.cc:330] T ac2b98ace3534e8c8656f91b3e044261 P ce1496a47d084365ad786cd09e472a2c: stopping tablet replica
I20240503 08:12:57.191365 32332 raft_consensus.cc:2229] T ac2b98ace3534e8c8656f91b3e044261 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:57.191570 32332 raft_consensus.cc:2258] T ac2b98ace3534e8c8656f91b3e044261 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:57.191802 32334 ts_tablet_manager.cc:1875] T ac2b98ace3534e8c8656f91b3e044261 P 6d1d7de19459435fbde1999e3dfd226b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:57.191872 32333 ts_tablet_manager.cc:1875] T ac2b98ace3534e8c8656f91b3e044261 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:57.192319 32332 ts_tablet_manager.cc:1875] T ac2b98ace3534e8c8656f91b3e044261 P ce1496a47d084365ad786cd09e472a2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:57.196290 32334 ts_tablet_manager.cc:1888] T ac2b98ace3534e8c8656f91b3e044261 P 6d1d7de19459435fbde1999e3dfd226b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:57.196394 32334 log.cc:1196] T ac2b98ace3534e8c8656f91b3e044261 P 6d1d7de19459435fbde1999e3dfd226b: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal/wals/ac2b98ace3534e8c8656f91b3e044261
I20240503 08:12:57.196966 32334 ts_tablet_manager.cc:1909] T ac2b98ace3534e8c8656f91b3e044261 P 6d1d7de19459435fbde1999e3dfd226b: Deleting consensus metadata
I20240503 08:12:57.197898 27848 catalog_manager.cc:4830] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): tablet ac2b98ace3534e8c8656f91b3e044261 (table rebalancer_dynamic_table_new_name [id=e400d7b778084ee8b001d93eda4fa2ab]) successfully deleted
I20240503 08:12:57.198103 32332 ts_tablet_manager.cc:1888] T ac2b98ace3534e8c8656f91b3e044261 P ce1496a47d084365ad786cd09e472a2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:57.198253 32332 log.cc:1196] T ac2b98ace3534e8c8656f91b3e044261 P ce1496a47d084365ad786cd09e472a2c: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal/wals/ac2b98ace3534e8c8656f91b3e044261
I20240503 08:12:57.199054 32332 ts_tablet_manager.cc:1909] T ac2b98ace3534e8c8656f91b3e044261 P ce1496a47d084365ad786cd09e472a2c: Deleting consensus metadata
I20240503 08:12:57.200496 27846 catalog_manager.cc:4830] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): tablet ac2b98ace3534e8c8656f91b3e044261 (table rebalancer_dynamic_table_new_name [id=e400d7b778084ee8b001d93eda4fa2ab]) successfully deleted
I20240503 08:12:57.202299 32333 ts_tablet_manager.cc:1888] T ac2b98ace3534e8c8656f91b3e044261 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:57.202405 32333 log.cc:1196] T ac2b98ace3534e8c8656f91b3e044261 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal/wals/ac2b98ace3534e8c8656f91b3e044261
I20240503 08:12:57.202801 32333 ts_tablet_manager.cc:1909] T ac2b98ace3534e8c8656f91b3e044261 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting consensus metadata
I20240503 08:12:57.203691 27847 catalog_manager.cc:4830] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): tablet ac2b98ace3534e8c8656f91b3e044261 (table rebalancer_dynamic_table_new_name [id=e400d7b778084ee8b001d93eda4fa2ab]) successfully deleted
I20240503 08:12:57.207821 32312 consensus_queue.cc:1035] T ebe1a0b781404070a5326a80f8f9bfb4 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:57.211652 32312 consensus_queue.cc:1035] T ebe1a0b781404070a5326a80f8f9bfb4 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:57.238495 27862 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_extra_table_38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:57.247800 28111 tablet_service.cc:1467] Processing CreateTablet for tablet 23fc18f9417148c2a6d451bcca395338 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=1eba5f663686410fb06575b0f37c6126]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:57.247987 31007 tablet_service.cc:1467] Processing CreateTablet for tablet 23fc18f9417148c2a6d451bcca395338 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=1eba5f663686410fb06575b0f37c6126]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:57.248134 31140 tablet_service.cc:1467] Processing CreateTablet for tablet 23fc18f9417148c2a6d451bcca395338 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=1eba5f663686410fb06575b0f37c6126]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:57.248297 28111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23fc18f9417148c2a6d451bcca395338. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:57.248407 31007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23fc18f9417148c2a6d451bcca395338. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:57.248528 31140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23fc18f9417148c2a6d451bcca395338. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:57.252770 31008 tablet_service.cc:1467] Processing CreateTablet for tablet ab35ebabd6fe45fab890739a800794b1 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=1eba5f663686410fb06575b0f37c6126]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:57.253171 31008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab35ebabd6fe45fab890739a800794b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:57.253343 27983 tablet_service.cc:1467] Processing CreateTablet for tablet ab35ebabd6fe45fab890739a800794b1 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=1eba5f663686410fb06575b0f37c6126]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:57.253726 27983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab35ebabd6fe45fab890739a800794b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:57.260263 30740 tablet_service.cc:1467] Processing CreateTablet for tablet 3bc8f4968b9743baab5d25c8c7237c9c (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=1eba5f663686410fb06575b0f37c6126]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:57.260381 28112 tablet_service.cc:1467] Processing CreateTablet for tablet ab35ebabd6fe45fab890739a800794b1 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=1eba5f663686410fb06575b0f37c6126]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:57.260735 30740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bc8f4968b9743baab5d25c8c7237c9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:57.260825 28112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab35ebabd6fe45fab890739a800794b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:57.269140 31141 tablet_service.cc:1467] Processing CreateTablet for tablet 3bc8f4968b9743baab5d25c8c7237c9c (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=1eba5f663686410fb06575b0f37c6126]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:57.269639 31141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bc8f4968b9743baab5d25c8c7237c9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:57.269573 30872 tablet_service.cc:1467] Processing CreateTablet for tablet 3bc8f4968b9743baab5d25c8c7237c9c (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=1eba5f663686410fb06575b0f37c6126]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:57.270047 30872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bc8f4968b9743baab5d25c8c7237c9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:57.279053 32340 tablet_bootstrap.cc:492] T ab35ebabd6fe45fab890739a800794b1 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:57.280885 32340 tablet_bootstrap.cc:654] T ab35ebabd6fe45fab890739a800794b1 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:57.282596 32342 tablet_bootstrap.cc:492] T 3bc8f4968b9743baab5d25c8c7237c9c P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:57.284417 32342 tablet_bootstrap.cc:654] T 3bc8f4968b9743baab5d25c8c7237c9c P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:57.284886 32339 tablet_bootstrap.cc:492] T 23fc18f9417148c2a6d451bcca395338 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:57.288173 32339 tablet_bootstrap.cc:654] T 23fc18f9417148c2a6d451bcca395338 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:57.291095 32338 tablet_bootstrap.cc:492] T 23fc18f9417148c2a6d451bcca395338 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:57.291728 27855 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:57.292052 32340 tablet_bootstrap.cc:492] T ab35ebabd6fe45fab890739a800794b1 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:57.292215 32340 ts_tablet_manager.cc:1367] T ab35ebabd6fe45fab890739a800794b1 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240503 08:12:57.293067 32340 raft_consensus.cc:348] T ab35ebabd6fe45fab890739a800794b1 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:57.293292 32340 raft_consensus.cc:374] T ab35ebabd6fe45fab890739a800794b1 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:57.293382 32340 raft_consensus.cc:729] T ab35ebabd6fe45fab890739a800794b1 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:57.294764 32338 tablet_bootstrap.cc:654] T 23fc18f9417148c2a6d451bcca395338 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:57.293622 32340 consensus_queue.cc:260] T ab35ebabd6fe45fab890739a800794b1 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:57.298017 32340 ts_tablet_manager.cc:1398] T ab35ebabd6fe45fab890739a800794b1 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240503 08:12:57.301982 31141 tablet_service.cc:1467] Processing CreateTablet for tablet 145d2467d33146878273b21a717a3eda (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fa945b7b6372466aad42465594a5801e]), partition=
I20240503 08:12:57.302397 31141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 145d2467d33146878273b21a717a3eda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:57.302703 30740 tablet_service.cc:1467] Processing CreateTablet for tablet 145d2467d33146878273b21a717a3eda (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fa945b7b6372466aad42465594a5801e]), partition=
I20240503 08:12:57.303277 28113 tablet_service.cc:1467] Processing CreateTablet for tablet 145d2467d33146878273b21a717a3eda (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fa945b7b6372466aad42465594a5801e]), partition=
I20240503 08:12:57.303611 28113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 145d2467d33146878273b21a717a3eda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:57.309576 32341 tablet_bootstrap.cc:492] T 23fc18f9417148c2a6d451bcca395338 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:57.310117 30740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 145d2467d33146878273b21a717a3eda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:57.315379 32341 tablet_bootstrap.cc:654] T 23fc18f9417148c2a6d451bcca395338 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:57.317138 32346 tablet_bootstrap.cc:492] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:57.318758 32339 tablet_bootstrap.cc:492] T 23fc18f9417148c2a6d451bcca395338 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:57.318967 32339 ts_tablet_manager.cc:1367] T 23fc18f9417148c2a6d451bcca395338 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.034s user 0.009s sys 0.001s
I20240503 08:12:57.319000 32346 tablet_bootstrap.cc:654] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:57.319904 32339 raft_consensus.cc:348] T 23fc18f9417148c2a6d451bcca395338 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:57.320250 32339 raft_consensus.cc:374] T 23fc18f9417148c2a6d451bcca395338 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:57.320405 32339 raft_consensus.cc:729] T 23fc18f9417148c2a6d451bcca395338 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:57.320932 32339 consensus_queue.cc:260] T 23fc18f9417148c2a6d451bcca395338 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:57.321689 32339 ts_tablet_manager.cc:1398] T 23fc18f9417148c2a6d451bcca395338 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:12:57.322129 32339 tablet_bootstrap.cc:492] T ab35ebabd6fe45fab890739a800794b1 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:57.324227 32339 tablet_bootstrap.cc:654] T ab35ebabd6fe45fab890739a800794b1 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:57.333940 32338 tablet_bootstrap.cc:492] T 23fc18f9417148c2a6d451bcca395338 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:57.335104 32338 ts_tablet_manager.cc:1367] T 23fc18f9417148c2a6d451bcca395338 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.007s
I20240503 08:12:57.335985 32338 raft_consensus.cc:348] T 23fc18f9417148c2a6d451bcca395338 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:57.337590 32338 raft_consensus.cc:374] T 23fc18f9417148c2a6d451bcca395338 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:57.337791 32338 raft_consensus.cc:729] T 23fc18f9417148c2a6d451bcca395338 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:57.338531 32338 consensus_queue.cc:260] T 23fc18f9417148c2a6d451bcca395338 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:57.342329 32338 ts_tablet_manager.cc:1398] T 23fc18f9417148c2a6d451bcca395338 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240503 08:12:57.343322 32338 tablet_bootstrap.cc:492] T ab35ebabd6fe45fab890739a800794b1 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:57.345817 32342 tablet_bootstrap.cc:492] T 3bc8f4968b9743baab5d25c8c7237c9c P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:57.346125 32342 ts_tablet_manager.cc:1367] T 3bc8f4968b9743baab5d25c8c7237c9c P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.064s user 0.029s sys 0.013s
I20240503 08:12:57.347388 32342 raft_consensus.cc:348] T 3bc8f4968b9743baab5d25c8c7237c9c P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:57.347685 32342 raft_consensus.cc:374] T 3bc8f4968b9743baab5d25c8c7237c9c P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:57.347831 32342 raft_consensus.cc:729] T 3bc8f4968b9743baab5d25c8c7237c9c P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:57.348111 32342 consensus_queue.cc:260] T 3bc8f4968b9743baab5d25c8c7237c9c P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:57.348851 32342 ts_tablet_manager.cc:1398] T 3bc8f4968b9743baab5d25c8c7237c9c P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:57.349213 32342 tablet_bootstrap.cc:492] T 145d2467d33146878273b21a717a3eda P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:57.349581 32339 tablet_bootstrap.cc:492] T ab35ebabd6fe45fab890739a800794b1 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:57.349819 32339 ts_tablet_manager.cc:1367] T ab35ebabd6fe45fab890739a800794b1 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.028s user 0.015s sys 0.009s
I20240503 08:12:57.350708 32342 tablet_bootstrap.cc:654] T 145d2467d33146878273b21a717a3eda P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:57.350731 32338 tablet_bootstrap.cc:654] T ab35ebabd6fe45fab890739a800794b1 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:57.350720 32339 raft_consensus.cc:348] T ab35ebabd6fe45fab890739a800794b1 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:57.351006 32339 raft_consensus.cc:374] T ab35ebabd6fe45fab890739a800794b1 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:57.351119 32339 raft_consensus.cc:729] T ab35ebabd6fe45fab890739a800794b1 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:57.352121 32339 consensus_queue.cc:260] T ab35ebabd6fe45fab890739a800794b1 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:57.352950 32339 ts_tablet_manager.cc:1398] T ab35ebabd6fe45fab890739a800794b1 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 08:12:57.358956 32338 tablet_bootstrap.cc:492] T ab35ebabd6fe45fab890739a800794b1 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:57.359102 32338 ts_tablet_manager.cc:1367] T ab35ebabd6fe45fab890739a800794b1 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.002s
I20240503 08:12:57.359648 32338 raft_consensus.cc:348] T ab35ebabd6fe45fab890739a800794b1 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:57.359810 32338 raft_consensus.cc:374] T ab35ebabd6fe45fab890739a800794b1 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:57.359881 32338 raft_consensus.cc:729] T ab35ebabd6fe45fab890739a800794b1 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:57.360093 32338 consensus_queue.cc:260] T ab35ebabd6fe45fab890739a800794b1 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:57.360175 32346 tablet_bootstrap.cc:492] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:57.360353 32346 ts_tablet_manager.cc:1367] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.043s user 0.006s sys 0.000s
I20240503 08:12:57.360589 32338 ts_tablet_manager.cc:1398] T ab35ebabd6fe45fab890739a800794b1 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 08:12:57.361047 32346 raft_consensus.cc:348] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:57.361244 32338 tablet_bootstrap.cc:492] T 145d2467d33146878273b21a717a3eda P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:57.361562 32342 tablet_bootstrap.cc:492] T 145d2467d33146878273b21a717a3eda P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:57.361744 32342 ts_tablet_manager.cc:1367] T 145d2467d33146878273b21a717a3eda P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.007s
I20240503 08:12:57.362551 32342 raft_consensus.cc:348] T 145d2467d33146878273b21a717a3eda P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:57.362748 32341 tablet_bootstrap.cc:492] T 23fc18f9417148c2a6d451bcca395338 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:57.362771 32342 raft_consensus.cc:374] T 145d2467d33146878273b21a717a3eda P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:57.362771 32338 tablet_bootstrap.cc:654] T 145d2467d33146878273b21a717a3eda P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:57.362847 32342 raft_consensus.cc:729] T 145d2467d33146878273b21a717a3eda P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:57.362905 32341 ts_tablet_manager.cc:1367] T 23fc18f9417148c2a6d451bcca395338 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.003s
I20240503 08:12:57.363082 32342 consensus_queue.cc:260] T 145d2467d33146878273b21a717a3eda P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:57.363929 32342 ts_tablet_manager.cc:1398] T 145d2467d33146878273b21a717a3eda P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:12:57.364735 32341 raft_consensus.cc:348] T 23fc18f9417148c2a6d451bcca395338 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:57.364936 32341 raft_consensus.cc:374] T 23fc18f9417148c2a6d451bcca395338 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:57.365181 32341 raft_consensus.cc:729] T 23fc18f9417148c2a6d451bcca395338 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:57.365401 32341 consensus_queue.cc:260] T 23fc18f9417148c2a6d451bcca395338 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:57.365931 32341 ts_tablet_manager.cc:1398] T 23fc18f9417148c2a6d451bcca395338 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 08:12:57.366441 32338 tablet_bootstrap.cc:492] T 145d2467d33146878273b21a717a3eda P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:57.366582 32338 ts_tablet_manager.cc:1367] T 145d2467d33146878273b21a717a3eda P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 08:12:57.366729 32346 raft_consensus.cc:374] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:57.367213 32341 tablet_bootstrap.cc:492] T 3bc8f4968b9743baab5d25c8c7237c9c P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:57.367412 32346 raft_consensus.cc:729] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:57.367671 32346 consensus_queue.cc:260] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:57.368232 32346 ts_tablet_manager.cc:1398] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240503 08:12:57.368942 32338 raft_consensus.cc:348] T 145d2467d33146878273b21a717a3eda P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:57.369175 32338 raft_consensus.cc:374] T 145d2467d33146878273b21a717a3eda P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:57.369307 32338 raft_consensus.cc:729] T 145d2467d33146878273b21a717a3eda P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:57.369544 32338 consensus_queue.cc:260] T 145d2467d33146878273b21a717a3eda P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:57.370258 32338 ts_tablet_manager.cc:1398] T 145d2467d33146878273b21a717a3eda P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:57.371840 32341 tablet_bootstrap.cc:654] T 3bc8f4968b9743baab5d25c8c7237c9c P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:57.375653 32341 tablet_bootstrap.cc:492] T 3bc8f4968b9743baab5d25c8c7237c9c P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:57.375965 32341 ts_tablet_manager.cc:1367] T 3bc8f4968b9743baab5d25c8c7237c9c P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
I20240503 08:12:57.376583 32341 raft_consensus.cc:348] T 3bc8f4968b9743baab5d25c8c7237c9c P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:57.376778 32341 raft_consensus.cc:374] T 3bc8f4968b9743baab5d25c8c7237c9c P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:57.376854 32341 raft_consensus.cc:729] T 3bc8f4968b9743baab5d25c8c7237c9c P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:57.377056 32341 consensus_queue.cc:260] T 3bc8f4968b9743baab5d25c8c7237c9c P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:57.377523 32341 ts_tablet_manager.cc:1398] T 3bc8f4968b9743baab5d25c8c7237c9c P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:12:57.377986 32341 tablet_bootstrap.cc:492] T 145d2467d33146878273b21a717a3eda P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:57.379259 32341 tablet_bootstrap.cc:654] T 145d2467d33146878273b21a717a3eda P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:57.381021 32341 tablet_bootstrap.cc:492] T 145d2467d33146878273b21a717a3eda P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:57.381136 32341 ts_tablet_manager.cc:1367] T 145d2467d33146878273b21a717a3eda P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:57.382292 32341 raft_consensus.cc:348] T 145d2467d33146878273b21a717a3eda P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:57.382496 32341 raft_consensus.cc:374] T 145d2467d33146878273b21a717a3eda P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:57.382591 32341 raft_consensus.cc:729] T 145d2467d33146878273b21a717a3eda P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:57.382800 32341 consensus_queue.cc:260] T 145d2467d33146878273b21a717a3eda P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:57.383302 32341 ts_tablet_manager.cc:1398] T 145d2467d33146878273b21a717a3eda P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:57.396027 32280 raft_consensus.cc:482] T 145d2467d33146878273b21a717a3eda P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:57.396169 32280 raft_consensus.cc:504] T 145d2467d33146878273b21a717a3eda P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:57.396896 32280 leader_election.cc:288] T 145d2467d33146878273b21a717a3eda P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:57.397265 28131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "145d2467d33146878273b21a717a3eda" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:12:57.397354 31160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "145d2467d33146878273b21a717a3eda" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:12:57.397493 28131 raft_consensus.cc:2454] T 145d2467d33146878273b21a717a3eda P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:12:57.397564 31160 raft_consensus.cc:2454] T 145d2467d33146878273b21a717a3eda P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:12:57.397979 30696 leader_election.cc:302] T 145d2467d33146878273b21a717a3eda P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 6d1d7de19459435fbde1999e3dfd226b, b00f244c69df401997ea99d492218a98; no voters:
I20240503 08:12:57.398249 32280 raft_consensus.cc:2781] T 145d2467d33146878273b21a717a3eda P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:57.398370 32280 raft_consensus.cc:482] T 145d2467d33146878273b21a717a3eda P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:57.398447 32280 raft_consensus.cc:3037] T 145d2467d33146878273b21a717a3eda P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:57.400329 32280 raft_consensus.cc:504] T 145d2467d33146878273b21a717a3eda P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:57.400857 32280 leader_election.cc:288] T 145d2467d33146878273b21a717a3eda P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Requested vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:57.401227 28131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "145d2467d33146878273b21a717a3eda" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:12:57.401294 31160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "145d2467d33146878273b21a717a3eda" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:12:57.401441 31160 raft_consensus.cc:3037] T 145d2467d33146878273b21a717a3eda P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:57.401433 28131 raft_consensus.cc:3037] T 145d2467d33146878273b21a717a3eda P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:57.404155 28131 raft_consensus.cc:2454] T 145d2467d33146878273b21a717a3eda P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:12:57.404165 31160 raft_consensus.cc:2454] T 145d2467d33146878273b21a717a3eda P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:12:57.404522 30694 leader_election.cc:302] T 145d2467d33146878273b21a717a3eda P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, b00f244c69df401997ea99d492218a98; no voters:
I20240503 08:12:57.404809 32280 raft_consensus.cc:2781] T 145d2467d33146878273b21a717a3eda P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:57.404947 32280 raft_consensus.cc:686] T 145d2467d33146878273b21a717a3eda P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Becoming Leader. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Running, Role: LEADER
I20240503 08:12:57.405195 32280 consensus_queue.cc:237] T 145d2467d33146878273b21a717a3eda P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:12:57.408478 27862 catalog_manager.cc:5484] T 145d2467d33146878273b21a717a3eda P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69). New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:12:57.498281 32324 raft_consensus.cc:482] T 23fc18f9417148c2a6d451bcca395338 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:57.498453 32324 raft_consensus.cc:504] T 23fc18f9417148c2a6d451bcca395338 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:57.499231 32324 leader_election.cc:288] T 23fc18f9417148c2a6d451bcca395338 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:57.499665 28131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23fc18f9417148c2a6d451bcca395338" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:12:57.499897 28131 raft_consensus.cc:2454] T 23fc18f9417148c2a6d451bcca395338 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 0.
I20240503 08:12:57.499903 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23fc18f9417148c2a6d451bcca395338" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:57.500114 31027 raft_consensus.cc:2454] T 23fc18f9417148c2a6d451bcca395338 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 0.
I20240503 08:12:57.500348 31096 leader_election.cc:302] T 23fc18f9417148c2a6d451bcca395338 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, b00f244c69df401997ea99d492218a98; no voters:
I20240503 08:12:57.500597 32324 raft_consensus.cc:2781] T 23fc18f9417148c2a6d451bcca395338 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:57.500708 32324 raft_consensus.cc:482] T 23fc18f9417148c2a6d451bcca395338 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:57.500793 32324 raft_consensus.cc:3037] T 23fc18f9417148c2a6d451bcca395338 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:57.503163 32324 raft_consensus.cc:504] T 23fc18f9417148c2a6d451bcca395338 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:57.503767 32324 leader_election.cc:288] T 23fc18f9417148c2a6d451bcca395338 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 election: Requested vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:57.504171 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23fc18f9417148c2a6d451bcca395338" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:12:57.504173 28131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23fc18f9417148c2a6d451bcca395338" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:12:57.504357 28131 raft_consensus.cc:3037] T 23fc18f9417148c2a6d451bcca395338 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:57.504357 31027 raft_consensus.cc:3037] T 23fc18f9417148c2a6d451bcca395338 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:57.506996 28131 raft_consensus.cc:2454] T 23fc18f9417148c2a6d451bcca395338 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 1.
I20240503 08:12:57.506995 31027 raft_consensus.cc:2454] T 23fc18f9417148c2a6d451bcca395338 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 1.
I20240503 08:12:57.507329 31095 leader_election.cc:302] T 23fc18f9417148c2a6d451bcca395338 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:57.507606 32324 raft_consensus.cc:2781] T 23fc18f9417148c2a6d451bcca395338 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:57.507933 32324 raft_consensus.cc:686] T 23fc18f9417148c2a6d451bcca395338 P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Becoming Leader. State: Replica: b00f244c69df401997ea99d492218a98, State: Running, Role: LEADER
I20240503 08:12:57.508211 32324 consensus_queue.cc:237] T 23fc18f9417148c2a6d451bcca395338 P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:57.511374 27862 catalog_manager.cc:5484] T 23fc18f9417148c2a6d451bcca395338 P b00f244c69df401997ea99d492218a98 reported cstate change: term changed from 0 to 1, leader changed from <none> to b00f244c69df401997ea99d492218a98 (127.20.230.72). New cstate: current_term: 1 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:12:57.514741 32312 raft_consensus.cc:482] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:57.514891 32312 raft_consensus.cc:504] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:57.515512 32312 leader_election.cc:288] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:57.515877 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bc8f4968b9743baab5d25c8c7237c9c" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:12:57.515905 31160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bc8f4968b9743baab5d25c8c7237c9c" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:12:57.516103 31160 raft_consensus.cc:2454] T 3bc8f4968b9743baab5d25c8c7237c9c P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:12:57.516112 30762 raft_consensus.cc:2454] T 3bc8f4968b9743baab5d25c8c7237c9c P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:12:57.516419 30827 leader_election.cc:302] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:57.516733 32312 raft_consensus.cc:2781] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:57.516845 32312 raft_consensus.cc:482] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:57.516920 32312 raft_consensus.cc:3037] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:57.518865 32312 raft_consensus.cc:504] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:57.519388 32312 leader_election.cc:288] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Requested vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:57.519748 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bc8f4968b9743baab5d25c8c7237c9c" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:12:57.519874 31160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bc8f4968b9743baab5d25c8c7237c9c" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:12:57.519932 30762 raft_consensus.cc:3037] T 3bc8f4968b9743baab5d25c8c7237c9c P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:57.520066 31160 raft_consensus.cc:3037] T 3bc8f4968b9743baab5d25c8c7237c9c P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:57.521713 30762 raft_consensus.cc:2454] T 3bc8f4968b9743baab5d25c8c7237c9c P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:12:57.522153 30829 leader_election.cc:302] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:57.522432 32312 raft_consensus.cc:2781] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:57.522850 32312 raft_consensus.cc:686] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Becoming Leader. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Running, Role: LEADER
I20240503 08:12:57.523111 27862 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:57.523159 32312 consensus_queue.cc:237] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:57.523607 31160 raft_consensus.cc:2454] T 3bc8f4968b9743baab5d25c8c7237c9c P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:12:57.532364 28131 raft_consensus.cc:1261] T 145d2467d33146878273b21a717a3eda P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:57.532370 31159 raft_consensus.cc:1261] T 145d2467d33146878273b21a717a3eda P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:57.533259 32319 consensus_queue.cc:1035] T 145d2467d33146878273b21a717a3eda P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:57.534916 32319 consensus_queue.cc:1035] T 145d2467d33146878273b21a717a3eda P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:57.539665 27855 catalog_manager.cc:5484] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c reported cstate change: term changed from 0 to 1, leader changed from <none> to ce1496a47d084365ad786cd09e472a2c (127.20.230.70). New cstate: current_term: 1 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:12:57.541814 32276 tablet.cc:1722] T 145d2467d33146878273b21a717a3eda P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:57.542225 32276 tablet.cc:1620] T 145d2467d33146878273b21a717a3eda P 6d1d7de19459435fbde1999e3dfd226b: MemRowSet was empty: no flush needed.
I20240503 08:12:57.546933 27855 catalog_manager.cc:5576] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143) does not have the latest schema for tablet 145d2467d33146878273b21a717a3eda (table rebalancer_dynamic_table [id=fa945b7b6372466aad42465594a5801e]). Expected version 1 got 0
I20240503 08:12:57.550779 27854 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=fa945b7b6372466aad42465594a5801e] alter complete (version 1)
I20240503 08:12:57.553413 32365 tablet.cc:1722] T 145d2467d33146878273b21a717a3eda P 1b411c6bb25549368041773d7b710d48: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:57.554080 32365 tablet.cc:1620] T 145d2467d33146878273b21a717a3eda P 1b411c6bb25549368041773d7b710d48: MemRowSet was empty: no flush needed.
I20240503 08:12:57.555119 32366 tablet.cc:1722] T 145d2467d33146878273b21a717a3eda P b00f244c69df401997ea99d492218a98: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:57.555605 27854 catalog_manager.cc:5576] TS b00f244c69df401997ea99d492218a98 (127.20.230.72:40893) does not have the latest schema for tablet 145d2467d33146878273b21a717a3eda (table rebalancer_dynamic_table [id=fa945b7b6372466aad42465594a5801e]). Expected version 1 got 0
I20240503 08:12:57.555851 32366 tablet.cc:1620] T 145d2467d33146878273b21a717a3eda P b00f244c69df401997ea99d492218a98: MemRowSet was empty: no flush needed.
I20240503 08:12:57.629863 32360 raft_consensus.cc:482] T ab35ebabd6fe45fab890739a800794b1 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:57.630028 32360 raft_consensus.cc:504] T ab35ebabd6fe45fab890739a800794b1 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:57.631078 32360 leader_election.cc:288] T ab35ebabd6fe45fab890739a800794b1 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:57.631181 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab35ebabd6fe45fab890739a800794b1" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:12:57.631444 28002 raft_consensus.cc:2454] T ab35ebabd6fe45fab890739a800794b1 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:57.632019 28069 leader_election.cc:302] T ab35ebabd6fe45fab890739a800794b1 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 833582ff68c94269be364ebd187a75db; no voters:
I20240503 08:12:57.632653 32360 raft_consensus.cc:2781] T ab35ebabd6fe45fab890739a800794b1 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:57.632797 32360 raft_consensus.cc:482] T ab35ebabd6fe45fab890739a800794b1 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:57.632884 32360 raft_consensus.cc:3037] T ab35ebabd6fe45fab890739a800794b1 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:57.633100 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab35ebabd6fe45fab890739a800794b1" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:57.633469 31027 raft_consensus.cc:2454] T ab35ebabd6fe45fab890739a800794b1 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:57.635838 32360 raft_consensus.cc:504] T ab35ebabd6fe45fab890739a800794b1 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:57.637866 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab35ebabd6fe45fab890739a800794b1" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:12:57.638067 28002 raft_consensus.cc:3037] T ab35ebabd6fe45fab890739a800794b1 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:57.638533 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab35ebabd6fe45fab890739a800794b1" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:12:57.638726 31027 raft_consensus.cc:3037] T ab35ebabd6fe45fab890739a800794b1 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:57.640723 28002 raft_consensus.cc:2454] T ab35ebabd6fe45fab890739a800794b1 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:12:57.641058 28069 leader_election.cc:302] T ab35ebabd6fe45fab890739a800794b1 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 833582ff68c94269be364ebd187a75db; no voters:
I20240503 08:12:57.641340 32309 raft_consensus.cc:2781] T ab35ebabd6fe45fab890739a800794b1 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:57.646553 32360 leader_election.cc:288] T ab35ebabd6fe45fab890739a800794b1 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Requested vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:57.648420 31027 raft_consensus.cc:2454] T ab35ebabd6fe45fab890739a800794b1 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:12:57.650014 32309 raft_consensus.cc:686] T ab35ebabd6fe45fab890739a800794b1 P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Becoming Leader. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Running, Role: LEADER
I20240503 08:12:57.650353 32309 consensus_queue.cc:237] T ab35ebabd6fe45fab890739a800794b1 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:57.654173 27855 catalog_manager.cc:5484] T ab35ebabd6fe45fab890739a800794b1 P 1b411c6bb25549368041773d7b710d48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b411c6bb25549368041773d7b710d48 (127.20.230.66). New cstate: current_term: 1 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:12:57.664240 27855 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:57.671473 32276 tablet.cc:1722] T 145d2467d33146878273b21a717a3eda P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:57.671877 32366 tablet.cc:1722] T 145d2467d33146878273b21a717a3eda P b00f244c69df401997ea99d492218a98: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:57.671978 32276 tablet.cc:1620] T 145d2467d33146878273b21a717a3eda P 6d1d7de19459435fbde1999e3dfd226b: MemRowSet was empty: no flush needed.
I20240503 08:12:57.672314 32366 tablet.cc:1620] T 145d2467d33146878273b21a717a3eda P b00f244c69df401997ea99d492218a98: MemRowSet was empty: no flush needed.
I20240503 08:12:57.672370 32365 tablet.cc:1722] T 145d2467d33146878273b21a717a3eda P 1b411c6bb25549368041773d7b710d48: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:57.672804 32365 tablet.cc:1620] T 145d2467d33146878273b21a717a3eda P 1b411c6bb25549368041773d7b710d48: MemRowSet was empty: no flush needed.
I20240503 08:12:57.683845 27854 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=fa945b7b6372466aad42465594a5801e] alter complete (version 2)
I20240503 08:12:57.789294 27862 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:57.796489 32276 tablet.cc:1722] T 145d2467d33146878273b21a717a3eda P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:57.797204 32366 tablet.cc:1722] T 145d2467d33146878273b21a717a3eda P b00f244c69df401997ea99d492218a98: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:57.797420 32365 tablet.cc:1722] T 145d2467d33146878273b21a717a3eda P 1b411c6bb25549368041773d7b710d48: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:57.805292 27862 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table_new_name [id=fa945b7b6372466aad42465594a5801e] alter complete (version 3)
I20240503 08:12:57.907635 27854 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:57.907899 27854 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:57.911037 27854 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet 145d2467d33146878273b21a717a3eda
I20240503 08:12:57.911854 28113 tablet_service.cc:1514] Processing DeleteTablet for tablet 145d2467d33146878273b21a717a3eda with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:57 UTC) from {username='slave'} at 127.0.0.1:57912
I20240503 08:12:57.911870 31140 tablet_service.cc:1514] Processing DeleteTablet for tablet 145d2467d33146878273b21a717a3eda with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:57 UTC) from {username='slave'} at 127.0.0.1:45814
I20240503 08:12:57.912310 30740 tablet_service.cc:1514] Processing DeleteTablet for tablet 145d2467d33146878273b21a717a3eda with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:57 UTC) from {username='slave'} at 127.0.0.1:46794
I20240503 08:12:57.913141 32371 tablet_replica.cc:330] T 145d2467d33146878273b21a717a3eda P 1b411c6bb25549368041773d7b710d48: stopping tablet replica
I20240503 08:12:57.913334 32371 raft_consensus.cc:2229] T 145d2467d33146878273b21a717a3eda P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:57.913523 32371 raft_consensus.cc:2258] T 145d2467d33146878273b21a717a3eda P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:57.914026 32372 tablet_replica.cc:330] T 145d2467d33146878273b21a717a3eda P b00f244c69df401997ea99d492218a98: stopping tablet replica
I20240503 08:12:57.914222 32373 tablet_replica.cc:330] T 145d2467d33146878273b21a717a3eda P 6d1d7de19459435fbde1999e3dfd226b: stopping tablet replica
I20240503 08:12:57.914232 32372 raft_consensus.cc:2229] T 145d2467d33146878273b21a717a3eda P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:57.914424 32372 raft_consensus.cc:2258] T 145d2467d33146878273b21a717a3eda P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:57.914515 32371 ts_tablet_manager.cc:1875] T 145d2467d33146878273b21a717a3eda P 1b411c6bb25549368041773d7b710d48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:57.914521 32373 raft_consensus.cc:2229] T 145d2467d33146878273b21a717a3eda P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:57.914906 32373 raft_consensus.cc:2258] T 145d2467d33146878273b21a717a3eda P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:57.915217 32372 ts_tablet_manager.cc:1875] T 145d2467d33146878273b21a717a3eda P b00f244c69df401997ea99d492218a98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:57.915807 32373 ts_tablet_manager.cc:1875] T 145d2467d33146878273b21a717a3eda P 6d1d7de19459435fbde1999e3dfd226b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:57.920068 32371 ts_tablet_manager.cc:1888] T 145d2467d33146878273b21a717a3eda P 1b411c6bb25549368041773d7b710d48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:57.920172 32371 log.cc:1196] T 145d2467d33146878273b21a717a3eda P 1b411c6bb25549368041773d7b710d48: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/145d2467d33146878273b21a717a3eda
I20240503 08:12:57.920571 32371 ts_tablet_manager.cc:1909] T 145d2467d33146878273b21a717a3eda P 1b411c6bb25549368041773d7b710d48: Deleting consensus metadata
I20240503 08:12:57.921675 27848 catalog_manager.cc:4830] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): tablet 145d2467d33146878273b21a717a3eda (table rebalancer_dynamic_table_new_name [id=fa945b7b6372466aad42465594a5801e]) successfully deleted
I20240503 08:12:57.922091 32372 ts_tablet_manager.cc:1888] T 145d2467d33146878273b21a717a3eda P b00f244c69df401997ea99d492218a98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:57.922245 32372 log.cc:1196] T 145d2467d33146878273b21a717a3eda P b00f244c69df401997ea99d492218a98: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal/wals/145d2467d33146878273b21a717a3eda
I20240503 08:12:57.922247 32324 consensus_queue.cc:1035] T 23fc18f9417148c2a6d451bcca395338 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:57.922792 32373 ts_tablet_manager.cc:1888] T 145d2467d33146878273b21a717a3eda P 6d1d7de19459435fbde1999e3dfd226b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:57.922816 32372 ts_tablet_manager.cc:1909] T 145d2467d33146878273b21a717a3eda P b00f244c69df401997ea99d492218a98: Deleting consensus metadata
I20240503 08:12:57.923935 27846 catalog_manager.cc:4830] TS b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): tablet 145d2467d33146878273b21a717a3eda (table rebalancer_dynamic_table_new_name [id=fa945b7b6372466aad42465594a5801e]) successfully deleted
I20240503 08:12:57.924185 32373 log.cc:1196] T 145d2467d33146878273b21a717a3eda P 6d1d7de19459435fbde1999e3dfd226b: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal/wals/145d2467d33146878273b21a717a3eda
I20240503 08:12:57.924654 32373 ts_tablet_manager.cc:1909] T 145d2467d33146878273b21a717a3eda P 6d1d7de19459435fbde1999e3dfd226b: Deleting consensus metadata
I20240503 08:12:57.925666 27848 catalog_manager.cc:4830] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): tablet 145d2467d33146878273b21a717a3eda (table rebalancer_dynamic_table_new_name [id=fa945b7b6372466aad42465594a5801e]) successfully deleted
I20240503 08:12:57.935122 32357 consensus_queue.cc:1035] T 23fc18f9417148c2a6d451bcca395338 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:57.991601 32312 consensus_queue.cc:1035] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:57.998592 32311 consensus_queue.cc:1035] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:58.013633 27855 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:58.019757 30740 tablet_service.cc:1467] Processing CreateTablet for tablet 424e5ae48de94f4097879574940e11ee (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b8e0b303c65c458d87cc9745c161e5ff]), partition=
I20240503 08:12:58.020038 31007 tablet_service.cc:1467] Processing CreateTablet for tablet 424e5ae48de94f4097879574940e11ee (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b8e0b303c65c458d87cc9745c161e5ff]), partition=
I20240503 08:12:58.020087 30740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 424e5ae48de94f4097879574940e11ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:58.020166 30872 tablet_service.cc:1467] Processing CreateTablet for tablet 424e5ae48de94f4097879574940e11ee (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b8e0b303c65c458d87cc9745c161e5ff]), partition=
I20240503 08:12:58.020402 31007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 424e5ae48de94f4097879574940e11ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:58.020509 30872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 424e5ae48de94f4097879574940e11ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:58.025853 32381 tablet_bootstrap.cc:492] T 424e5ae48de94f4097879574940e11ee P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:58.027693 32381 tablet_bootstrap.cc:654] T 424e5ae48de94f4097879574940e11ee P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:58.027997 32382 tablet_bootstrap.cc:492] T 424e5ae48de94f4097879574940e11ee P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:58.029409 32382 tablet_bootstrap.cc:654] T 424e5ae48de94f4097879574940e11ee P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:58.031000 32383 tablet_bootstrap.cc:492] T 424e5ae48de94f4097879574940e11ee P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:58.031807 32381 tablet_bootstrap.cc:492] T 424e5ae48de94f4097879574940e11ee P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:58.031975 32381 ts_tablet_manager.cc:1367] T 424e5ae48de94f4097879574940e11ee P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 08:12:58.032438 32383 tablet_bootstrap.cc:654] T 424e5ae48de94f4097879574940e11ee P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:58.032688 32381 raft_consensus.cc:348] T 424e5ae48de94f4097879574940e11ee P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:58.032892 32381 raft_consensus.cc:374] T 424e5ae48de94f4097879574940e11ee P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:58.033022 32381 raft_consensus.cc:729] T 424e5ae48de94f4097879574940e11ee P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:58.033277 32381 consensus_queue.cc:260] T 424e5ae48de94f4097879574940e11ee P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:58.033975 32381 ts_tablet_manager.cc:1398] T 424e5ae48de94f4097879574940e11ee P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:58.038604 32383 tablet_bootstrap.cc:492] T 424e5ae48de94f4097879574940e11ee P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:58.038779 32383 ts_tablet_manager.cc:1367] T 424e5ae48de94f4097879574940e11ee P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240503 08:12:58.039373 32382 tablet_bootstrap.cc:492] T 424e5ae48de94f4097879574940e11ee P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:58.039558 32382 ts_tablet_manager.cc:1367] T 424e5ae48de94f4097879574940e11ee P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.006s
I20240503 08:12:58.039681 32383 raft_consensus.cc:348] T 424e5ae48de94f4097879574940e11ee P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:58.039933 32383 raft_consensus.cc:374] T 424e5ae48de94f4097879574940e11ee P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:58.040066 32383 raft_consensus.cc:729] T 424e5ae48de94f4097879574940e11ee P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:58.040328 32383 consensus_queue.cc:260] T 424e5ae48de94f4097879574940e11ee P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:58.040443 32382 raft_consensus.cc:348] T 424e5ae48de94f4097879574940e11ee P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:58.040796 32382 raft_consensus.cc:374] T 424e5ae48de94f4097879574940e11ee P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:58.040869 32383 ts_tablet_manager.cc:1398] T 424e5ae48de94f4097879574940e11ee P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:58.040923 32382 raft_consensus.cc:729] T 424e5ae48de94f4097879574940e11ee P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:58.041179 32382 consensus_queue.cc:260] T 424e5ae48de94f4097879574940e11ee P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:58.041702 32382 ts_tablet_manager.cc:1398] T 424e5ae48de94f4097879574940e11ee P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:12:58.097688 32367 consensus_queue.cc:1035] T ab35ebabd6fe45fab890739a800794b1 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:58.104728 32309 consensus_queue.cc:1035] T ab35ebabd6fe45fab890739a800794b1 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:58.161530 27855 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_extra_table_40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:58.171310 31007 tablet_service.cc:1467] Processing CreateTablet for tablet f1b2f3b8111649a7b5725647b8251698 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=3aef15bedeb544e9a15460f0b11c368a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:58.171418 31008 tablet_service.cc:1467] Processing CreateTablet for tablet b40a43ad7ed44c30a070189ebfe5aab6 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=3aef15bedeb544e9a15460f0b11c368a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:58.171375 28372 tablet_service.cc:1467] Processing CreateTablet for tablet 76d79d8be9e94e99a7ba51d903ca8a91 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=3aef15bedeb544e9a15460f0b11c368a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:58.171856 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76d79d8be9e94e99a7ba51d903ca8a91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:58.171914 31006 tablet_service.cc:1467] Processing CreateTablet for tablet 76d79d8be9e94e99a7ba51d903ca8a91 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=3aef15bedeb544e9a15460f0b11c368a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:58.172017 27983 tablet_service.cc:1467] Processing CreateTablet for tablet f1b2f3b8111649a7b5725647b8251698 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=3aef15bedeb544e9a15460f0b11c368a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:58.172300 27981 tablet_service.cc:1467] Processing CreateTablet for tablet b40a43ad7ed44c30a070189ebfe5aab6 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=3aef15bedeb544e9a15460f0b11c368a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:58.172610 27983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1b2f3b8111649a7b5725647b8251698. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:58.173043 27981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b40a43ad7ed44c30a070189ebfe5aab6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:58.173247 31007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1b2f3b8111649a7b5725647b8251698. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:58.173986 31008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b40a43ad7ed44c30a070189ebfe5aab6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:58.178066 31006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76d79d8be9e94e99a7ba51d903ca8a91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:58.182610 31140 tablet_service.cc:1467] Processing CreateTablet for tablet f1b2f3b8111649a7b5725647b8251698 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=3aef15bedeb544e9a15460f0b11c368a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:58.182777 31141 tablet_service.cc:1467] Processing CreateTablet for tablet b40a43ad7ed44c30a070189ebfe5aab6 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=3aef15bedeb544e9a15460f0b11c368a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:58.183025 31140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1b2f3b8111649a7b5725647b8251698. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:58.183252 31141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b40a43ad7ed44c30a070189ebfe5aab6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:58.187824 30872 tablet_service.cc:1467] Processing CreateTablet for tablet 76d79d8be9e94e99a7ba51d903ca8a91 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=3aef15bedeb544e9a15460f0b11c368a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:58.188254 30872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76d79d8be9e94e99a7ba51d903ca8a91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:58.192996 32383 tablet_bootstrap.cc:492] T f1b2f3b8111649a7b5725647b8251698 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:58.194823 32391 tablet_bootstrap.cc:492] T 76d79d8be9e94e99a7ba51d903ca8a91 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:58.195312 32383 tablet_bootstrap.cc:654] T f1b2f3b8111649a7b5725647b8251698 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:58.196475 32391 tablet_bootstrap.cc:654] T 76d79d8be9e94e99a7ba51d903ca8a91 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:58.198424 32393 tablet_bootstrap.cc:492] T b40a43ad7ed44c30a070189ebfe5aab6 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:58.200194 32393 tablet_bootstrap.cc:654] T b40a43ad7ed44c30a070189ebfe5aab6 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:58.202636 32382 tablet_bootstrap.cc:492] T 76d79d8be9e94e99a7ba51d903ca8a91 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:58.204773 32382 tablet_bootstrap.cc:654] T 76d79d8be9e94e99a7ba51d903ca8a91 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:58.213071 32383 tablet_bootstrap.cc:492] T f1b2f3b8111649a7b5725647b8251698 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:58.212267 32393 tablet_bootstrap.cc:492] T b40a43ad7ed44c30a070189ebfe5aab6 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:58.213416 32383 ts_tablet_manager.cc:1367] T f1b2f3b8111649a7b5725647b8251698 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.004s
I20240503 08:12:58.213682 32393 ts_tablet_manager.cc:1367] T b40a43ad7ed44c30a070189ebfe5aab6 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.008s
I20240503 08:12:58.214485 32383 raft_consensus.cc:348] T f1b2f3b8111649a7b5725647b8251698 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:58.214524 32393 raft_consensus.cc:348] T b40a43ad7ed44c30a070189ebfe5aab6 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:58.214818 32383 raft_consensus.cc:374] T f1b2f3b8111649a7b5725647b8251698 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:58.214937 32383 raft_consensus.cc:729] T f1b2f3b8111649a7b5725647b8251698 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:58.215242 32383 consensus_queue.cc:260] T f1b2f3b8111649a7b5725647b8251698 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:58.215952 32383 ts_tablet_manager.cc:1398] T f1b2f3b8111649a7b5725647b8251698 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:58.218398 32382 tablet_bootstrap.cc:492] T 76d79d8be9e94e99a7ba51d903ca8a91 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:58.219384 32382 ts_tablet_manager.cc:1367] T 76d79d8be9e94e99a7ba51d903ca8a91 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.001s
I20240503 08:12:58.220202 32382 raft_consensus.cc:348] T 76d79d8be9e94e99a7ba51d903ca8a91 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:58.220513 32382 raft_consensus.cc:374] T 76d79d8be9e94e99a7ba51d903ca8a91 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:58.220624 32382 raft_consensus.cc:729] T 76d79d8be9e94e99a7ba51d903ca8a91 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:58.220938 32382 consensus_queue.cc:260] T 76d79d8be9e94e99a7ba51d903ca8a91 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:58.221623 32382 ts_tablet_manager.cc:1398] T 76d79d8be9e94e99a7ba51d903ca8a91 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:58.222357 32391 tablet_bootstrap.cc:492] T 76d79d8be9e94e99a7ba51d903ca8a91 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:58.222546 32391 ts_tablet_manager.cc:1367] T 76d79d8be9e94e99a7ba51d903ca8a91 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.000s
I20240503 08:12:58.222769 32392 tablet_bootstrap.cc:492] T b40a43ad7ed44c30a070189ebfe5aab6 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:58.223309 32391 raft_consensus.cc:348] T 76d79d8be9e94e99a7ba51d903ca8a91 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:58.223589 32383 tablet_bootstrap.cc:492] T 76d79d8be9e94e99a7ba51d903ca8a91 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:58.223585 32391 raft_consensus.cc:374] T 76d79d8be9e94e99a7ba51d903ca8a91 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:58.223732 32391 raft_consensus.cc:729] T 76d79d8be9e94e99a7ba51d903ca8a91 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:58.224030 32391 consensus_queue.cc:260] T 76d79d8be9e94e99a7ba51d903ca8a91 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:58.225147 32383 tablet_bootstrap.cc:654] T 76d79d8be9e94e99a7ba51d903ca8a91 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:58.225463 32392 tablet_bootstrap.cc:654] T b40a43ad7ed44c30a070189ebfe5aab6 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:58.214721 32393 raft_consensus.cc:374] T b40a43ad7ed44c30a070189ebfe5aab6 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:58.225904 32391 ts_tablet_manager.cc:1398] T 76d79d8be9e94e99a7ba51d903ca8a91 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:58.225945 32393 raft_consensus.cc:729] T b40a43ad7ed44c30a070189ebfe5aab6 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:58.227054 32393 consensus_queue.cc:260] T b40a43ad7ed44c30a070189ebfe5aab6 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:58.231454 32393 ts_tablet_manager.cc:1398] T b40a43ad7ed44c30a070189ebfe5aab6 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.018s user 0.000s sys 0.002s
I20240503 08:12:58.231707 32393 tablet_bootstrap.cc:492] T f1b2f3b8111649a7b5725647b8251698 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:58.234910 32393 tablet_bootstrap.cc:654] T f1b2f3b8111649a7b5725647b8251698 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:58.235464 32383 tablet_bootstrap.cc:492] T 76d79d8be9e94e99a7ba51d903ca8a91 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:58.235646 32383 ts_tablet_manager.cc:1367] T 76d79d8be9e94e99a7ba51d903ca8a91 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.002s
I20240503 08:12:58.236341 32383 raft_consensus.cc:348] T 76d79d8be9e94e99a7ba51d903ca8a91 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:58.236548 32383 raft_consensus.cc:374] T 76d79d8be9e94e99a7ba51d903ca8a91 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:58.236657 32383 raft_consensus.cc:729] T 76d79d8be9e94e99a7ba51d903ca8a91 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:58.237028 32383 consensus_queue.cc:260] T 76d79d8be9e94e99a7ba51d903ca8a91 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:58.237524 32383 ts_tablet_manager.cc:1398] T 76d79d8be9e94e99a7ba51d903ca8a91 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:58.237864 32383 tablet_bootstrap.cc:492] T b40a43ad7ed44c30a070189ebfe5aab6 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:58.239409 32383 tablet_bootstrap.cc:654] T b40a43ad7ed44c30a070189ebfe5aab6 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:58.254187 32392 tablet_bootstrap.cc:492] T b40a43ad7ed44c30a070189ebfe5aab6 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:58.254439 32392 ts_tablet_manager.cc:1367] T b40a43ad7ed44c30a070189ebfe5aab6 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.032s user 0.008s sys 0.000s
I20240503 08:12:58.255218 32392 raft_consensus.cc:348] T b40a43ad7ed44c30a070189ebfe5aab6 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:58.255425 32392 raft_consensus.cc:374] T b40a43ad7ed44c30a070189ebfe5aab6 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:58.255512 32392 raft_consensus.cc:729] T b40a43ad7ed44c30a070189ebfe5aab6 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:58.255731 32392 consensus_queue.cc:260] T b40a43ad7ed44c30a070189ebfe5aab6 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:58.256497 32392 ts_tablet_manager.cc:1398] T b40a43ad7ed44c30a070189ebfe5aab6 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:58.256891 32392 tablet_bootstrap.cc:492] T f1b2f3b8111649a7b5725647b8251698 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:58.256860 32390 raft_consensus.cc:482] T 76d79d8be9e94e99a7ba51d903ca8a91 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:58.257005 32390 raft_consensus.cc:504] T 76d79d8be9e94e99a7ba51d903ca8a91 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:58.257953 32390 leader_election.cc:288] T 76d79d8be9e94e99a7ba51d903ca8a91 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:58.258307 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76d79d8be9e94e99a7ba51d903ca8a91" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:12:58.258538 30895 raft_consensus.cc:2454] T 76d79d8be9e94e99a7ba51d903ca8a91 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:12:58.259025 32393 tablet_bootstrap.cc:492] T f1b2f3b8111649a7b5725647b8251698 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:58.259318 32393 ts_tablet_manager.cc:1367] T f1b2f3b8111649a7b5725647b8251698 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.003s
I20240503 08:12:58.259266 31028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76d79d8be9e94e99a7ba51d903ca8a91" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:58.259471 31028 raft_consensus.cc:2454] T 76d79d8be9e94e99a7ba51d903ca8a91 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:12:58.259912 28326 leader_election.cc:302] T 76d79d8be9e94e99a7ba51d903ca8a91 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:58.260058 32393 raft_consensus.cc:348] T f1b2f3b8111649a7b5725647b8251698 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:58.260298 32393 raft_consensus.cc:374] T f1b2f3b8111649a7b5725647b8251698 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:58.260553 32393 raft_consensus.cc:729] T f1b2f3b8111649a7b5725647b8251698 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:58.261900 32390 raft_consensus.cc:2781] T 76d79d8be9e94e99a7ba51d903ca8a91 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:58.262012 32390 raft_consensus.cc:482] T 76d79d8be9e94e99a7ba51d903ca8a91 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:58.262091 32390 raft_consensus.cc:3037] T 76d79d8be9e94e99a7ba51d903ca8a91 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:58.260852 32393 consensus_queue.cc:260] T f1b2f3b8111649a7b5725647b8251698 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:58.267105 32392 tablet_bootstrap.cc:654] T f1b2f3b8111649a7b5725647b8251698 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:58.268951 32393 ts_tablet_manager.cc:1398] T f1b2f3b8111649a7b5725647b8251698 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240503 08:12:58.269867 32383 tablet_bootstrap.cc:492] T b40a43ad7ed44c30a070189ebfe5aab6 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:58.270023 32383 ts_tablet_manager.cc:1367] T b40a43ad7ed44c30a070189ebfe5aab6 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.032s user 0.012s sys 0.000s
I20240503 08:12:58.270192 32390 raft_consensus.cc:504] T 76d79d8be9e94e99a7ba51d903ca8a91 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:58.271091 32390 leader_election.cc:288] T 76d79d8be9e94e99a7ba51d903ca8a91 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Requested vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:58.272202 31028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76d79d8be9e94e99a7ba51d903ca8a91" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:12:58.272400 31028 raft_consensus.cc:3037] T 76d79d8be9e94e99a7ba51d903ca8a91 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:58.272420 32383 raft_consensus.cc:348] T b40a43ad7ed44c30a070189ebfe5aab6 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:58.272676 32383 raft_consensus.cc:374] T b40a43ad7ed44c30a070189ebfe5aab6 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:58.272753 32383 raft_consensus.cc:729] T b40a43ad7ed44c30a070189ebfe5aab6 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:58.273406 32383 consensus_queue.cc:260] T b40a43ad7ed44c30a070189ebfe5aab6 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:58.274016 32383 ts_tablet_manager.cc:1398] T b40a43ad7ed44c30a070189ebfe5aab6 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:58.274475 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76d79d8be9e94e99a7ba51d903ca8a91" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:12:58.274689 30895 raft_consensus.cc:3037] T 76d79d8be9e94e99a7ba51d903ca8a91 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:58.276156 31028 raft_consensus.cc:2454] T 76d79d8be9e94e99a7ba51d903ca8a91 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:12:58.277563 28327 leader_election.cc:302] T 76d79d8be9e94e99a7ba51d903ca8a91 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:58.277855 32392 tablet_bootstrap.cc:492] T f1b2f3b8111649a7b5725647b8251698 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:58.278021 32217 raft_consensus.cc:2781] T 76d79d8be9e94e99a7ba51d903ca8a91 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:58.278057 32392 ts_tablet_manager.cc:1367] T f1b2f3b8111649a7b5725647b8251698 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.003s
I20240503 08:12:58.278142 32217 raft_consensus.cc:686] T 76d79d8be9e94e99a7ba51d903ca8a91 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Becoming Leader. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Running, Role: LEADER
I20240503 08:12:58.278421 32217 consensus_queue.cc:237] T 76d79d8be9e94e99a7ba51d903ca8a91 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:12:58.279239 32392 raft_consensus.cc:348] T f1b2f3b8111649a7b5725647b8251698 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:58.279454 32392 raft_consensus.cc:374] T f1b2f3b8111649a7b5725647b8251698 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:58.279517 32392 raft_consensus.cc:729] T f1b2f3b8111649a7b5725647b8251698 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:58.279898 32392 consensus_queue.cc:260] T f1b2f3b8111649a7b5725647b8251698 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:58.277137 30895 raft_consensus.cc:2454] T 76d79d8be9e94e99a7ba51d903ca8a91 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:12:58.280488 32392 ts_tablet_manager.cc:1398] T f1b2f3b8111649a7b5725647b8251698 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:58.285483 27854 catalog_manager.cc:5484] T 76d79d8be9e94e99a7ba51d903ca8a91 P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68). New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:12:58.307163 32370 raft_consensus.cc:482] T 424e5ae48de94f4097879574940e11ee P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:58.307353 32370 raft_consensus.cc:504] T 424e5ae48de94f4097879574940e11ee P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:58.308202 32370 leader_election.cc:288] T 424e5ae48de94f4097879574940e11ee P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:12:58.308595 31028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "424e5ae48de94f4097879574940e11ee" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:58.308666 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "424e5ae48de94f4097879574940e11ee" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:12:58.308806 31028 raft_consensus.cc:2454] T 424e5ae48de94f4097879574940e11ee P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:12:58.308902 30895 raft_consensus.cc:2454] T 424e5ae48de94f4097879574940e11ee P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:12:58.309218 30695 leader_election.cc:302] T 424e5ae48de94f4097879574940e11ee P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:58.309578 32370 raft_consensus.cc:2781] T 424e5ae48de94f4097879574940e11ee P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:58.309675 32370 raft_consensus.cc:482] T 424e5ae48de94f4097879574940e11ee P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:58.309757 32370 raft_consensus.cc:3037] T 424e5ae48de94f4097879574940e11ee P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:58.311702 32370 raft_consensus.cc:504] T 424e5ae48de94f4097879574940e11ee P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:58.312582 32370 leader_election.cc:288] T 424e5ae48de94f4097879574940e11ee P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Requested vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:12:58.312673 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "424e5ae48de94f4097879574940e11ee" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:12:58.312732 31028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "424e5ae48de94f4097879574940e11ee" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:12:58.312891 30895 raft_consensus.cc:3037] T 424e5ae48de94f4097879574940e11ee P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:58.312923 31028 raft_consensus.cc:3037] T 424e5ae48de94f4097879574940e11ee P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:58.314733 31028 raft_consensus.cc:2454] T 424e5ae48de94f4097879574940e11ee P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:12:58.314811 30895 raft_consensus.cc:2454] T 424e5ae48de94f4097879574940e11ee P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:12:58.315048 30695 leader_election.cc:302] T 424e5ae48de94f4097879574940e11ee P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:58.315310 32370 raft_consensus.cc:2781] T 424e5ae48de94f4097879574940e11ee P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:58.315426 32370 raft_consensus.cc:686] T 424e5ae48de94f4097879574940e11ee P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Becoming Leader. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Running, Role: LEADER
I20240503 08:12:58.315721 32370 consensus_queue.cc:237] T 424e5ae48de94f4097879574940e11ee P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:58.318940 27862 catalog_manager.cc:5484] T 424e5ae48de94f4097879574940e11ee P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69). New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:12:58.386775 32324 raft_consensus.cc:482] T b40a43ad7ed44c30a070189ebfe5aab6 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:58.386963 32324 raft_consensus.cc:504] T b40a43ad7ed44c30a070189ebfe5aab6 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:58.387560 32324 leader_election.cc:288] T b40a43ad7ed44c30a070189ebfe5aab6 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:12:58.387921 31028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b40a43ad7ed44c30a070189ebfe5aab6" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:58.388067 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b40a43ad7ed44c30a070189ebfe5aab6" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:12:58.388165 31028 raft_consensus.cc:2454] T b40a43ad7ed44c30a070189ebfe5aab6 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 0.
I20240503 08:12:58.388293 28003 raft_consensus.cc:2454] T b40a43ad7ed44c30a070189ebfe5aab6 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 0.
I20240503 08:12:58.388610 31095 leader_election.cc:302] T b40a43ad7ed44c30a070189ebfe5aab6 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:58.389015 32324 raft_consensus.cc:2781] T b40a43ad7ed44c30a070189ebfe5aab6 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:58.389109 32324 raft_consensus.cc:482] T b40a43ad7ed44c30a070189ebfe5aab6 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:58.389169 32324 raft_consensus.cc:3037] T b40a43ad7ed44c30a070189ebfe5aab6 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:58.391259 32324 raft_consensus.cc:504] T b40a43ad7ed44c30a070189ebfe5aab6 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:58.391884 32324 leader_election.cc:288] T b40a43ad7ed44c30a070189ebfe5aab6 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 election: Requested vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:12:58.392400 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b40a43ad7ed44c30a070189ebfe5aab6" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:12:58.392527 31028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b40a43ad7ed44c30a070189ebfe5aab6" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:12:58.392604 28003 raft_consensus.cc:3037] T b40a43ad7ed44c30a070189ebfe5aab6 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:58.392705 31028 raft_consensus.cc:3037] T b40a43ad7ed44c30a070189ebfe5aab6 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:58.394554 28003 raft_consensus.cc:2454] T b40a43ad7ed44c30a070189ebfe5aab6 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 1.
I20240503 08:12:58.394846 31028 raft_consensus.cc:2454] T b40a43ad7ed44c30a070189ebfe5aab6 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 1.
I20240503 08:12:58.394891 31097 leader_election.cc:302] T b40a43ad7ed44c30a070189ebfe5aab6 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, b00f244c69df401997ea99d492218a98; no voters:
I20240503 08:12:58.395282 32324 raft_consensus.cc:2781] T b40a43ad7ed44c30a070189ebfe5aab6 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:58.395503 32324 raft_consensus.cc:686] T b40a43ad7ed44c30a070189ebfe5aab6 P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Becoming Leader. State: Replica: b00f244c69df401997ea99d492218a98, State: Running, Role: LEADER
I20240503 08:12:58.395794 32324 consensus_queue.cc:237] T b40a43ad7ed44c30a070189ebfe5aab6 P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:58.399648 27862 catalog_manager.cc:5484] T b40a43ad7ed44c30a070189ebfe5aab6 P b00f244c69df401997ea99d492218a98 reported cstate change: term changed from 0 to 1, leader changed from <none> to b00f244c69df401997ea99d492218a98 (127.20.230.72). New cstate: current_term: 1 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:12:58.426503 27862 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:58.431476 30895 raft_consensus.cc:1261] T 424e5ae48de94f4097879574940e11ee P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:58.431746 31028 raft_consensus.cc:1261] T 424e5ae48de94f4097879574940e11ee P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:58.432054 32370 consensus_queue.cc:1035] T 424e5ae48de94f4097879574940e11ee P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:58.432266 32369 consensus_queue.cc:1035] T 424e5ae48de94f4097879574940e11ee P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:58.440810 27862 catalog_manager.cc:5576] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747) does not have the latest schema for tablet 424e5ae48de94f4097879574940e11ee (table rebalancer_dynamic_table [id=b8e0b303c65c458d87cc9745c161e5ff]). Expected version 1 got 0
I20240503 08:12:58.441804 27854 catalog_manager.cc:5576] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915) does not have the latest schema for tablet 424e5ae48de94f4097879574940e11ee (table rebalancer_dynamic_table [id=b8e0b303c65c458d87cc9745c161e5ff]). Expected version 1 got 0
I20240503 08:12:58.443297 32409 tablet.cc:1722] T 424e5ae48de94f4097879574940e11ee P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:58.443699 32410 tablet.cc:1722] T 424e5ae48de94f4097879574940e11ee P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:58.443869 32409 tablet.cc:1620] T 424e5ae48de94f4097879574940e11ee P 6d1d7de19459435fbde1999e3dfd226b: MemRowSet was empty: no flush needed.
I20240503 08:12:58.444404 32410 tablet.cc:1620] T 424e5ae48de94f4097879574940e11ee P ced21ca983b749d0b4d4f6a0e2426bf8: MemRowSet was empty: no flush needed.
I20240503 08:12:58.445667 27854 catalog_manager.cc:5576] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915) does not have the latest schema for tablet 424e5ae48de94f4097879574940e11ee (table rebalancer_dynamic_table [id=b8e0b303c65c458d87cc9745c161e5ff]). Expected version 1 got 0
I20240503 08:12:58.450114 32411 tablet.cc:1722] T 424e5ae48de94f4097879574940e11ee P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:58.450848 32411 tablet.cc:1620] T 424e5ae48de94f4097879574940e11ee P ce1496a47d084365ad786cd09e472a2c: MemRowSet was empty: no flush needed.
I20240503 08:12:58.452641 27858 catalog_manager.cc:5576] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747) does not have the latest schema for tablet 424e5ae48de94f4097879574940e11ee (table rebalancer_dynamic_table [id=b8e0b303c65c458d87cc9745c161e5ff]). Expected version 1 got 0
I20240503 08:12:58.455039 27854 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=b8e0b303c65c458d87cc9745c161e5ff] alter complete (version 1)
I20240503 08:12:58.510457 32256 raft_consensus.cc:482] T f1b2f3b8111649a7b5725647b8251698 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:58.510658 32256 raft_consensus.cc:504] T f1b2f3b8111649a7b5725647b8251698 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:58.511464 32256 leader_election.cc:288] T f1b2f3b8111649a7b5725647b8251698 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:58.511873 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1b2f3b8111649a7b5725647b8251698" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:58.512102 31027 raft_consensus.cc:2454] T f1b2f3b8111649a7b5725647b8251698 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:58.512164 31160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1b2f3b8111649a7b5725647b8251698" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:12:58.512382 31160 raft_consensus.cc:2454] T f1b2f3b8111649a7b5725647b8251698 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:12:58.512529 27937 leader_election.cc:302] T f1b2f3b8111649a7b5725647b8251698 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:58.512836 32256 raft_consensus.cc:2781] T f1b2f3b8111649a7b5725647b8251698 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:58.512933 32256 raft_consensus.cc:482] T f1b2f3b8111649a7b5725647b8251698 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:58.513026 32256 raft_consensus.cc:3037] T f1b2f3b8111649a7b5725647b8251698 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:58.518775 32256 raft_consensus.cc:504] T f1b2f3b8111649a7b5725647b8251698 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:58.519578 32256 leader_election.cc:288] T f1b2f3b8111649a7b5725647b8251698 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Requested vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:12:58.519992 31160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1b2f3b8111649a7b5725647b8251698" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:12:58.520223 31160 raft_consensus.cc:3037] T f1b2f3b8111649a7b5725647b8251698 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:58.522099 31160 raft_consensus.cc:2454] T f1b2f3b8111649a7b5725647b8251698 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:58.522461 27936 leader_election.cc:302] T f1b2f3b8111649a7b5725647b8251698 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, b00f244c69df401997ea99d492218a98; no voters:
I20240503 08:12:58.522487 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1b2f3b8111649a7b5725647b8251698" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:12:58.522759 32256 raft_consensus.cc:2781] T f1b2f3b8111649a7b5725647b8251698 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:58.522751 31027 raft_consensus.cc:3037] T f1b2f3b8111649a7b5725647b8251698 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:58.522922 32256 raft_consensus.cc:686] T f1b2f3b8111649a7b5725647b8251698 P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Becoming Leader. State: Replica: 833582ff68c94269be364ebd187a75db, State: Running, Role: LEADER
I20240503 08:12:58.523207 32256 consensus_queue.cc:237] T f1b2f3b8111649a7b5725647b8251698 P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:12:58.527890 27858 catalog_manager.cc:5484] T f1b2f3b8111649a7b5725647b8251698 P 833582ff68c94269be364ebd187a75db reported cstate change: term changed from 0 to 1, leader changed from <none> to 833582ff68c94269be364ebd187a75db (127.20.230.65). New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:12:58.528249 31027 raft_consensus.cc:2454] T f1b2f3b8111649a7b5725647b8251698 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:12:58.558192 27858 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:58.565110 32409 tablet.cc:1722] T 424e5ae48de94f4097879574940e11ee P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:58.565622 32409 tablet.cc:1620] T 424e5ae48de94f4097879574940e11ee P 6d1d7de19459435fbde1999e3dfd226b: MemRowSet was empty: no flush needed.
I20240503 08:12:58.572439 32410 tablet.cc:1722] T 424e5ae48de94f4097879574940e11ee P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:58.572903 32410 tablet.cc:1620] T 424e5ae48de94f4097879574940e11ee P ced21ca983b749d0b4d4f6a0e2426bf8: MemRowSet was empty: no flush needed.
I20240503 08:12:58.576083 27854 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=b8e0b303c65c458d87cc9745c161e5ff] alter complete (version 2)
I20240503 08:12:58.582957 32411 tablet.cc:1722] T 424e5ae48de94f4097879574940e11ee P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:58.583415 32411 tablet.cc:1620] T 424e5ae48de94f4097879574940e11ee P ce1496a47d084365ad786cd09e472a2c: MemRowSet was empty: no flush needed.
I20240503 08:12:58.680752 27854 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:58.687494 32409 tablet.cc:1722] T 424e5ae48de94f4097879574940e11ee P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:58.687783 32411 tablet.cc:1722] T 424e5ae48de94f4097879574940e11ee P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:58.692122 32410 tablet.cc:1722] T 424e5ae48de94f4097879574940e11ee P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:58.695448 27854 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table_new_name [id=b8e0b303c65c458d87cc9745c161e5ff] alter complete (version 3)
I20240503 08:12:58.702559 32217 consensus_queue.cc:1035] T 76d79d8be9e94e99a7ba51d903ca8a91 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:58.707159 32390 consensus_queue.cc:1035] T 76d79d8be9e94e99a7ba51d903ca8a91 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:58.803467 27858 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:58.803661 27858 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:58.806408 27858 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet 424e5ae48de94f4097879574940e11ee
I20240503 08:12:58.807135 30740 tablet_service.cc:1514] Processing DeleteTablet for tablet 424e5ae48de94f4097879574940e11ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:58 UTC) from {username='slave'} at 127.0.0.1:46794
I20240503 08:12:58.807224 31006 tablet_service.cc:1514] Processing DeleteTablet for tablet 424e5ae48de94f4097879574940e11ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:58 UTC) from {username='slave'} at 127.0.0.1:47020
I20240503 08:12:58.807377 30872 tablet_service.cc:1514] Processing DeleteTablet for tablet 424e5ae48de94f4097879574940e11ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:58 UTC) from {username='slave'} at 127.0.0.1:34426
I20240503 08:12:58.807785 32418 tablet_replica.cc:330] T 424e5ae48de94f4097879574940e11ee P 6d1d7de19459435fbde1999e3dfd226b: stopping tablet replica
I20240503 08:12:58.808076 32418 raft_consensus.cc:2229] T 424e5ae48de94f4097879574940e11ee P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:58.808276 32419 tablet_replica.cc:330] T 424e5ae48de94f4097879574940e11ee P ced21ca983b749d0b4d4f6a0e2426bf8: stopping tablet replica
I20240503 08:12:58.808449 32420 tablet_replica.cc:330] T 424e5ae48de94f4097879574940e11ee P ce1496a47d084365ad786cd09e472a2c: stopping tablet replica
I20240503 08:12:58.808508 32418 raft_consensus.cc:2258] T 424e5ae48de94f4097879574940e11ee P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:58.808544 32419 raft_consensus.cc:2229] T 424e5ae48de94f4097879574940e11ee P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:58.808652 32420 raft_consensus.cc:2229] T 424e5ae48de94f4097879574940e11ee P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:58.808792 32419 raft_consensus.cc:2258] T 424e5ae48de94f4097879574940e11ee P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:58.808853 32420 raft_consensus.cc:2258] T 424e5ae48de94f4097879574940e11ee P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:58.809440 32418 ts_tablet_manager.cc:1875] T 424e5ae48de94f4097879574940e11ee P 6d1d7de19459435fbde1999e3dfd226b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:58.809665 32420 ts_tablet_manager.cc:1875] T 424e5ae48de94f4097879574940e11ee P ce1496a47d084365ad786cd09e472a2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:58.809659 32419 ts_tablet_manager.cc:1875] T 424e5ae48de94f4097879574940e11ee P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:58.814913 32420 ts_tablet_manager.cc:1888] T 424e5ae48de94f4097879574940e11ee P ce1496a47d084365ad786cd09e472a2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:58.815016 32420 log.cc:1196] T 424e5ae48de94f4097879574940e11ee P ce1496a47d084365ad786cd09e472a2c: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal/wals/424e5ae48de94f4097879574940e11ee
I20240503 08:12:58.815361 32420 ts_tablet_manager.cc:1909] T 424e5ae48de94f4097879574940e11ee P ce1496a47d084365ad786cd09e472a2c: Deleting consensus metadata
I20240503 08:12:58.815625 32419 ts_tablet_manager.cc:1888] T 424e5ae48de94f4097879574940e11ee P ced21ca983b749d0b4d4f6a0e2426bf8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:58.815737 32419 log.cc:1196] T 424e5ae48de94f4097879574940e11ee P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal/wals/424e5ae48de94f4097879574940e11ee
I20240503 08:12:58.816128 32419 ts_tablet_manager.cc:1909] T 424e5ae48de94f4097879574940e11ee P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting consensus metadata
I20240503 08:12:58.816218 27846 catalog_manager.cc:4830] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): tablet 424e5ae48de94f4097879574940e11ee (table rebalancer_dynamic_table_new_name [id=b8e0b303c65c458d87cc9745c161e5ff]) successfully deleted
I20240503 08:12:58.816490 32418 ts_tablet_manager.cc:1888] T 424e5ae48de94f4097879574940e11ee P 6d1d7de19459435fbde1999e3dfd226b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:58.816605 32418 log.cc:1196] T 424e5ae48de94f4097879574940e11ee P 6d1d7de19459435fbde1999e3dfd226b: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal/wals/424e5ae48de94f4097879574940e11ee
I20240503 08:12:58.817015 32418 ts_tablet_manager.cc:1909] T 424e5ae48de94f4097879574940e11ee P 6d1d7de19459435fbde1999e3dfd226b: Deleting consensus metadata
I20240503 08:12:58.817080 27847 catalog_manager.cc:4830] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): tablet 424e5ae48de94f4097879574940e11ee (table rebalancer_dynamic_table_new_name [id=b8e0b303c65c458d87cc9745c161e5ff]) successfully deleted
I20240503 08:12:58.817790 27848 catalog_manager.cc:4830] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): tablet 424e5ae48de94f4097879574940e11ee (table rebalancer_dynamic_table_new_name [id=b8e0b303c65c458d87cc9745c161e5ff]) successfully deleted
I20240503 08:12:58.819916 32324 consensus_queue.cc:1035] T b40a43ad7ed44c30a070189ebfe5aab6 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:58.824815 32423 consensus_queue.cc:1035] T b40a43ad7ed44c30a070189ebfe5aab6 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:58.909320 27858 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:58.916831 30740 tablet_service.cc:1467] Processing CreateTablet for tablet 4d8583e264614121b401ba644b9900f5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=866e186bcbb647f78a8ecdcc1b593c58]), partition=
I20240503 08:12:58.917007 31006 tablet_service.cc:1467] Processing CreateTablet for tablet 4d8583e264614121b401ba644b9900f5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=866e186bcbb647f78a8ecdcc1b593c58]), partition=
I20240503 08:12:58.917385 30740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d8583e264614121b401ba644b9900f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:58.917392 31006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d8583e264614121b401ba644b9900f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:58.917529 28113 tablet_service.cc:1467] Processing CreateTablet for tablet 4d8583e264614121b401ba644b9900f5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=866e186bcbb647f78a8ecdcc1b593c58]), partition=
I20240503 08:12:58.917965 28113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d8583e264614121b401ba644b9900f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:58.926685 32427 tablet_bootstrap.cc:492] T 4d8583e264614121b401ba644b9900f5 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:58.927834 32428 tablet_bootstrap.cc:492] T 4d8583e264614121b401ba644b9900f5 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:58.928608 32426 tablet_bootstrap.cc:492] T 4d8583e264614121b401ba644b9900f5 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:12:58.930097 32427 tablet_bootstrap.cc:654] T 4d8583e264614121b401ba644b9900f5 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:58.930186 32428 tablet_bootstrap.cc:654] T 4d8583e264614121b401ba644b9900f5 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:58.931255 32426 tablet_bootstrap.cc:654] T 4d8583e264614121b401ba644b9900f5 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:58.935616 32428 tablet_bootstrap.cc:492] T 4d8583e264614121b401ba644b9900f5 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:58.935793 32428 ts_tablet_manager.cc:1367] T 4d8583e264614121b401ba644b9900f5 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20240503 08:12:58.936600 32428 raft_consensus.cc:348] T 4d8583e264614121b401ba644b9900f5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:58.937018 32428 raft_consensus.cc:374] T 4d8583e264614121b401ba644b9900f5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:58.937126 32428 raft_consensus.cc:729] T 4d8583e264614121b401ba644b9900f5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:58.937403 32428 consensus_queue.cc:260] T 4d8583e264614121b401ba644b9900f5 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:58.938100 32428 ts_tablet_manager.cc:1398] T 4d8583e264614121b401ba644b9900f5 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240503 08:12:58.940541 32426 tablet_bootstrap.cc:492] T 4d8583e264614121b401ba644b9900f5 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:12:58.940660 32426 ts_tablet_manager.cc:1367] T 4d8583e264614121b401ba644b9900f5 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20240503 08:12:58.941213 32426 raft_consensus.cc:348] T 4d8583e264614121b401ba644b9900f5 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:58.941370 32426 raft_consensus.cc:374] T 4d8583e264614121b401ba644b9900f5 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:58.941423 32426 raft_consensus.cc:729] T 4d8583e264614121b401ba644b9900f5 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:12:58.941668 32426 consensus_queue.cc:260] T 4d8583e264614121b401ba644b9900f5 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:58.945290 32426 ts_tablet_manager.cc:1398] T 4d8583e264614121b401ba644b9900f5 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:12:58.947101 32427 tablet_bootstrap.cc:492] T 4d8583e264614121b401ba644b9900f5 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:58.947235 32427 ts_tablet_manager.cc:1367] T 4d8583e264614121b401ba644b9900f5 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.000s
I20240503 08:12:58.948024 32427 raft_consensus.cc:348] T 4d8583e264614121b401ba644b9900f5 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:58.948276 32427 raft_consensus.cc:374] T 4d8583e264614121b401ba644b9900f5 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:58.948407 32427 raft_consensus.cc:729] T 4d8583e264614121b401ba644b9900f5 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:58.948626 32427 consensus_queue.cc:260] T 4d8583e264614121b401ba644b9900f5 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:58.949121 32427 ts_tablet_manager.cc:1398] T 4d8583e264614121b401ba644b9900f5 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:59.030447 32337 consensus_queue.cc:1035] T f1b2f3b8111649a7b5725647b8251698 P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:59.035058 32256 consensus_queue.cc:1035] T f1b2f3b8111649a7b5725647b8251698 P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:59.066083 27854 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_extra_table_42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:59.083642 31006 tablet_service.cc:1467] Processing CreateTablet for tablet 98b3c906cfe54f49bb88ce5e00e3cc04 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=3efe105f64254e5eb0be59c6b266462e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:59.083868 31140 tablet_service.cc:1467] Processing CreateTablet for tablet 98b3c906cfe54f49bb88ce5e00e3cc04 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=3efe105f64254e5eb0be59c6b266462e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:59.084295 31006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98b3c906cfe54f49bb88ce5e00e3cc04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:59.084401 31140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98b3c906cfe54f49bb88ce5e00e3cc04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:59.086344 30872 tablet_service.cc:1467] Processing CreateTablet for tablet 98b3c906cfe54f49bb88ce5e00e3cc04 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=3efe105f64254e5eb0be59c6b266462e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:59.086827 30872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98b3c906cfe54f49bb88ce5e00e3cc04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:59.089736 30874 tablet_service.cc:1467] Processing CreateTablet for tablet ab7b01c2e05a430a9873398268b1e41f (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=3efe105f64254e5eb0be59c6b266462e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:59.090236 30874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab7b01c2e05a430a9873398268b1e41f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:59.090842 27981 tablet_service.cc:1467] Processing CreateTablet for tablet 0f25688be4e74d31aaab85b7a4329a34 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=3efe105f64254e5eb0be59c6b266462e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:59.091286 27981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f25688be4e74d31aaab85b7a4329a34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:59.092336 32428 tablet_bootstrap.cc:492] T 98b3c906cfe54f49bb88ce5e00e3cc04 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:59.093978 32428 tablet_bootstrap.cc:654] T 98b3c906cfe54f49bb88ce5e00e3cc04 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:59.094170 28372 tablet_service.cc:1467] Processing CreateTablet for tablet ab7b01c2e05a430a9873398268b1e41f (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=3efe105f64254e5eb0be59c6b266462e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:59.094570 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab7b01c2e05a430a9873398268b1e41f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:59.096340 32428 tablet_bootstrap.cc:492] T 98b3c906cfe54f49bb88ce5e00e3cc04 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:59.096493 32428 ts_tablet_manager.cc:1367] T 98b3c906cfe54f49bb88ce5e00e3cc04 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240503 08:12:59.097111 32428 raft_consensus.cc:348] T 98b3c906cfe54f49bb88ce5e00e3cc04 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:59.097283 32428 raft_consensus.cc:374] T 98b3c906cfe54f49bb88ce5e00e3cc04 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:59.097362 32428 raft_consensus.cc:729] T 98b3c906cfe54f49bb88ce5e00e3cc04 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:59.097592 32428 consensus_queue.cc:260] T 98b3c906cfe54f49bb88ce5e00e3cc04 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:59.089748 30873 tablet_service.cc:1467] Processing CreateTablet for tablet 0f25688be4e74d31aaab85b7a4329a34 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=3efe105f64254e5eb0be59c6b266462e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:59.098166 32428 ts_tablet_manager.cc:1398] T 98b3c906cfe54f49bb88ce5e00e3cc04 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:59.098242 30873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f25688be4e74d31aaab85b7a4329a34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:59.099380 31006 tablet_service.cc:1467] Processing CreateTablet for tablet ab7b01c2e05a430a9873398268b1e41f (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=3efe105f64254e5eb0be59c6b266462e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:59.099620 28243 tablet_service.cc:1467] Processing CreateTablet for tablet 0f25688be4e74d31aaab85b7a4329a34 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=3efe105f64254e5eb0be59c6b266462e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:59.099753 31006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab7b01c2e05a430a9873398268b1e41f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:59.100032 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f25688be4e74d31aaab85b7a4329a34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:59.118614 32438 tablet_bootstrap.cc:492] T 0f25688be4e74d31aaab85b7a4329a34 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:59.119829 32428 tablet_bootstrap.cc:492] T ab7b01c2e05a430a9873398268b1e41f P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:59.120460 32440 tablet_bootstrap.cc:492] T 0f25688be4e74d31aaab85b7a4329a34 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:12:59.120761 32439 tablet_bootstrap.cc:492] T 98b3c906cfe54f49bb88ce5e00e3cc04 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:59.121918 32428 tablet_bootstrap.cc:654] T ab7b01c2e05a430a9873398268b1e41f P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:59.126984 32440 tablet_bootstrap.cc:654] T 0f25688be4e74d31aaab85b7a4329a34 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:59.127368 32438 tablet_bootstrap.cc:654] T 0f25688be4e74d31aaab85b7a4329a34 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:59.131934 32444 tablet_bootstrap.cc:492] T 98b3c906cfe54f49bb88ce5e00e3cc04 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:12:59.132205 32443 tablet_bootstrap.cc:492] T ab7b01c2e05a430a9873398268b1e41f P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:12:59.135360 32444 tablet_bootstrap.cc:654] T 98b3c906cfe54f49bb88ce5e00e3cc04 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:59.141062 32439 tablet_bootstrap.cc:654] T 98b3c906cfe54f49bb88ce5e00e3cc04 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:59.143487 32428 tablet_bootstrap.cc:492] T ab7b01c2e05a430a9873398268b1e41f P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:59.143710 32428 ts_tablet_manager.cc:1367] T ab7b01c2e05a430a9873398268b1e41f P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.004s
I20240503 08:12:59.145287 32428 raft_consensus.cc:348] T ab7b01c2e05a430a9873398268b1e41f P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:59.145541 32428 raft_consensus.cc:374] T ab7b01c2e05a430a9873398268b1e41f P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:59.145709 32428 raft_consensus.cc:729] T ab7b01c2e05a430a9873398268b1e41f P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:59.146023 32428 consensus_queue.cc:260] T ab7b01c2e05a430a9873398268b1e41f P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:59.146715 32428 ts_tablet_manager.cc:1398] T ab7b01c2e05a430a9873398268b1e41f P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:59.151392 32444 tablet_bootstrap.cc:492] T 98b3c906cfe54f49bb88ce5e00e3cc04 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:12:59.151682 32444 ts_tablet_manager.cc:1367] T 98b3c906cfe54f49bb88ce5e00e3cc04 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.009s
I20240503 08:12:59.152755 32438 tablet_bootstrap.cc:492] T 0f25688be4e74d31aaab85b7a4329a34 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:59.153120 32444 raft_consensus.cc:348] T 98b3c906cfe54f49bb88ce5e00e3cc04 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:59.155509 32444 raft_consensus.cc:374] T 98b3c906cfe54f49bb88ce5e00e3cc04 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:59.155718 32444 raft_consensus.cc:729] T 98b3c906cfe54f49bb88ce5e00e3cc04 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:12:59.156242 32444 consensus_queue.cc:260] T 98b3c906cfe54f49bb88ce5e00e3cc04 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:59.157248 32444 ts_tablet_manager.cc:1398] T 98b3c906cfe54f49bb88ce5e00e3cc04 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20240503 08:12:59.160532 32443 tablet_bootstrap.cc:654] T ab7b01c2e05a430a9873398268b1e41f P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:59.163527 32438 ts_tablet_manager.cc:1367] T 0f25688be4e74d31aaab85b7a4329a34 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.045s user 0.008s sys 0.000s
I20240503 08:12:59.166142 32369 raft_consensus.cc:482] T 4d8583e264614121b401ba644b9900f5 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:59.166396 32369 raft_consensus.cc:504] T 4d8583e264614121b401ba644b9900f5 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:59.167157 32369 leader_election.cc:288] T 4d8583e264614121b401ba644b9900f5 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:59.168249 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d8583e264614121b401ba644b9900f5" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:59.168555 31027 raft_consensus.cc:2454] T 4d8583e264614121b401ba644b9900f5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:12:59.169045 30695 leader_election.cc:302] T 4d8583e264614121b401ba644b9900f5 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:59.169536 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d8583e264614121b401ba644b9900f5" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:12:59.169898 28132 raft_consensus.cc:2454] T 4d8583e264614121b401ba644b9900f5 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:12:59.169991 32440 tablet_bootstrap.cc:492] T 0f25688be4e74d31aaab85b7a4329a34 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:12:59.170181 32440 ts_tablet_manager.cc:1367] T 0f25688be4e74d31aaab85b7a4329a34 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.050s user 0.001s sys 0.007s
I20240503 08:12:59.170902 32369 raft_consensus.cc:2781] T 4d8583e264614121b401ba644b9900f5 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:59.171056 32369 raft_consensus.cc:482] T 4d8583e264614121b401ba644b9900f5 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:59.171226 32369 raft_consensus.cc:3037] T 4d8583e264614121b401ba644b9900f5 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:59.171236 32440 raft_consensus.cc:348] T 0f25688be4e74d31aaab85b7a4329a34 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:59.171504 32440 raft_consensus.cc:374] T 0f25688be4e74d31aaab85b7a4329a34 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:59.171662 32440 raft_consensus.cc:729] T 0f25688be4e74d31aaab85b7a4329a34 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:12:59.166070 32438 raft_consensus.cc:348] T 0f25688be4e74d31aaab85b7a4329a34 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:59.173496 32438 raft_consensus.cc:374] T 0f25688be4e74d31aaab85b7a4329a34 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:59.173656 32438 raft_consensus.cc:729] T 0f25688be4e74d31aaab85b7a4329a34 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:59.173959 32438 consensus_queue.cc:260] T 0f25688be4e74d31aaab85b7a4329a34 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:59.175398 32369 raft_consensus.cc:504] T 4d8583e264614121b401ba644b9900f5 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:59.176813 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d8583e264614121b401ba644b9900f5" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:12:59.177132 31027 raft_consensus.cc:3037] T 4d8583e264614121b401ba644b9900f5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:59.177388 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d8583e264614121b401ba644b9900f5" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:12:59.177634 28132 raft_consensus.cc:3037] T 4d8583e264614121b401ba644b9900f5 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:59.179234 32438 ts_tablet_manager.cc:1398] T 0f25688be4e74d31aaab85b7a4329a34 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
I20240503 08:12:59.180080 32440 consensus_queue.cc:260] T 0f25688be4e74d31aaab85b7a4329a34 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:59.180855 28132 raft_consensus.cc:2454] T 4d8583e264614121b401ba644b9900f5 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:12:59.180850 31027 raft_consensus.cc:2454] T 4d8583e264614121b401ba644b9900f5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:12:59.181376 30696 leader_election.cc:302] T 4d8583e264614121b401ba644b9900f5 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 6d1d7de19459435fbde1999e3dfd226b; no voters:
I20240503 08:12:59.181877 32414 raft_consensus.cc:2781] T 4d8583e264614121b401ba644b9900f5 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:59.183838 32414 raft_consensus.cc:686] T 4d8583e264614121b401ba644b9900f5 P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Becoming Leader. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Running, Role: LEADER
I20240503 08:12:59.184355 32414 consensus_queue.cc:237] T 4d8583e264614121b401ba644b9900f5 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:12:59.187711 32440 ts_tablet_manager.cc:1398] T 0f25688be4e74d31aaab85b7a4329a34 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.017s user 0.002s sys 0.001s
I20240503 08:12:59.192649 32443 tablet_bootstrap.cc:492] T ab7b01c2e05a430a9873398268b1e41f P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:12:59.192936 32443 ts_tablet_manager.cc:1367] T ab7b01c2e05a430a9873398268b1e41f P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.061s user 0.000s sys 0.006s
I20240503 08:12:59.193367 32369 leader_election.cc:288] T 4d8583e264614121b401ba644b9900f5 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Requested vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:12:59.195524 32443 raft_consensus.cc:348] T ab7b01c2e05a430a9873398268b1e41f P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:59.195823 32443 raft_consensus.cc:374] T ab7b01c2e05a430a9873398268b1e41f P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:59.196002 32443 raft_consensus.cc:729] T ab7b01c2e05a430a9873398268b1e41f P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:12:59.196382 32443 consensus_queue.cc:260] T ab7b01c2e05a430a9873398268b1e41f P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:59.196398 27854 catalog_manager.cc:5484] T 4d8583e264614121b401ba644b9900f5 P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69). New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:12:59.198505 32439 tablet_bootstrap.cc:492] T 98b3c906cfe54f49bb88ce5e00e3cc04 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:59.199040 32443 ts_tablet_manager.cc:1398] T ab7b01c2e05a430a9873398268b1e41f P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240503 08:12:59.199136 32439 ts_tablet_manager.cc:1367] T 98b3c906cfe54f49bb88ce5e00e3cc04 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.078s user 0.006s sys 0.000s
I20240503 08:12:59.200146 32439 raft_consensus.cc:348] T 98b3c906cfe54f49bb88ce5e00e3cc04 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:59.200400 32439 raft_consensus.cc:374] T 98b3c906cfe54f49bb88ce5e00e3cc04 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:59.200553 32439 raft_consensus.cc:729] T 98b3c906cfe54f49bb88ce5e00e3cc04 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:59.205708 32439 consensus_queue.cc:260] T 98b3c906cfe54f49bb88ce5e00e3cc04 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:59.209524 32439 ts_tablet_manager.cc:1398] T 98b3c906cfe54f49bb88ce5e00e3cc04 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 08:12:59.209998 32439 tablet_bootstrap.cc:492] T ab7b01c2e05a430a9873398268b1e41f P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
W20240503 08:12:59.213063 30828 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.71:41915, user_credentials={real_user=slave}} blocked reactor thread for 71388.6us
I20240503 08:12:59.220208 32439 tablet_bootstrap.cc:654] T ab7b01c2e05a430a9873398268b1e41f P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:59.227078 32439 tablet_bootstrap.cc:492] T ab7b01c2e05a430a9873398268b1e41f P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:59.227336 32439 ts_tablet_manager.cc:1367] T ab7b01c2e05a430a9873398268b1e41f P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240503 08:12:59.228379 32439 raft_consensus.cc:348] T ab7b01c2e05a430a9873398268b1e41f P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:59.228690 32439 raft_consensus.cc:374] T ab7b01c2e05a430a9873398268b1e41f P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:59.228802 32439 raft_consensus.cc:729] T ab7b01c2e05a430a9873398268b1e41f P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:59.229115 32439 consensus_queue.cc:260] T ab7b01c2e05a430a9873398268b1e41f P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:59.229938 32439 ts_tablet_manager.cc:1398] T ab7b01c2e05a430a9873398268b1e41f P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 08:12:59.230494 32439 tablet_bootstrap.cc:492] T 0f25688be4e74d31aaab85b7a4329a34 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:12:59.233667 32439 tablet_bootstrap.cc:654] T 0f25688be4e74d31aaab85b7a4329a34 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:59.236424 32439 tablet_bootstrap.cc:492] T 0f25688be4e74d31aaab85b7a4329a34 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:12:59.236714 32439 ts_tablet_manager.cc:1367] T 0f25688be4e74d31aaab85b7a4329a34 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240503 08:12:59.237958 32439 raft_consensus.cc:348] T 0f25688be4e74d31aaab85b7a4329a34 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:59.238282 32439 raft_consensus.cc:374] T 0f25688be4e74d31aaab85b7a4329a34 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:59.239470 32439 raft_consensus.cc:729] T 0f25688be4e74d31aaab85b7a4329a34 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:12:59.239871 32439 consensus_queue.cc:260] T 0f25688be4e74d31aaab85b7a4329a34 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:59.240762 32439 ts_tablet_manager.cc:1398] T 0f25688be4e74d31aaab85b7a4329a34 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:59.297118 32441 raft_consensus.cc:482] T ab7b01c2e05a430a9873398268b1e41f P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:59.297354 32441 raft_consensus.cc:504] T ab7b01c2e05a430a9873398268b1e41f P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:59.298264 32441 leader_election.cc:288] T ab7b01c2e05a430a9873398268b1e41f P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:12:59.298923 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7b01c2e05a430a9873398268b1e41f" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:59.299189 31027 raft_consensus.cc:2454] T ab7b01c2e05a430a9873398268b1e41f P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:12:59.299257 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7b01c2e05a430a9873398268b1e41f" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:12:59.299520 30895 raft_consensus.cc:2454] T ab7b01c2e05a430a9873398268b1e41f P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:12:59.299646 28327 leader_election.cc:302] T ab7b01c2e05a430a9873398268b1e41f P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:59.299984 32441 raft_consensus.cc:2781] T ab7b01c2e05a430a9873398268b1e41f P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:59.300149 32441 raft_consensus.cc:482] T ab7b01c2e05a430a9873398268b1e41f P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:59.300280 32441 raft_consensus.cc:3037] T ab7b01c2e05a430a9873398268b1e41f P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:59.302790 32441 raft_consensus.cc:504] T ab7b01c2e05a430a9873398268b1e41f P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:59.303454 32441 leader_election.cc:288] T ab7b01c2e05a430a9873398268b1e41f P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Requested vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:12:59.303901 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7b01c2e05a430a9873398268b1e41f" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:12:59.303932 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7b01c2e05a430a9873398268b1e41f" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:12:59.304101 31027 raft_consensus.cc:3037] T ab7b01c2e05a430a9873398268b1e41f P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:59.304124 30895 raft_consensus.cc:3037] T ab7b01c2e05a430a9873398268b1e41f P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:59.306466 30895 raft_consensus.cc:2454] T ab7b01c2e05a430a9873398268b1e41f P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:12:59.306509 31027 raft_consensus.cc:2454] T ab7b01c2e05a430a9873398268b1e41f P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:12:59.306844 28326 leader_election.cc:302] T ab7b01c2e05a430a9873398268b1e41f P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:12:59.307186 32441 raft_consensus.cc:2781] T ab7b01c2e05a430a9873398268b1e41f P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:59.307341 32441 raft_consensus.cc:686] T ab7b01c2e05a430a9873398268b1e41f P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Becoming Leader. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Running, Role: LEADER
I20240503 08:12:59.307654 32441 consensus_queue.cc:237] T ab7b01c2e05a430a9873398268b1e41f P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:12:59.311200 27862 catalog_manager.cc:5484] T ab7b01c2e05a430a9873398268b1e41f P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68). New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:12:59.315052 32437 raft_consensus.cc:482] T 0f25688be4e74d31aaab85b7a4329a34 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:59.315222 32437 raft_consensus.cc:504] T 0f25688be4e74d31aaab85b7a4329a34 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:59.316036 32437 leader_election.cc:288] T 0f25688be4e74d31aaab85b7a4329a34 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:12:59.316460 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f25688be4e74d31aaab85b7a4329a34" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:12:59.316578 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f25688be4e74d31aaab85b7a4329a34" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:12:59.316680 28002 raft_consensus.cc:2454] T 0f25688be4e74d31aaab85b7a4329a34 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:12:59.316787 28262 raft_consensus.cc:2454] T 0f25688be4e74d31aaab85b7a4329a34 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:12:59.317086 30830 leader_election.cc:302] T 0f25688be4e74d31aaab85b7a4329a34 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:59.317509 32437 raft_consensus.cc:2781] T 0f25688be4e74d31aaab85b7a4329a34 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:59.317656 32437 raft_consensus.cc:482] T 0f25688be4e74d31aaab85b7a4329a34 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:59.317806 32437 raft_consensus.cc:3037] T 0f25688be4e74d31aaab85b7a4329a34 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:59.320398 32437 raft_consensus.cc:504] T 0f25688be4e74d31aaab85b7a4329a34 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:59.320900 32437 leader_election.cc:288] T 0f25688be4e74d31aaab85b7a4329a34 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Requested vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:12:59.321404 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f25688be4e74d31aaab85b7a4329a34" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:12:59.321436 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f25688be4e74d31aaab85b7a4329a34" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:12:59.321650 28002 raft_consensus.cc:3037] T 0f25688be4e74d31aaab85b7a4329a34 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:59.321661 28262 raft_consensus.cc:3037] T 0f25688be4e74d31aaab85b7a4329a34 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:59.322324 27854 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:59.325520 28262 raft_consensus.cc:2454] T 0f25688be4e74d31aaab85b7a4329a34 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:12:59.326460 30828 leader_election.cc:302] T 0f25688be4e74d31aaab85b7a4329a34 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:59.327263 32437 raft_consensus.cc:2781] T 0f25688be4e74d31aaab85b7a4329a34 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:59.327567 32437 raft_consensus.cc:686] T 0f25688be4e74d31aaab85b7a4329a34 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Becoming Leader. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Running, Role: LEADER
I20240503 08:12:59.328390 32437 consensus_queue.cc:237] T 0f25688be4e74d31aaab85b7a4329a34 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:12:59.328637 28132 raft_consensus.cc:1261] T 4d8583e264614121b401ba644b9900f5 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:59.328687 31027 raft_consensus.cc:1261] T 4d8583e264614121b401ba644b9900f5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:59.329177 32414 consensus_queue.cc:1035] T 4d8583e264614121b401ba644b9900f5 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:59.329404 32369 consensus_queue.cc:1035] T 4d8583e264614121b401ba644b9900f5 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:59.329602 28002 raft_consensus.cc:2454] T 0f25688be4e74d31aaab85b7a4329a34 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:12:59.335426 27862 catalog_manager.cc:5484] T 0f25688be4e74d31aaab85b7a4329a34 P ce1496a47d084365ad786cd09e472a2c reported cstate change: term changed from 0 to 1, leader changed from <none> to ce1496a47d084365ad786cd09e472a2c (127.20.230.70). New cstate: current_term: 1 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:12:59.336521 27854 catalog_manager.cc:5576] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915) does not have the latest schema for tablet 4d8583e264614121b401ba644b9900f5 (table rebalancer_dynamic_table [id=866e186bcbb647f78a8ecdcc1b593c58]). Expected version 1 got 0
I20240503 08:12:59.343003 32465 tablet.cc:1722] T 4d8583e264614121b401ba644b9900f5 P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:59.344007 32465 tablet.cc:1620] T 4d8583e264614121b401ba644b9900f5 P ced21ca983b749d0b4d4f6a0e2426bf8: MemRowSet was empty: no flush needed.
I20240503 08:12:59.344117 32464 tablet.cc:1722] T 4d8583e264614121b401ba644b9900f5 P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:59.345899 27862 catalog_manager.cc:5576] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915) does not have the latest schema for tablet 4d8583e264614121b401ba644b9900f5 (table rebalancer_dynamic_table [id=866e186bcbb647f78a8ecdcc1b593c58]). Expected version 1 got 0
I20240503 08:12:59.346581 32464 tablet.cc:1620] T 4d8583e264614121b401ba644b9900f5 P 6d1d7de19459435fbde1999e3dfd226b: MemRowSet was empty: no flush needed.
I20240503 08:12:59.355018 27854 catalog_manager.cc:5576] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143) does not have the latest schema for tablet 4d8583e264614121b401ba644b9900f5 (table rebalancer_dynamic_table [id=866e186bcbb647f78a8ecdcc1b593c58]). Expected version 1 got 0
I20240503 08:12:59.357883 32466 tablet.cc:1722] T 4d8583e264614121b401ba644b9900f5 P 1b411c6bb25549368041773d7b710d48: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:59.360272 32466 tablet.cc:1620] T 4d8583e264614121b401ba644b9900f5 P 1b411c6bb25549368041773d7b710d48: MemRowSet was empty: no flush needed.
I20240503 08:12:59.362650 27858 catalog_manager.cc:5576] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143) does not have the latest schema for tablet 4d8583e264614121b401ba644b9900f5 (table rebalancer_dynamic_table [id=866e186bcbb647f78a8ecdcc1b593c58]). Expected version 1 got 0
I20240503 08:12:59.363140 27862 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=866e186bcbb647f78a8ecdcc1b593c58] alter complete (version 1)
I20240503 08:12:59.422683 32423 raft_consensus.cc:482] T 98b3c906cfe54f49bb88ce5e00e3cc04 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:59.422932 32423 raft_consensus.cc:504] T 98b3c906cfe54f49bb88ce5e00e3cc04 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:59.423717 32423 leader_election.cc:288] T 98b3c906cfe54f49bb88ce5e00e3cc04 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:12:59.424304 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98b3c906cfe54f49bb88ce5e00e3cc04" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:12:59.424319 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98b3c906cfe54f49bb88ce5e00e3cc04" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:59.424589 30895 raft_consensus.cc:2454] T 98b3c906cfe54f49bb88ce5e00e3cc04 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 0.
I20240503 08:12:59.424582 31027 raft_consensus.cc:2454] T 98b3c906cfe54f49bb88ce5e00e3cc04 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 0.
I20240503 08:12:59.425055 31095 leader_election.cc:302] T 98b3c906cfe54f49bb88ce5e00e3cc04 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:59.425611 32423 raft_consensus.cc:2781] T 98b3c906cfe54f49bb88ce5e00e3cc04 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:59.425753 32423 raft_consensus.cc:482] T 98b3c906cfe54f49bb88ce5e00e3cc04 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:59.426017 32423 raft_consensus.cc:3037] T 98b3c906cfe54f49bb88ce5e00e3cc04 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:59.428867 32423 raft_consensus.cc:504] T 98b3c906cfe54f49bb88ce5e00e3cc04 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:59.429587 32423 leader_election.cc:288] T 98b3c906cfe54f49bb88ce5e00e3cc04 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 election: Requested vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:12:59.430007 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98b3c906cfe54f49bb88ce5e00e3cc04" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:12:59.430029 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98b3c906cfe54f49bb88ce5e00e3cc04" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:12:59.430198 31027 raft_consensus.cc:3037] T 98b3c906cfe54f49bb88ce5e00e3cc04 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:59.430258 30895 raft_consensus.cc:3037] T 98b3c906cfe54f49bb88ce5e00e3cc04 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:59.432662 30895 raft_consensus.cc:2454] T 98b3c906cfe54f49bb88ce5e00e3cc04 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 1.
I20240503 08:12:59.432785 31027 raft_consensus.cc:2454] T 98b3c906cfe54f49bb88ce5e00e3cc04 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 1.
I20240503 08:12:59.433136 31094 leader_election.cc:302] T 98b3c906cfe54f49bb88ce5e00e3cc04 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:12:59.433426 32423 raft_consensus.cc:2781] T 98b3c906cfe54f49bb88ce5e00e3cc04 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:59.433689 32423 raft_consensus.cc:686] T 98b3c906cfe54f49bb88ce5e00e3cc04 P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Becoming Leader. State: Replica: b00f244c69df401997ea99d492218a98, State: Running, Role: LEADER
I20240503 08:12:59.434051 32423 consensus_queue.cc:237] T 98b3c906cfe54f49bb88ce5e00e3cc04 P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:12:59.437932 27858 catalog_manager.cc:5484] T 98b3c906cfe54f49bb88ce5e00e3cc04 P b00f244c69df401997ea99d492218a98 reported cstate change: term changed from 0 to 1, leader changed from <none> to b00f244c69df401997ea99d492218a98 (127.20.230.72). New cstate: current_term: 1 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:12:59.468158 27858 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:59.475993 32464 tablet.cc:1722] T 4d8583e264614121b401ba644b9900f5 P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:59.476680 32464 tablet.cc:1620] T 4d8583e264614121b401ba644b9900f5 P 6d1d7de19459435fbde1999e3dfd226b: MemRowSet was empty: no flush needed.
I20240503 08:12:59.476907 32465 tablet.cc:1722] T 4d8583e264614121b401ba644b9900f5 P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:59.477433 32465 tablet.cc:1620] T 4d8583e264614121b401ba644b9900f5 P ced21ca983b749d0b4d4f6a0e2426bf8: MemRowSet was empty: no flush needed.
I20240503 08:12:59.486552 27858 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=866e186bcbb647f78a8ecdcc1b593c58] alter complete (version 2)
I20240503 08:12:59.493963 32466 tablet.cc:1722] T 4d8583e264614121b401ba644b9900f5 P 1b411c6bb25549368041773d7b710d48: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:59.494415 32466 tablet.cc:1620] T 4d8583e264614121b401ba644b9900f5 P 1b411c6bb25549368041773d7b710d48: MemRowSet was empty: no flush needed.
I20240503 08:12:59.590802 27858 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:59.597827 32464 tablet.cc:1722] T 4d8583e264614121b401ba644b9900f5 P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:59.598258 32466 tablet.cc:1722] T 4d8583e264614121b401ba644b9900f5 P 1b411c6bb25549368041773d7b710d48: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:59.598292 32465 tablet.cc:1722] T 4d8583e264614121b401ba644b9900f5 P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:59.607980 27854 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table_new_name [id=866e186bcbb647f78a8ecdcc1b593c58] alter complete (version 3)
I20240503 08:12:59.710310 27858 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:59.710554 27858 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:59.713369 27858 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet 4d8583e264614121b401ba644b9900f5
I20240503 08:12:59.714138 31006 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d8583e264614121b401ba644b9900f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:59 UTC) from {username='slave'} at 127.0.0.1:47020
I20240503 08:12:59.714512 30740 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d8583e264614121b401ba644b9900f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:59 UTC) from {username='slave'} at 127.0.0.1:46794
I20240503 08:12:59.714540 28113 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d8583e264614121b401ba644b9900f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:59 UTC) from {username='slave'} at 127.0.0.1:57912
I20240503 08:12:59.729238 32471 tablet_replica.cc:330] T 4d8583e264614121b401ba644b9900f5 P 1b411c6bb25549368041773d7b710d48: stopping tablet replica
I20240503 08:12:59.729746 32471 raft_consensus.cc:2229] T 4d8583e264614121b401ba644b9900f5 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:59.730109 32471 raft_consensus.cc:2258] T 4d8583e264614121b401ba644b9900f5 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:59.730333 32470 tablet_replica.cc:330] T 4d8583e264614121b401ba644b9900f5 P 6d1d7de19459435fbde1999e3dfd226b: stopping tablet replica
I20240503 08:12:59.730582 32470 raft_consensus.cc:2229] T 4d8583e264614121b401ba644b9900f5 P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:59.730729 32469 tablet_replica.cc:330] T 4d8583e264614121b401ba644b9900f5 P ced21ca983b749d0b4d4f6a0e2426bf8: stopping tablet replica
I20240503 08:12:59.731058 32470 raft_consensus.cc:2258] T 4d8583e264614121b401ba644b9900f5 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:59.731056 32469 raft_consensus.cc:2229] T 4d8583e264614121b401ba644b9900f5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:59.731189 32471 ts_tablet_manager.cc:1875] T 4d8583e264614121b401ba644b9900f5 P 1b411c6bb25549368041773d7b710d48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:59.731413 32469 raft_consensus.cc:2258] T 4d8583e264614121b401ba644b9900f5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:59.732008 32470 ts_tablet_manager.cc:1875] T 4d8583e264614121b401ba644b9900f5 P 6d1d7de19459435fbde1999e3dfd226b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:59.732439 32469 ts_tablet_manager.cc:1875] T 4d8583e264614121b401ba644b9900f5 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:59.738484 32470 ts_tablet_manager.cc:1888] T 4d8583e264614121b401ba644b9900f5 P 6d1d7de19459435fbde1999e3dfd226b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:59.738484 32471 ts_tablet_manager.cc:1888] T 4d8583e264614121b401ba644b9900f5 P 1b411c6bb25549368041773d7b710d48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:59.738605 32471 log.cc:1196] T 4d8583e264614121b401ba644b9900f5 P 1b411c6bb25549368041773d7b710d48: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/4d8583e264614121b401ba644b9900f5
I20240503 08:12:59.738613 32470 log.cc:1196] T 4d8583e264614121b401ba644b9900f5 P 6d1d7de19459435fbde1999e3dfd226b: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal/wals/4d8583e264614121b401ba644b9900f5
I20240503 08:12:59.738965 32469 ts_tablet_manager.cc:1888] T 4d8583e264614121b401ba644b9900f5 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:59.739024 32471 ts_tablet_manager.cc:1909] T 4d8583e264614121b401ba644b9900f5 P 1b411c6bb25549368041773d7b710d48: Deleting consensus metadata
I20240503 08:12:59.739028 32470 ts_tablet_manager.cc:1909] T 4d8583e264614121b401ba644b9900f5 P 6d1d7de19459435fbde1999e3dfd226b: Deleting consensus metadata
I20240503 08:12:59.739071 32469 log.cc:1196] T 4d8583e264614121b401ba644b9900f5 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal/wals/4d8583e264614121b401ba644b9900f5
I20240503 08:12:59.739442 32469 ts_tablet_manager.cc:1909] T 4d8583e264614121b401ba644b9900f5 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting consensus metadata
I20240503 08:12:59.739985 27848 catalog_manager.cc:4830] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): tablet 4d8583e264614121b401ba644b9900f5 (table rebalancer_dynamic_table_new_name [id=866e186bcbb647f78a8ecdcc1b593c58]) successfully deleted
I20240503 08:12:59.740228 27848 catalog_manager.cc:4830] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): tablet 4d8583e264614121b401ba644b9900f5 (table rebalancer_dynamic_table_new_name [id=866e186bcbb647f78a8ecdcc1b593c58]) successfully deleted
I20240503 08:12:59.740345 27847 catalog_manager.cc:4830] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): tablet 4d8583e264614121b401ba644b9900f5 (table rebalancer_dynamic_table_new_name [id=866e186bcbb647f78a8ecdcc1b593c58]) successfully deleted
I20240503 08:12:59.811136 32448 consensus_queue.cc:1035] T 0f25688be4e74d31aaab85b7a4329a34 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:59.816800 27858 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:59.818957 32437 consensus_queue.cc:1035] T 0f25688be4e74d31aaab85b7a4329a34 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:59.827081 31006 tablet_service.cc:1467] Processing CreateTablet for tablet 50147f76133440a8a5d4cbfd441a04cf (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a74fa0f143d44f648f6805683b9ce91a]), partition=
I20240503 08:12:59.827096 28243 tablet_service.cc:1467] Processing CreateTablet for tablet 50147f76133440a8a5d4cbfd441a04cf (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a74fa0f143d44f648f6805683b9ce91a]), partition=
I20240503 08:12:59.827438 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50147f76133440a8a5d4cbfd441a04cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:59.827438 31006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50147f76133440a8a5d4cbfd441a04cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:59.834667 28113 tablet_service.cc:1467] Processing CreateTablet for tablet 50147f76133440a8a5d4cbfd441a04cf (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a74fa0f143d44f648f6805683b9ce91a]), partition=
I20240503 08:12:59.835012 28113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50147f76133440a8a5d4cbfd441a04cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:59.838147 32477 tablet_bootstrap.cc:492] T 50147f76133440a8a5d4cbfd441a04cf P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:12:59.839350 32441 consensus_queue.cc:1035] T ab7b01c2e05a430a9873398268b1e41f P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:59.840418 32477 tablet_bootstrap.cc:654] T 50147f76133440a8a5d4cbfd441a04cf P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:59.842535 32477 tablet_bootstrap.cc:492] T 50147f76133440a8a5d4cbfd441a04cf P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:12:59.842737 32477 ts_tablet_manager.cc:1367] T 50147f76133440a8a5d4cbfd441a04cf P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 08:12:59.843905 32477 raft_consensus.cc:348] T 50147f76133440a8a5d4cbfd441a04cf P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:59.844204 32477 raft_consensus.cc:374] T 50147f76133440a8a5d4cbfd441a04cf P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:59.844372 32477 raft_consensus.cc:729] T 50147f76133440a8a5d4cbfd441a04cf P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:12:59.844759 32477 consensus_queue.cc:260] T 50147f76133440a8a5d4cbfd441a04cf P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:59.845404 32477 ts_tablet_manager.cc:1398] T 50147f76133440a8a5d4cbfd441a04cf P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:59.847167 32441 consensus_queue.cc:1035] T ab7b01c2e05a430a9873398268b1e41f P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:59.847402 32482 tablet_bootstrap.cc:492] T 50147f76133440a8a5d4cbfd441a04cf P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:12:59.849246 32482 tablet_bootstrap.cc:654] T 50147f76133440a8a5d4cbfd441a04cf P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:59.862738 32456 raft_consensus.cc:482] T 50147f76133440a8a5d4cbfd441a04cf P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:59.862975 32456 raft_consensus.cc:504] T 50147f76133440a8a5d4cbfd441a04cf P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:59.864078 32456 leader_election.cc:288] T 50147f76133440a8a5d4cbfd441a04cf P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:12:59.864957 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50147f76133440a8a5d4cbfd441a04cf" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
W20240503 08:12:59.865705 30960 leader_election.cc:341] T 50147f76133440a8a5d4cbfd441a04cf P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:59.866556 28131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50147f76133440a8a5d4cbfd441a04cf" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
W20240503 08:12:59.867305 30961 leader_election.cc:341] T 50147f76133440a8a5d4cbfd441a04cf P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:59.867480 30961 leader_election.cc:302] T 50147f76133440a8a5d4cbfd441a04cf P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:12:59.867976 32456 raft_consensus.cc:2726] T 50147f76133440a8a5d4cbfd441a04cf P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:12:59.876937 32482 tablet_bootstrap.cc:492] T 50147f76133440a8a5d4cbfd441a04cf P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:12:59.877152 32482 ts_tablet_manager.cc:1367] T 50147f76133440a8a5d4cbfd441a04cf P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.005s
I20240503 08:12:59.878310 32482 raft_consensus.cc:348] T 50147f76133440a8a5d4cbfd441a04cf P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:59.878713 32482 raft_consensus.cc:374] T 50147f76133440a8a5d4cbfd441a04cf P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:59.878922 32482 raft_consensus.cc:729] T 50147f76133440a8a5d4cbfd441a04cf P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:12:59.879333 32482 consensus_queue.cc:260] T 50147f76133440a8a5d4cbfd441a04cf P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:59.880184 32482 ts_tablet_manager.cc:1398] T 50147f76133440a8a5d4cbfd441a04cf P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:12:59.883627 32481 tablet_bootstrap.cc:492] T 50147f76133440a8a5d4cbfd441a04cf P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:12:59.886801 32481 tablet_bootstrap.cc:654] T 50147f76133440a8a5d4cbfd441a04cf P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:59.899387 32481 tablet_bootstrap.cc:492] T 50147f76133440a8a5d4cbfd441a04cf P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:12:59.899601 32481 ts_tablet_manager.cc:1367] T 50147f76133440a8a5d4cbfd441a04cf P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20240503 08:12:59.900647 32481 raft_consensus.cc:348] T 50147f76133440a8a5d4cbfd441a04cf P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:59.900997 32481 raft_consensus.cc:374] T 50147f76133440a8a5d4cbfd441a04cf P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:59.901158 32481 raft_consensus.cc:729] T 50147f76133440a8a5d4cbfd441a04cf P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:12:59.901392 32481 consensus_queue.cc:260] T 50147f76133440a8a5d4cbfd441a04cf P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:59.908643 32481 ts_tablet_manager.cc:1398] T 50147f76133440a8a5d4cbfd441a04cf P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.009s user 0.003s sys 0.006s
I20240503 08:12:59.940268 32490 raft_consensus.cc:482] T 50147f76133440a8a5d4cbfd441a04cf P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:59.940433 32490 raft_consensus.cc:504] T 50147f76133440a8a5d4cbfd441a04cf P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:59.941179 32490 leader_election.cc:288] T 50147f76133440a8a5d4cbfd441a04cf P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:59.941591 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50147f76133440a8a5d4cbfd441a04cf" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:12:59.941655 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50147f76133440a8a5d4cbfd441a04cf" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:12:59.941836 28262 raft_consensus.cc:2454] T 50147f76133440a8a5d4cbfd441a04cf P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:59.941877 31027 raft_consensus.cc:2454] T 50147f76133440a8a5d4cbfd441a04cf P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:12:59.942184 28067 leader_election.cc:302] T 50147f76133440a8a5d4cbfd441a04cf P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:12:59.942420 32490 raft_consensus.cc:2781] T 50147f76133440a8a5d4cbfd441a04cf P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:59.942535 32490 raft_consensus.cc:482] T 50147f76133440a8a5d4cbfd441a04cf P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:59.942622 32490 raft_consensus.cc:3037] T 50147f76133440a8a5d4cbfd441a04cf P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:59.944548 32490 raft_consensus.cc:504] T 50147f76133440a8a5d4cbfd441a04cf P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:59.945034 32490 leader_election.cc:288] T 50147f76133440a8a5d4cbfd441a04cf P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:12:59.945472 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50147f76133440a8a5d4cbfd441a04cf" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:12:59.945660 31027 raft_consensus.cc:3037] T 50147f76133440a8a5d4cbfd441a04cf P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:59.945976 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50147f76133440a8a5d4cbfd441a04cf" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:12:59.946143 28262 raft_consensus.cc:3037] T 50147f76133440a8a5d4cbfd441a04cf P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:59.948334 31027 raft_consensus.cc:2454] T 50147f76133440a8a5d4cbfd441a04cf P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:12:59.948716 28067 leader_election.cc:302] T 50147f76133440a8a5d4cbfd441a04cf P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:12:59.948922 28262 raft_consensus.cc:2454] T 50147f76133440a8a5d4cbfd441a04cf P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:12:59.949280 32490 raft_consensus.cc:2781] T 50147f76133440a8a5d4cbfd441a04cf P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:59.949460 32490 raft_consensus.cc:686] T 50147f76133440a8a5d4cbfd441a04cf P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Becoming Leader. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Running, Role: LEADER
I20240503 08:12:59.949745 32490 consensus_queue.cc:237] T 50147f76133440a8a5d4cbfd441a04cf P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:12:59.953125 27858 catalog_manager.cc:5484] T 50147f76133440a8a5d4cbfd441a04cf P 1b411c6bb25549368041773d7b710d48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b411c6bb25549368041773d7b710d48 (127.20.230.66). New cstate: current_term: 1 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:12:59.960132 32423 consensus_queue.cc:1035] T 98b3c906cfe54f49bb88ce5e00e3cc04 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:59.965109 32493 consensus_queue.cc:1035] T 98b3c906cfe54f49bb88ce5e00e3cc04 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:59.998683 27858 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_extra_table_44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:13:00.007858 28372 tablet_service.cc:1467] Processing CreateTablet for tablet 58611f4ce8b04ac39064a51580d73f31 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=2af0a6bdbc404add8684cd11068a06f9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:00.007953 30873 tablet_service.cc:1467] Processing CreateTablet for tablet 58611f4ce8b04ac39064a51580d73f31 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=2af0a6bdbc404add8684cd11068a06f9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:00.008395 30873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58611f4ce8b04ac39064a51580d73f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:00.008395 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58611f4ce8b04ac39064a51580d73f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:00.008837 30740 tablet_service.cc:1467] Processing CreateTablet for tablet 58611f4ce8b04ac39064a51580d73f31 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=2af0a6bdbc404add8684cd11068a06f9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:00.009411 30740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58611f4ce8b04ac39064a51580d73f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:00.010236 30741 tablet_service.cc:1467] Processing CreateTablet for tablet 2c77993083a049668875b1ecd5a893ec (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=2af0a6bdbc404add8684cd11068a06f9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:00.010605 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c77993083a049668875b1ecd5a893ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:00.014109 31006 tablet_service.cc:1467] Processing CreateTablet for tablet 2c77993083a049668875b1ecd5a893ec (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=2af0a6bdbc404add8684cd11068a06f9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:00.014613 31006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c77993083a049668875b1ecd5a893ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:00.020851 28373 tablet_service.cc:1467] Processing CreateTablet for tablet 2c77993083a049668875b1ecd5a893ec (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=2af0a6bdbc404add8684cd11068a06f9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:00.021070 31140 tablet_service.cc:1467] Processing CreateTablet for tablet 8d0be44e70ce4c15aaf845fd898e01c3 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=2af0a6bdbc404add8684cd11068a06f9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:00.021291 28373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c77993083a049668875b1ecd5a893ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:00.021533 31140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d0be44e70ce4c15aaf845fd898e01c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:00.024574 32477 tablet_bootstrap.cc:492] T 2c77993083a049668875b1ecd5a893ec P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:13:00.026350 32477 tablet_bootstrap.cc:654] T 2c77993083a049668875b1ecd5a893ec P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:00.026603 32497 tablet_bootstrap.cc:492] T 58611f4ce8b04ac39064a51580d73f31 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:13:00.029000 32497 tablet_bootstrap.cc:654] T 58611f4ce8b04ac39064a51580d73f31 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:00.030848 30742 tablet_service.cc:1467] Processing CreateTablet for tablet 8d0be44e70ce4c15aaf845fd898e01c3 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=2af0a6bdbc404add8684cd11068a06f9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:00.031312 30742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d0be44e70ce4c15aaf845fd898e01c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:00.039484 32501 tablet_bootstrap.cc:492] T 8d0be44e70ce4c15aaf845fd898e01c3 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:13:00.041750 32501 tablet_bootstrap.cc:654] T 8d0be44e70ce4c15aaf845fd898e01c3 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:00.043063 31006 tablet_service.cc:1467] Processing CreateTablet for tablet 8d0be44e70ce4c15aaf845fd898e01c3 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=2af0a6bdbc404add8684cd11068a06f9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:00.043628 31006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d0be44e70ce4c15aaf845fd898e01c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:00.051704 32503 tablet_bootstrap.cc:492] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:13:00.054950 32477 tablet_bootstrap.cc:492] T 2c77993083a049668875b1ecd5a893ec P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:13:00.055142 32477 ts_tablet_manager.cc:1367] T 2c77993083a049668875b1ecd5a893ec P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.031s user 0.019s sys 0.000s
I20240503 08:13:00.055896 32477 raft_consensus.cc:348] T 2c77993083a049668875b1ecd5a893ec P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:00.056275 32477 raft_consensus.cc:374] T 2c77993083a049668875b1ecd5a893ec P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:00.056464 32477 raft_consensus.cc:729] T 2c77993083a049668875b1ecd5a893ec P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:13:00.057443 32500 tablet_bootstrap.cc:492] T 58611f4ce8b04ac39064a51580d73f31 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:13:00.057296 32477 consensus_queue.cc:260] T 2c77993083a049668875b1ecd5a893ec P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:00.060048 32500 tablet_bootstrap.cc:654] T 58611f4ce8b04ac39064a51580d73f31 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:00.064258 32477 ts_tablet_manager.cc:1398] T 2c77993083a049668875b1ecd5a893ec P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 08:13:00.069983 32477 tablet_bootstrap.cc:492] T 8d0be44e70ce4c15aaf845fd898e01c3 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:13:00.071944 32497 tablet_bootstrap.cc:492] T 58611f4ce8b04ac39064a51580d73f31 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:13:00.072203 32497 ts_tablet_manager.cc:1367] T 58611f4ce8b04ac39064a51580d73f31 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.008s
I20240503 08:13:00.072291 32477 tablet_bootstrap.cc:654] T 8d0be44e70ce4c15aaf845fd898e01c3 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:00.072335 32503 tablet_bootstrap.cc:654] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:00.073745 32497 raft_consensus.cc:348] T 58611f4ce8b04ac39064a51580d73f31 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:00.074204 27858 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:13:00.076303 32497 raft_consensus.cc:374] T 58611f4ce8b04ac39064a51580d73f31 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:00.076545 32497 raft_consensus.cc:729] T 58611f4ce8b04ac39064a51580d73f31 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:13:00.077097 32497 consensus_queue.cc:260] T 58611f4ce8b04ac39064a51580d73f31 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:00.077751 32501 tablet_bootstrap.cc:492] T 8d0be44e70ce4c15aaf845fd898e01c3 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:13:00.077931 32497 ts_tablet_manager.cc:1398] T 58611f4ce8b04ac39064a51580d73f31 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 08:13:00.078037 32501 ts_tablet_manager.cc:1367] T 8d0be44e70ce4c15aaf845fd898e01c3 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.005s
I20240503 08:13:00.079164 32501 raft_consensus.cc:348] T 8d0be44e70ce4c15aaf845fd898e01c3 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:00.079444 32497 tablet_bootstrap.cc:492] T 2c77993083a049668875b1ecd5a893ec P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:13:00.080240 32501 raft_consensus.cc:374] T 8d0be44e70ce4c15aaf845fd898e01c3 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:00.080991 32501 raft_consensus.cc:729] T 8d0be44e70ce4c15aaf845fd898e01c3 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:13:00.081602 32497 tablet_bootstrap.cc:654] T 2c77993083a049668875b1ecd5a893ec P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:00.087805 32501 consensus_queue.cc:260] T 8d0be44e70ce4c15aaf845fd898e01c3 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:00.089264 31027 raft_consensus.cc:1261] T 50147f76133440a8a5d4cbfd441a04cf P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:00.089534 32501 ts_tablet_manager.cc:1398] T 8d0be44e70ce4c15aaf845fd898e01c3 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20240503 08:13:00.091545 28262 raft_consensus.cc:1261] T 50147f76133440a8a5d4cbfd441a04cf P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:00.093211 32490 consensus_queue.cc:1035] T 50147f76133440a8a5d4cbfd441a04cf P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:00.093740 32500 tablet_bootstrap.cc:492] T 58611f4ce8b04ac39064a51580d73f31 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:13:00.093748 32434 consensus_queue.cc:1035] T 50147f76133440a8a5d4cbfd441a04cf P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:00.093917 32500 ts_tablet_manager.cc:1367] T 58611f4ce8b04ac39064a51580d73f31 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.008s
I20240503 08:13:00.094269 32497 tablet_bootstrap.cc:492] T 2c77993083a049668875b1ecd5a893ec P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:13:00.094484 32497 ts_tablet_manager.cc:1367] T 2c77993083a049668875b1ecd5a893ec P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.000s
I20240503 08:13:00.094787 32500 raft_consensus.cc:348] T 58611f4ce8b04ac39064a51580d73f31 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:00.095038 32500 raft_consensus.cc:374] T 58611f4ce8b04ac39064a51580d73f31 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:00.095333 32500 raft_consensus.cc:729] T 58611f4ce8b04ac39064a51580d73f31 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:13:00.095517 32497 raft_consensus.cc:348] T 2c77993083a049668875b1ecd5a893ec P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:00.095567 32500 consensus_queue.cc:260] T 58611f4ce8b04ac39064a51580d73f31 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:00.095819 32497 raft_consensus.cc:374] T 2c77993083a049668875b1ecd5a893ec P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:00.096149 32497 raft_consensus.cc:729] T 2c77993083a049668875b1ecd5a893ec P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:13:00.096447 32497 consensus_queue.cc:260] T 2c77993083a049668875b1ecd5a893ec P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:00.098067 32497 ts_tablet_manager.cc:1398] T 2c77993083a049668875b1ecd5a893ec P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:13:00.100222 32500 ts_tablet_manager.cc:1398] T 58611f4ce8b04ac39064a51580d73f31 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240503 08:13:00.106616 27862 catalog_manager.cc:5576] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915) does not have the latest schema for tablet 50147f76133440a8a5d4cbfd441a04cf (table rebalancer_dynamic_table [id=a74fa0f143d44f648f6805683b9ce91a]). Expected version 1 got 0
I20240503 08:13:00.110572 27855 catalog_manager.cc:5576] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919) does not have the latest schema for tablet 50147f76133440a8a5d4cbfd441a04cf (table rebalancer_dynamic_table [id=a74fa0f143d44f648f6805683b9ce91a]). Expected version 1 got 0
I20240503 08:13:00.118664 32477 tablet_bootstrap.cc:492] T 8d0be44e70ce4c15aaf845fd898e01c3 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:13:00.118844 32477 ts_tablet_manager.cc:1367] T 8d0be44e70ce4c15aaf845fd898e01c3 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.049s user 0.004s sys 0.013s
I20240503 08:13:00.119771 32477 raft_consensus.cc:348] T 8d0be44e70ce4c15aaf845fd898e01c3 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:00.119776 32509 tablet.cc:1722] T 50147f76133440a8a5d4cbfd441a04cf P 1b411c6bb25549368041773d7b710d48: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:13:00.119973 32477 raft_consensus.cc:374] T 8d0be44e70ce4c15aaf845fd898e01c3 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:00.120083 32477 raft_consensus.cc:729] T 8d0be44e70ce4c15aaf845fd898e01c3 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:13:00.120312 32477 consensus_queue.cc:260] T 8d0be44e70ce4c15aaf845fd898e01c3 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:00.120545 32509 tablet.cc:1620] T 50147f76133440a8a5d4cbfd441a04cf P 1b411c6bb25549368041773d7b710d48: MemRowSet was empty: no flush needed.
I20240503 08:13:00.121315 32477 ts_tablet_manager.cc:1398] T 8d0be44e70ce4c15aaf845fd898e01c3 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:00.130885 27855 catalog_manager.cc:5576] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915) does not have the latest schema for tablet 50147f76133440a8a5d4cbfd441a04cf (table rebalancer_dynamic_table [id=a74fa0f143d44f648f6805683b9ce91a]). Expected version 1 got 0
I20240503 08:13:00.134035 27858 catalog_manager.cc:5576] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919) does not have the latest schema for tablet 50147f76133440a8a5d4cbfd441a04cf (table rebalancer_dynamic_table [id=a74fa0f143d44f648f6805683b9ce91a]). Expected version 1 got 0
I20240503 08:13:00.134429 32510 tablet.cc:1722] T 50147f76133440a8a5d4cbfd441a04cf P aae860d7ae2b4002bf7794c278f9cc56: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:13:00.135083 32510 tablet.cc:1620] T 50147f76133440a8a5d4cbfd441a04cf P aae860d7ae2b4002bf7794c278f9cc56: MemRowSet was empty: no flush needed.
I20240503 08:13:00.136910 27862 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=a74fa0f143d44f648f6805683b9ce91a] alter complete (version 1)
I20240503 08:13:00.137419 32503 tablet_bootstrap.cc:492] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:13:00.137578 32503 ts_tablet_manager.cc:1367] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.086s user 0.004s sys 0.000s
I20240503 08:13:00.138600 32503 raft_consensus.cc:348] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:00.138819 32503 raft_consensus.cc:374] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:00.138908 32503 raft_consensus.cc:729] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:13:00.139124 32503 consensus_queue.cc:260] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:00.139719 32503 ts_tablet_manager.cc:1398] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:00.136765 32511 tablet.cc:1722] T 50147f76133440a8a5d4cbfd441a04cf P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:13:00.140604 32511 tablet.cc:1620] T 50147f76133440a8a5d4cbfd441a04cf P ced21ca983b749d0b4d4f6a0e2426bf8: MemRowSet was empty: no flush needed.
I20240503 08:13:00.144829 32503 tablet_bootstrap.cc:492] T 8d0be44e70ce4c15aaf845fd898e01c3 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:13:00.146262 32503 tablet_bootstrap.cc:654] T 8d0be44e70ce4c15aaf845fd898e01c3 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:00.155105 32503 tablet_bootstrap.cc:492] T 8d0be44e70ce4c15aaf845fd898e01c3 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:13:00.155261 32503 ts_tablet_manager.cc:1367] T 8d0be44e70ce4c15aaf845fd898e01c3 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 08:13:00.155833 32503 raft_consensus.cc:348] T 8d0be44e70ce4c15aaf845fd898e01c3 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:00.156486 32503 raft_consensus.cc:374] T 8d0be44e70ce4c15aaf845fd898e01c3 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:00.156570 32503 raft_consensus.cc:729] T 8d0be44e70ce4c15aaf845fd898e01c3 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:13:00.156808 32503 consensus_queue.cc:260] T 8d0be44e70ce4c15aaf845fd898e01c3 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:00.157219 32503 ts_tablet_manager.cc:1398] T 8d0be44e70ce4c15aaf845fd898e01c3 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:13:00.157542 32503 tablet_bootstrap.cc:492] T 58611f4ce8b04ac39064a51580d73f31 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:13:00.159066 32503 tablet_bootstrap.cc:654] T 58611f4ce8b04ac39064a51580d73f31 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:00.161513 32503 tablet_bootstrap.cc:492] T 58611f4ce8b04ac39064a51580d73f31 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:13:00.161700 32503 ts_tablet_manager.cc:1367] T 58611f4ce8b04ac39064a51580d73f31 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:13:00.162570 32487 raft_consensus.cc:482] T 58611f4ce8b04ac39064a51580d73f31 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:00.162540 32503 raft_consensus.cc:348] T 58611f4ce8b04ac39064a51580d73f31 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:00.162727 32487 raft_consensus.cc:504] T 58611f4ce8b04ac39064a51580d73f31 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:00.163033 32503 raft_consensus.cc:374] T 58611f4ce8b04ac39064a51580d73f31 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:00.163165 32503 raft_consensus.cc:729] T 58611f4ce8b04ac39064a51580d73f31 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:13:00.163395 32487 leader_election.cc:288] T 58611f4ce8b04ac39064a51580d73f31 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:13:00.163900 30894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58611f4ce8b04ac39064a51580d73f31" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:13:00.164136 30894 raft_consensus.cc:2454] T 58611f4ce8b04ac39064a51580d73f31 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:13:00.164724 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58611f4ce8b04ac39064a51580d73f31" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:13:00.165297 28326 leader_election.cc:302] T 58611f4ce8b04ac39064a51580d73f31 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:13:00.165589 32486 raft_consensus.cc:2781] T 58611f4ce8b04ac39064a51580d73f31 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:00.165690 32486 raft_consensus.cc:482] T 58611f4ce8b04ac39064a51580d73f31 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:00.165800 32486 raft_consensus.cc:3037] T 58611f4ce8b04ac39064a51580d73f31 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:00.165930 32503 consensus_queue.cc:260] T 58611f4ce8b04ac39064a51580d73f31 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:00.166889 30761 raft_consensus.cc:2454] T 58611f4ce8b04ac39064a51580d73f31 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:13:00.167889 32486 raft_consensus.cc:504] T 58611f4ce8b04ac39064a51580d73f31 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:00.168380 32486 leader_election.cc:288] T 58611f4ce8b04ac39064a51580d73f31 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Requested vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:13:00.168990 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58611f4ce8b04ac39064a51580d73f31" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:13:00.169117 32503 ts_tablet_manager.cc:1398] T 58611f4ce8b04ac39064a51580d73f31 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240503 08:13:00.169193 30895 raft_consensus.cc:3037] T 58611f4ce8b04ac39064a51580d73f31 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:00.171867 30895 raft_consensus.cc:2454] T 58611f4ce8b04ac39064a51580d73f31 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:13:00.172940 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58611f4ce8b04ac39064a51580d73f31" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:13:00.173125 30761 raft_consensus.cc:3037] T 58611f4ce8b04ac39064a51580d73f31 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:00.175200 30761 raft_consensus.cc:2454] T 58611f4ce8b04ac39064a51580d73f31 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:13:00.175560 28328 leader_election.cc:302] T 58611f4ce8b04ac39064a51580d73f31 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:13:00.175812 32486 raft_consensus.cc:2781] T 58611f4ce8b04ac39064a51580d73f31 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:00.175944 32486 raft_consensus.cc:686] T 58611f4ce8b04ac39064a51580d73f31 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Becoming Leader. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Running, Role: LEADER
I20240503 08:13:00.176183 32486 consensus_queue.cc:237] T 58611f4ce8b04ac39064a51580d73f31 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:00.179522 27862 catalog_manager.cc:5484] T 58611f4ce8b04ac39064a51580d73f31 P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68). New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:00.245221 27862 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:13:00.252951 32509 tablet.cc:1722] T 50147f76133440a8a5d4cbfd441a04cf P 1b411c6bb25549368041773d7b710d48: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:13:00.253496 32509 tablet.cc:1620] T 50147f76133440a8a5d4cbfd441a04cf P 1b411c6bb25549368041773d7b710d48: MemRowSet was empty: no flush needed.
I20240503 08:13:00.253683 32511 tablet.cc:1722] T 50147f76133440a8a5d4cbfd441a04cf P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:13:00.254137 32511 tablet.cc:1620] T 50147f76133440a8a5d4cbfd441a04cf P ced21ca983b749d0b4d4f6a0e2426bf8: MemRowSet was empty: no flush needed.
I20240503 08:13:00.259027 32510 tablet.cc:1722] T 50147f76133440a8a5d4cbfd441a04cf P aae860d7ae2b4002bf7794c278f9cc56: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:13:00.259495 32510 tablet.cc:1620] T 50147f76133440a8a5d4cbfd441a04cf P aae860d7ae2b4002bf7794c278f9cc56: MemRowSet was empty: no flush needed.
I20240503 08:13:00.264768 27855 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=a74fa0f143d44f648f6805683b9ce91a] alter complete (version 2)
I20240503 08:13:00.369051 27855 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:13:00.375044 32452 raft_consensus.cc:482] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:00.375216 32452 raft_consensus.cc:504] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:00.376029 32452 leader_election.cc:288] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:13:00.376668 32509 tablet.cc:1722] T 50147f76133440a8a5d4cbfd441a04cf P 1b411c6bb25549368041773d7b710d48: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:13:00.377274 28392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c77993083a049668875b1ecd5a893ec" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:13:00.377506 28392 raft_consensus.cc:2454] T 2c77993083a049668875b1ecd5a893ec P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:13:00.377913 30697 leader_election.cc:302] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:13:00.378383 32452 raft_consensus.cc:2781] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:00.378501 32452 raft_consensus.cc:482] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:00.378594 32452 raft_consensus.cc:3037] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:00.380406 32468 raft_consensus.cc:482] T 8d0be44e70ce4c15aaf845fd898e01c3 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:00.380528 32468 raft_consensus.cc:504] T 8d0be44e70ce4c15aaf845fd898e01c3 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:00.381158 32468 leader_election.cc:288] T 8d0be44e70ce4c15aaf845fd898e01c3 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:13:00.381616 31026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d0be44e70ce4c15aaf845fd898e01c3" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:13:00.381846 31026 raft_consensus.cc:2454] T 8d0be44e70ce4c15aaf845fd898e01c3 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:13:00.382283 30695 leader_election.cc:302] T 8d0be44e70ce4c15aaf845fd898e01c3 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:13:00.382544 32468 raft_consensus.cc:2781] T 8d0be44e70ce4c15aaf845fd898e01c3 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:00.382673 32468 raft_consensus.cc:482] T 8d0be44e70ce4c15aaf845fd898e01c3 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:00.382616 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d0be44e70ce4c15aaf845fd898e01c3" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:13:00.382755 32468 raft_consensus.cc:3037] T 8d0be44e70ce4c15aaf845fd898e01c3 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:00.382763 32511 tablet.cc:1722] T 50147f76133440a8a5d4cbfd441a04cf P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:13:00.382899 31159 raft_consensus.cc:2454] T 8d0be44e70ce4c15aaf845fd898e01c3 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:13:00.383168 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c77993083a049668875b1ecd5a893ec" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:13:00.383483 31027 raft_consensus.cc:2454] T 2c77993083a049668875b1ecd5a893ec P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:13:00.384897 32510 tablet.cc:1722] T 50147f76133440a8a5d4cbfd441a04cf P aae860d7ae2b4002bf7794c278f9cc56: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:13:00.385306 32468 raft_consensus.cc:504] T 8d0be44e70ce4c15aaf845fd898e01c3 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:00.386348 32468 leader_election.cc:288] T 8d0be44e70ce4c15aaf845fd898e01c3 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Requested vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:13:00.386833 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d0be44e70ce4c15aaf845fd898e01c3" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:13:00.387619 31027 raft_consensus.cc:3037] T 8d0be44e70ce4c15aaf845fd898e01c3 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:00.387915 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d0be44e70ce4c15aaf845fd898e01c3" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:13:00.388178 31159 raft_consensus.cc:3037] T 8d0be44e70ce4c15aaf845fd898e01c3 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:00.390354 31027 raft_consensus.cc:2454] T 8d0be44e70ce4c15aaf845fd898e01c3 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:13:00.390709 30695 leader_election.cc:302] T 8d0be44e70ce4c15aaf845fd898e01c3 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:13:00.391057 32468 raft_consensus.cc:2781] T 8d0be44e70ce4c15aaf845fd898e01c3 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:00.391330 32452 raft_consensus.cc:504] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:00.392249 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c77993083a049668875b1ecd5a893ec" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:13:00.392417 31027 raft_consensus.cc:3037] T 2c77993083a049668875b1ecd5a893ec P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:00.393226 32452 leader_election.cc:288] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Requested vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:13:00.393270 28392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c77993083a049668875b1ecd5a893ec" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:13:00.394029 28392 raft_consensus.cc:3037] T 2c77993083a049668875b1ecd5a893ec P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:00.394094 31159 raft_consensus.cc:2454] T 8d0be44e70ce4c15aaf845fd898e01c3 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:13:00.394699 32468 raft_consensus.cc:686] T 8d0be44e70ce4c15aaf845fd898e01c3 P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Becoming Leader. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Running, Role: LEADER
I20240503 08:13:00.394806 31027 raft_consensus.cc:2454] T 2c77993083a049668875b1ecd5a893ec P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:13:00.394974 32468 consensus_queue.cc:237] T 8d0be44e70ce4c15aaf845fd898e01c3 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:00.395118 30695 leader_election.cc:302] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:13:00.396389 32452 raft_consensus.cc:2781] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:00.396528 32452 raft_consensus.cc:686] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Becoming Leader. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Running, Role: LEADER
I20240503 08:13:00.397069 32452 consensus_queue.cc:237] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:00.397606 28392 raft_consensus.cc:2454] T 2c77993083a049668875b1ecd5a893ec P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:13:00.398504 27862 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table_new_name [id=a74fa0f143d44f648f6805683b9ce91a] alter complete (version 3)
I20240503 08:13:00.410722 27855 catalog_manager.cc:5484] T 8d0be44e70ce4c15aaf845fd898e01c3 P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69). New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:13:00.415409 27855 catalog_manager.cc:5484] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69). New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:00.499846 27855 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:13:00.500048 27855 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:13:00.503454 27855 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet 50147f76133440a8a5d4cbfd441a04cf
I20240503 08:13:00.504201 28113 tablet_service.cc:1514] Processing DeleteTablet for tablet 50147f76133440a8a5d4cbfd441a04cf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:00 UTC) from {username='slave'} at 127.0.0.1:57912
I20240503 08:13:00.504284 28243 tablet_service.cc:1514] Processing DeleteTablet for tablet 50147f76133440a8a5d4cbfd441a04cf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:00 UTC) from {username='slave'} at 127.0.0.1:41960
I20240503 08:13:00.504894 31006 tablet_service.cc:1514] Processing DeleteTablet for tablet 50147f76133440a8a5d4cbfd441a04cf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:00 UTC) from {username='slave'} at 127.0.0.1:47020
I20240503 08:13:00.505658 32522 tablet_replica.cc:330] T 50147f76133440a8a5d4cbfd441a04cf P ced21ca983b749d0b4d4f6a0e2426bf8: stopping tablet replica
I20240503 08:13:00.505980 32522 raft_consensus.cc:2229] T 50147f76133440a8a5d4cbfd441a04cf P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:00.506944 32522 raft_consensus.cc:2258] T 50147f76133440a8a5d4cbfd441a04cf P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:00.507874 32522 ts_tablet_manager.cc:1875] T 50147f76133440a8a5d4cbfd441a04cf P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:00.512073 32521 tablet_replica.cc:330] T 50147f76133440a8a5d4cbfd441a04cf P aae860d7ae2b4002bf7794c278f9cc56: stopping tablet replica
I20240503 08:13:00.512210 32520 tablet_replica.cc:330] T 50147f76133440a8a5d4cbfd441a04cf P 1b411c6bb25549368041773d7b710d48: stopping tablet replica
I20240503 08:13:00.512316 32521 raft_consensus.cc:2229] T 50147f76133440a8a5d4cbfd441a04cf P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:00.512428 32520 raft_consensus.cc:2229] T 50147f76133440a8a5d4cbfd441a04cf P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:13:00.512534 32521 raft_consensus.cc:2258] T 50147f76133440a8a5d4cbfd441a04cf P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:00.512872 32520 raft_consensus.cc:2258] T 50147f76133440a8a5d4cbfd441a04cf P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:00.513334 32521 ts_tablet_manager.cc:1875] T 50147f76133440a8a5d4cbfd441a04cf P aae860d7ae2b4002bf7794c278f9cc56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:00.513854 32520 ts_tablet_manager.cc:1875] T 50147f76133440a8a5d4cbfd441a04cf P 1b411c6bb25549368041773d7b710d48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:00.514513 32522 ts_tablet_manager.cc:1888] T 50147f76133440a8a5d4cbfd441a04cf P ced21ca983b749d0b4d4f6a0e2426bf8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:13:00.514674 32522 log.cc:1196] T 50147f76133440a8a5d4cbfd441a04cf P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal/wals/50147f76133440a8a5d4cbfd441a04cf
I20240503 08:13:00.515094 32522 ts_tablet_manager.cc:1909] T 50147f76133440a8a5d4cbfd441a04cf P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting consensus metadata
I20240503 08:13:00.516269 27847 catalog_manager.cc:4830] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): tablet 50147f76133440a8a5d4cbfd441a04cf (table rebalancer_dynamic_table_new_name [id=a74fa0f143d44f648f6805683b9ce91a]) successfully deleted
I20240503 08:13:00.518687 32521 ts_tablet_manager.cc:1888] T 50147f76133440a8a5d4cbfd441a04cf P aae860d7ae2b4002bf7794c278f9cc56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:13:00.519074 32521 log.cc:1196] T 50147f76133440a8a5d4cbfd441a04cf P aae860d7ae2b4002bf7794c278f9cc56: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal/wals/50147f76133440a8a5d4cbfd441a04cf
I20240503 08:13:00.519686 32521 ts_tablet_manager.cc:1909] T 50147f76133440a8a5d4cbfd441a04cf P aae860d7ae2b4002bf7794c278f9cc56: Deleting consensus metadata
I20240503 08:13:00.520948 27847 catalog_manager.cc:4830] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): tablet 50147f76133440a8a5d4cbfd441a04cf (table rebalancer_dynamic_table_new_name [id=a74fa0f143d44f648f6805683b9ce91a]) successfully deleted
I20240503 08:13:00.525527 32520 ts_tablet_manager.cc:1888] T 50147f76133440a8a5d4cbfd441a04cf P 1b411c6bb25549368041773d7b710d48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:13:00.525687 32520 log.cc:1196] T 50147f76133440a8a5d4cbfd441a04cf P 1b411c6bb25549368041773d7b710d48: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/50147f76133440a8a5d4cbfd441a04cf
I20240503 08:13:00.526144 32520 ts_tablet_manager.cc:1909] T 50147f76133440a8a5d4cbfd441a04cf P 1b411c6bb25549368041773d7b710d48: Deleting consensus metadata
I20240503 08:13:00.527094 27848 catalog_manager.cc:4830] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): tablet 50147f76133440a8a5d4cbfd441a04cf (table rebalancer_dynamic_table_new_name [id=a74fa0f143d44f648f6805683b9ce91a]) successfully deleted
I20240503 08:13:00.587353 32478 consensus_queue.cc:1035] T 58611f4ce8b04ac39064a51580d73f31 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:00.593154 32486 consensus_queue.cc:1035] T 58611f4ce8b04ac39064a51580d73f31 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:00.606266 27855 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:13:00.614241 27981 tablet_service.cc:1467] Processing CreateTablet for tablet df973fab80774584b29be6802dd5ca77 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=749808d425844a80bc7fa108a2e26403]), partition=
I20240503 08:13:00.614241 28113 tablet_service.cc:1467] Processing CreateTablet for tablet df973fab80774584b29be6802dd5ca77 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=749808d425844a80bc7fa108a2e26403]), partition=
I20240503 08:13:00.614625 27981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df973fab80774584b29be6802dd5ca77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:00.614655 28113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df973fab80774584b29be6802dd5ca77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:00.618141 30741 tablet_service.cc:1467] Processing CreateTablet for tablet df973fab80774584b29be6802dd5ca77 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=749808d425844a80bc7fa108a2e26403]), partition=
I20240503 08:13:00.618486 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df973fab80774584b29be6802dd5ca77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:00.626086 32503 tablet_bootstrap.cc:492] T df973fab80774584b29be6802dd5ca77 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:13:00.627568 32503 tablet_bootstrap.cc:654] T df973fab80774584b29be6802dd5ca77 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:00.634660 32529 tablet_bootstrap.cc:492] T df973fab80774584b29be6802dd5ca77 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:13:00.636687 32529 tablet_bootstrap.cc:654] T df973fab80774584b29be6802dd5ca77 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:00.641614 32528 tablet_bootstrap.cc:492] T df973fab80774584b29be6802dd5ca77 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:13:00.643231 32528 tablet_bootstrap.cc:654] T df973fab80774584b29be6802dd5ca77 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:00.645606 32529 tablet_bootstrap.cc:492] T df973fab80774584b29be6802dd5ca77 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:13:00.645877 32529 ts_tablet_manager.cc:1367] T df973fab80774584b29be6802dd5ca77 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.003s
I20240503 08:13:00.646596 32503 tablet_bootstrap.cc:492] T df973fab80774584b29be6802dd5ca77 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:13:00.646916 32529 raft_consensus.cc:348] T df973fab80774584b29be6802dd5ca77 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:00.647192 32529 raft_consensus.cc:374] T df973fab80774584b29be6802dd5ca77 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:00.647321 32529 raft_consensus.cc:729] T df973fab80774584b29be6802dd5ca77 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:13:00.647578 32503 ts_tablet_manager.cc:1367] T df973fab80774584b29be6802dd5ca77 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20240503 08:13:00.647575 32529 consensus_queue.cc:260] T df973fab80774584b29be6802dd5ca77 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:00.649274 32529 ts_tablet_manager.cc:1398] T df973fab80774584b29be6802dd5ca77 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:13:00.659349 32503 raft_consensus.cc:348] T df973fab80774584b29be6802dd5ca77 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:00.659606 32503 raft_consensus.cc:374] T df973fab80774584b29be6802dd5ca77 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:00.659690 32503 raft_consensus.cc:729] T df973fab80774584b29be6802dd5ca77 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:13:00.659915 32503 consensus_queue.cc:260] T df973fab80774584b29be6802dd5ca77 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:00.660411 32503 ts_tablet_manager.cc:1398] T df973fab80774584b29be6802dd5ca77 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.013s user 0.000s sys 0.000s
I20240503 08:13:00.672638 32468 raft_consensus.cc:482] T df973fab80774584b29be6802dd5ca77 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:00.672837 32468 raft_consensus.cc:504] T df973fab80774584b29be6802dd5ca77 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:00.673636 32468 leader_election.cc:288] T df973fab80774584b29be6802dd5ca77 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:13:00.674206 28131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df973fab80774584b29be6802dd5ca77" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:13:00.674409 28131 raft_consensus.cc:2454] T df973fab80774584b29be6802dd5ca77 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:13:00.674552 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df973fab80774584b29be6802dd5ca77" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:13:00.675073 30696 leader_election.cc:302] T df973fab80774584b29be6802dd5ca77 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 6d1d7de19459435fbde1999e3dfd226b; no voters:
W20240503 08:13:00.675659 30697 leader_election.cc:341] T df973fab80774584b29be6802dd5ca77 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:13:00.676556 32468 raft_consensus.cc:2781] T df973fab80774584b29be6802dd5ca77 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:00.676666 32468 raft_consensus.cc:482] T df973fab80774584b29be6802dd5ca77 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:00.676724 32528 tablet_bootstrap.cc:492] T df973fab80774584b29be6802dd5ca77 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:13:00.676782 32468 raft_consensus.cc:3037] T df973fab80774584b29be6802dd5ca77 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:00.676918 32528 ts_tablet_manager.cc:1367] T df973fab80774584b29be6802dd5ca77 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.035s user 0.016s sys 0.011s
I20240503 08:13:00.677682 32528 raft_consensus.cc:348] T df973fab80774584b29be6802dd5ca77 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:00.677937 32528 raft_consensus.cc:374] T df973fab80774584b29be6802dd5ca77 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:00.678036 32528 raft_consensus.cc:729] T df973fab80774584b29be6802dd5ca77 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:13:00.678287 32528 consensus_queue.cc:260] T df973fab80774584b29be6802dd5ca77 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:00.678872 32528 ts_tablet_manager.cc:1398] T df973fab80774584b29be6802dd5ca77 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:00.679600 32468 raft_consensus.cc:504] T df973fab80774584b29be6802dd5ca77 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:00.680404 32468 leader_election.cc:288] T df973fab80774584b29be6802dd5ca77 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Requested vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:13:00.680649 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df973fab80774584b29be6802dd5ca77" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:13:00.680843 28003 raft_consensus.cc:3037] T df973fab80774584b29be6802dd5ca77 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:00.681959 28131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df973fab80774584b29be6802dd5ca77" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:13:00.682159 28131 raft_consensus.cc:3037] T df973fab80774584b29be6802dd5ca77 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:00.683789 28003 raft_consensus.cc:2454] T df973fab80774584b29be6802dd5ca77 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:13:00.684124 30697 leader_election.cc:302] T df973fab80774584b29be6802dd5ca77 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, 833582ff68c94269be364ebd187a75db; no voters:
I20240503 08:13:00.684413 32468 raft_consensus.cc:2781] T df973fab80774584b29be6802dd5ca77 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:00.684559 32468 raft_consensus.cc:686] T df973fab80774584b29be6802dd5ca77 P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Becoming Leader. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Running, Role: LEADER
I20240503 08:13:00.684778 32468 consensus_queue.cc:237] T df973fab80774584b29be6802dd5ca77 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:00.685213 28131 raft_consensus.cc:2454] T df973fab80774584b29be6802dd5ca77 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:13:00.704641 27855 catalog_manager.cc:5484] T df973fab80774584b29be6802dd5ca77 P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69). New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:00.802397 32468 consensus_queue.cc:1035] T 8d0be44e70ce4c15aaf845fd898e01c3 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:00.806545 32517 consensus_queue.cc:1035] T 8d0be44e70ce4c15aaf845fd898e01c3 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:00.822530 27855 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:13:00.828814 28131 raft_consensus.cc:1261] T df973fab80774584b29be6802dd5ca77 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:00.829316 32517 consensus_queue.cc:1035] T df973fab80774584b29be6802dd5ca77 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:00.830085 28003 raft_consensus.cc:1261] T df973fab80774584b29be6802dd5ca77 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:00.830722 32517 consensus_queue.cc:1035] T df973fab80774584b29be6802dd5ca77 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:00.836395 32525 mvcc.cc:205] Tried to move back new op lower bound from 7023509425467338752 to 7023509424887332864. Current Snapshot: MvccSnapshot[applied={T|T < 7023509425467338752}]
I20240503 08:13:00.836830 32542 tablet.cc:1722] T df973fab80774584b29be6802dd5ca77 P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:13:00.837354 27862 catalog_manager.cc:5576] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143) does not have the latest schema for tablet df973fab80774584b29be6802dd5ca77 (table rebalancer_dynamic_table [id=749808d425844a80bc7fa108a2e26403]). Expected version 1 got 0
I20240503 08:13:00.837481 32542 tablet.cc:1620] T df973fab80774584b29be6802dd5ca77 P 6d1d7de19459435fbde1999e3dfd226b: MemRowSet was empty: no flush needed.
I20240503 08:13:00.838622 32509 tablet.cc:1722] T df973fab80774584b29be6802dd5ca77 P 1b411c6bb25549368041773d7b710d48: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:13:00.839082 32509 tablet.cc:1620] T df973fab80774584b29be6802dd5ca77 P 1b411c6bb25549368041773d7b710d48: MemRowSet was empty: no flush needed.
I20240503 08:13:00.843871 27862 catalog_manager.cc:5576] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143) does not have the latest schema for tablet df973fab80774584b29be6802dd5ca77 (table rebalancer_dynamic_table [id=749808d425844a80bc7fa108a2e26403]). Expected version 1 got 0
I20240503 08:13:00.844887 27854 catalog_manager.cc:5576] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081) does not have the latest schema for tablet df973fab80774584b29be6802dd5ca77 (table rebalancer_dynamic_table [id=749808d425844a80bc7fa108a2e26403]). Expected version 1 got 0
I20240503 08:13:00.849916 27854 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=749808d425844a80bc7fa108a2e26403] alter complete (version 1)
I20240503 08:13:00.853559 27854 catalog_manager.cc:5576] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081) does not have the latest schema for tablet df973fab80774584b29be6802dd5ca77 (table rebalancer_dynamic_table [id=749808d425844a80bc7fa108a2e26403]). Expected version 1 got 0
I20240503 08:13:00.852295 32544 tablet.cc:1722] T df973fab80774584b29be6802dd5ca77 P 833582ff68c94269be364ebd187a75db: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:13:00.855414 32544 tablet.cc:1620] T df973fab80774584b29be6802dd5ca77 P 833582ff68c94269be364ebd187a75db: MemRowSet was empty: no flush needed.
I20240503 08:13:00.884014 32517 consensus_queue.cc:1035] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:00.888435 32452 consensus_queue.cc:1035] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:00.934903 27854 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_extra_table_46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:13:00.943168 28113 tablet_service.cc:1467] Processing CreateTablet for tablet 2e511a3ab0ee4eabb22240de99690603 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b8de6da1757347c7b752a17ffa18b098]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:00.943339 30741 tablet_service.cc:1467] Processing CreateTablet for tablet 2e511a3ab0ee4eabb22240de99690603 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b8de6da1757347c7b752a17ffa18b098]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:00.943727 28113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e511a3ab0ee4eabb22240de99690603. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:00.943782 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e511a3ab0ee4eabb22240de99690603. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:00.944856 31140 tablet_service.cc:1467] Processing CreateTablet for tablet 2c595805729d48fd974d761ed8a60158 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b8de6da1757347c7b752a17ffa18b098]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:00.945277 31140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c595805729d48fd974d761ed8a60158. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:00.946156 31006 tablet_service.cc:1467] Processing CreateTablet for tablet c8ca21ffe16e41ab80f5693dc4dbb682 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b8de6da1757347c7b752a17ffa18b098]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:00.946539 31006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8ca21ffe16e41ab80f5693dc4dbb682. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:00.948105 28111 tablet_service.cc:1467] Processing CreateTablet for tablet 2c595805729d48fd974d761ed8a60158 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b8de6da1757347c7b752a17ffa18b098]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:00.948499 28111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c595805729d48fd974d761ed8a60158. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:00.949517 28243 tablet_service.cc:1467] Processing CreateTablet for tablet c8ca21ffe16e41ab80f5693dc4dbb682 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b8de6da1757347c7b752a17ffa18b098]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:00.949654 28242 tablet_service.cc:1467] Processing CreateTablet for tablet 2c595805729d48fd974d761ed8a60158 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b8de6da1757347c7b752a17ffa18b098]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:00.949947 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8ca21ffe16e41ab80f5693dc4dbb682. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:00.953220 28242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c595805729d48fd974d761ed8a60158. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:00.954169 30873 tablet_service.cc:1467] Processing CreateTablet for tablet 2e511a3ab0ee4eabb22240de99690603 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b8de6da1757347c7b752a17ffa18b098]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:00.954578 30873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e511a3ab0ee4eabb22240de99690603. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:00.956662 27854 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:13:00.957576 32503 tablet_bootstrap.cc:492] T 2e511a3ab0ee4eabb22240de99690603 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:13:00.958180 27981 tablet_service.cc:1467] Processing CreateTablet for tablet c8ca21ffe16e41ab80f5693dc4dbb682 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b8de6da1757347c7b752a17ffa18b098]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:00.958606 27981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8ca21ffe16e41ab80f5693dc4dbb682. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:00.959344 32503 tablet_bootstrap.cc:654] T 2e511a3ab0ee4eabb22240de99690603 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:00.962033 32529 tablet_bootstrap.cc:492] T 2e511a3ab0ee4eabb22240de99690603 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:13:00.963721 32529 tablet_bootstrap.cc:654] T 2e511a3ab0ee4eabb22240de99690603 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:00.971103 32547 tablet_bootstrap.cc:492] T c8ca21ffe16e41ab80f5693dc4dbb682 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:13:00.974290 32547 tablet_bootstrap.cc:654] T c8ca21ffe16e41ab80f5693dc4dbb682 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:00.975894 32542 tablet.cc:1722] T df973fab80774584b29be6802dd5ca77 P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:13:00.976378 32542 tablet.cc:1620] T df973fab80774584b29be6802dd5ca77 P 6d1d7de19459435fbde1999e3dfd226b: MemRowSet was empty: no flush needed.
I20240503 08:13:00.977522 32509 tablet.cc:1722] T df973fab80774584b29be6802dd5ca77 P 1b411c6bb25549368041773d7b710d48: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:13:00.977988 32509 tablet.cc:1620] T df973fab80774584b29be6802dd5ca77 P 1b411c6bb25549368041773d7b710d48: MemRowSet was empty: no flush needed.
I20240503 08:13:00.982187 32528 tablet_bootstrap.cc:492] T c8ca21ffe16e41ab80f5693dc4dbb682 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:13:00.984253 32553 tablet_bootstrap.cc:492] T 2c595805729d48fd974d761ed8a60158 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:13:00.984634 32528 tablet_bootstrap.cc:654] T c8ca21ffe16e41ab80f5693dc4dbb682 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:00.991786 27854 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=749808d425844a80bc7fa108a2e26403] alter complete (version 2)
I20240503 08:13:00.996424 32553 tablet_bootstrap.cc:654] T 2c595805729d48fd974d761ed8a60158 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:01.002801 32553 tablet_bootstrap.cc:492] T 2c595805729d48fd974d761ed8a60158 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:13:01.003790 32544 tablet.cc:1722] T df973fab80774584b29be6802dd5ca77 P 833582ff68c94269be364ebd187a75db: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:13:01.003986 32550 tablet_bootstrap.cc:492] T c8ca21ffe16e41ab80f5693dc4dbb682 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:13:01.004259 32544 tablet.cc:1620] T df973fab80774584b29be6802dd5ca77 P 833582ff68c94269be364ebd187a75db: MemRowSet was empty: no flush needed.
I20240503 08:13:01.005839 32553 ts_tablet_manager.cc:1367] T 2c595805729d48fd974d761ed8a60158 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.001s
I20240503 08:13:01.006538 32528 tablet_bootstrap.cc:492] T c8ca21ffe16e41ab80f5693dc4dbb682 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:13:01.006718 32528 ts_tablet_manager.cc:1367] T c8ca21ffe16e41ab80f5693dc4dbb682 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.002s
I20240503 08:13:01.007459 32528 raft_consensus.cc:348] T c8ca21ffe16e41ab80f5693dc4dbb682 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:01.007711 32528 raft_consensus.cc:374] T c8ca21ffe16e41ab80f5693dc4dbb682 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:01.007797 32528 raft_consensus.cc:729] T c8ca21ffe16e41ab80f5693dc4dbb682 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:13:01.008015 32528 consensus_queue.cc:260] T c8ca21ffe16e41ab80f5693dc4dbb682 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:01.008263 32553 raft_consensus.cc:348] T 2c595805729d48fd974d761ed8a60158 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:01.008482 32553 raft_consensus.cc:374] T 2c595805729d48fd974d761ed8a60158 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:01.009582 32552 tablet_bootstrap.cc:492] T 2e511a3ab0ee4eabb22240de99690603 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:13:01.009735 32528 ts_tablet_manager.cc:1398] T c8ca21ffe16e41ab80f5693dc4dbb682 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240503 08:13:01.010108 32553 raft_consensus.cc:729] T 2c595805729d48fd974d761ed8a60158 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:13:01.010715 32550 tablet_bootstrap.cc:654] T c8ca21ffe16e41ab80f5693dc4dbb682 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:01.011181 32553 consensus_queue.cc:260] T 2c595805729d48fd974d761ed8a60158 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:01.014026 32503 tablet_bootstrap.cc:492] T 2e511a3ab0ee4eabb22240de99690603 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:13:01.014230 32503 ts_tablet_manager.cc:1367] T 2e511a3ab0ee4eabb22240de99690603 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.057s user 0.003s sys 0.006s
I20240503 08:13:01.015132 32503 raft_consensus.cc:348] T 2e511a3ab0ee4eabb22240de99690603 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:01.015398 32552 tablet_bootstrap.cc:654] T 2e511a3ab0ee4eabb22240de99690603 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:01.015539 32553 ts_tablet_manager.cc:1398] T 2c595805729d48fd974d761ed8a60158 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240503 08:13:01.016235 32529 tablet_bootstrap.cc:492] T 2e511a3ab0ee4eabb22240de99690603 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:13:01.016414 32529 ts_tablet_manager.cc:1367] T 2e511a3ab0ee4eabb22240de99690603 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.054s user 0.000s sys 0.012s
I20240503 08:13:01.017100 32547 tablet_bootstrap.cc:492] T c8ca21ffe16e41ab80f5693dc4dbb682 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:13:01.017251 32547 ts_tablet_manager.cc:1367] T c8ca21ffe16e41ab80f5693dc4dbb682 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.046s user 0.018s sys 0.009s
I20240503 08:13:01.017160 32529 raft_consensus.cc:348] T 2e511a3ab0ee4eabb22240de99690603 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:01.017364 32529 raft_consensus.cc:374] T 2e511a3ab0ee4eabb22240de99690603 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:01.017432 32529 raft_consensus.cc:729] T 2e511a3ab0ee4eabb22240de99690603 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:13:01.017601 32503 raft_consensus.cc:374] T 2e511a3ab0ee4eabb22240de99690603 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:01.017696 32503 raft_consensus.cc:729] T 2e511a3ab0ee4eabb22240de99690603 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:13:01.017941 32503 consensus_queue.cc:260] T 2e511a3ab0ee4eabb22240de99690603 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:01.018262 32547 raft_consensus.cc:348] T c8ca21ffe16e41ab80f5693dc4dbb682 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:01.018447 32547 raft_consensus.cc:374] T c8ca21ffe16e41ab80f5693dc4dbb682 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:01.018535 32547 raft_consensus.cc:729] T c8ca21ffe16e41ab80f5693dc4dbb682 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:13:01.018738 32547 consensus_queue.cc:260] T c8ca21ffe16e41ab80f5693dc4dbb682 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:01.019498 32547 ts_tablet_manager.cc:1398] T c8ca21ffe16e41ab80f5693dc4dbb682 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:01.029597 32529 consensus_queue.cc:260] T 2e511a3ab0ee4eabb22240de99690603 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:01.029758 32503 ts_tablet_manager.cc:1398] T 2e511a3ab0ee4eabb22240de99690603 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240503 08:13:01.031014 32529 ts_tablet_manager.cc:1398] T 2e511a3ab0ee4eabb22240de99690603 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240503 08:13:01.031311 32529 tablet_bootstrap.cc:492] T 2c595805729d48fd974d761ed8a60158 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:13:01.035301 32550 tablet_bootstrap.cc:492] T c8ca21ffe16e41ab80f5693dc4dbb682 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:13:01.035473 32550 ts_tablet_manager.cc:1367] T c8ca21ffe16e41ab80f5693dc4dbb682 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.000s
I20240503 08:13:01.036281 32550 raft_consensus.cc:348] T c8ca21ffe16e41ab80f5693dc4dbb682 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:01.036500 32550 raft_consensus.cc:374] T c8ca21ffe16e41ab80f5693dc4dbb682 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:01.036623 32550 raft_consensus.cc:729] T c8ca21ffe16e41ab80f5693dc4dbb682 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:13:01.036846 32550 consensus_queue.cc:260] T c8ca21ffe16e41ab80f5693dc4dbb682 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:01.055579 32529 tablet_bootstrap.cc:654] T 2c595805729d48fd974d761ed8a60158 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:01.057137 32552 tablet_bootstrap.cc:492] T 2e511a3ab0ee4eabb22240de99690603 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:13:01.057363 32552 ts_tablet_manager.cc:1367] T 2e511a3ab0ee4eabb22240de99690603 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.048s user 0.005s sys 0.016s
I20240503 08:13:01.058168 32552 raft_consensus.cc:348] T 2e511a3ab0ee4eabb22240de99690603 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:01.058470 32552 raft_consensus.cc:374] T 2e511a3ab0ee4eabb22240de99690603 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:01.058607 32552 raft_consensus.cc:729] T 2e511a3ab0ee4eabb22240de99690603 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:13:01.058864 32552 consensus_queue.cc:260] T 2e511a3ab0ee4eabb22240de99690603 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:01.059432 32550 ts_tablet_manager.cc:1398] T c8ca21ffe16e41ab80f5693dc4dbb682 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.024s user 0.003s sys 0.010s
I20240503 08:13:01.059473 32552 ts_tablet_manager.cc:1398] T 2e511a3ab0ee4eabb22240de99690603 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:01.060489 32529 tablet_bootstrap.cc:492] T 2c595805729d48fd974d761ed8a60158 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:13:01.060644 32529 ts_tablet_manager.cc:1367] T 2c595805729d48fd974d761ed8a60158 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.029s user 0.000s sys 0.003s
I20240503 08:13:01.061331 32529 raft_consensus.cc:348] T 2c595805729d48fd974d761ed8a60158 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:01.061553 32529 raft_consensus.cc:374] T 2c595805729d48fd974d761ed8a60158 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:01.061630 32529 raft_consensus.cc:729] T 2c595805729d48fd974d761ed8a60158 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:13:01.061854 32529 consensus_queue.cc:260] T 2c595805729d48fd974d761ed8a60158 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:01.063305 32550 tablet_bootstrap.cc:492] T 2c595805729d48fd974d761ed8a60158 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:13:01.063335 32529 ts_tablet_manager.cc:1398] T 2c595805729d48fd974d761ed8a60158 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:13:01.066715 32550 tablet_bootstrap.cc:654] T 2c595805729d48fd974d761ed8a60158 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:01.068996 32550 tablet_bootstrap.cc:492] T 2c595805729d48fd974d761ed8a60158 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:13:01.069212 32550 ts_tablet_manager.cc:1367] T 2c595805729d48fd974d761ed8a60158 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240503 08:13:01.070091 32550 raft_consensus.cc:348] T 2c595805729d48fd974d761ed8a60158 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:01.071017 32550 raft_consensus.cc:374] T 2c595805729d48fd974d761ed8a60158 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:01.071171 32550 raft_consensus.cc:729] T 2c595805729d48fd974d761ed8a60158 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:13:01.071501 32550 consensus_queue.cc:260] T 2c595805729d48fd974d761ed8a60158 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:01.075412 32550 ts_tablet_manager.cc:1398] T 2c595805729d48fd974d761ed8a60158 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:13:01.097182 27855 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:13:01.105696 32542 tablet.cc:1722] T df973fab80774584b29be6802dd5ca77 P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:13:01.106297 32509 tablet.cc:1722] T df973fab80774584b29be6802dd5ca77 P 1b411c6bb25549368041773d7b710d48: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:13:01.111423 32544 tablet.cc:1722] T df973fab80774584b29be6802dd5ca77 P 833582ff68c94269be364ebd187a75db: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:13:01.111699 32536 raft_consensus.cc:482] T c8ca21ffe16e41ab80f5693dc4dbb682 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:01.111863 32536 raft_consensus.cc:504] T c8ca21ffe16e41ab80f5693dc4dbb682 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:01.112603 32536 leader_election.cc:288] T c8ca21ffe16e41ab80f5693dc4dbb682 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:13:01.113032 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8ca21ffe16e41ab80f5693dc4dbb682" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:13:01.113148 28261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8ca21ffe16e41ab80f5693dc4dbb682" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:13:01.113240 31027 raft_consensus.cc:2454] T c8ca21ffe16e41ab80f5693dc4dbb682 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:13:01.113404 28261 raft_consensus.cc:2454] T c8ca21ffe16e41ab80f5693dc4dbb682 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:13:01.113708 27937 leader_election.cc:302] T c8ca21ffe16e41ab80f5693dc4dbb682 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:13:01.117743 32536 raft_consensus.cc:2781] T c8ca21ffe16e41ab80f5693dc4dbb682 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:01.117872 32536 raft_consensus.cc:482] T c8ca21ffe16e41ab80f5693dc4dbb682 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:01.117892 27855 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table_new_name [id=749808d425844a80bc7fa108a2e26403] alter complete (version 3)
I20240503 08:13:01.117957 32536 raft_consensus.cc:3037] T c8ca21ffe16e41ab80f5693dc4dbb682 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:01.119745 32536 raft_consensus.cc:504] T c8ca21ffe16e41ab80f5693dc4dbb682 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:01.120286 32536 leader_election.cc:288] T c8ca21ffe16e41ab80f5693dc4dbb682 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Requested vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:13:01.120695 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8ca21ffe16e41ab80f5693dc4dbb682" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:13:01.120873 31027 raft_consensus.cc:3037] T c8ca21ffe16e41ab80f5693dc4dbb682 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:01.122956 31027 raft_consensus.cc:2454] T c8ca21ffe16e41ab80f5693dc4dbb682 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:13:01.123327 27937 leader_election.cc:302] T c8ca21ffe16e41ab80f5693dc4dbb682 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:13:01.123600 32536 raft_consensus.cc:2781] T c8ca21ffe16e41ab80f5693dc4dbb682 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:01.123797 32536 raft_consensus.cc:686] T c8ca21ffe16e41ab80f5693dc4dbb682 P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Becoming Leader. State: Replica: 833582ff68c94269be364ebd187a75db, State: Running, Role: LEADER
I20240503 08:13:01.124040 32536 consensus_queue.cc:237] T c8ca21ffe16e41ab80f5693dc4dbb682 P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:01.124987 28261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8ca21ffe16e41ab80f5693dc4dbb682" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:13:01.125214 28261 raft_consensus.cc:3037] T c8ca21ffe16e41ab80f5693dc4dbb682 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:01.127355 28261 raft_consensus.cc:2454] T c8ca21ffe16e41ab80f5693dc4dbb682 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:13:01.129484 27854 catalog_manager.cc:5484] T c8ca21ffe16e41ab80f5693dc4dbb682 P 833582ff68c94269be364ebd187a75db reported cstate change: term changed from 0 to 1, leader changed from <none> to 833582ff68c94269be364ebd187a75db (127.20.230.65). New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:01.131410 32517 raft_consensus.cc:482] T 2e511a3ab0ee4eabb22240de99690603 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:01.131536 32517 raft_consensus.cc:504] T 2e511a3ab0ee4eabb22240de99690603 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:01.132186 32517 leader_election.cc:288] T 2e511a3ab0ee4eabb22240de99690603 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:13:01.132714 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e511a3ab0ee4eabb22240de99690603" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:13:01.132829 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e511a3ab0ee4eabb22240de99690603" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:13:01.132948 30895 raft_consensus.cc:2454] T 2e511a3ab0ee4eabb22240de99690603 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:13:01.133044 28132 raft_consensus.cc:2454] T 2e511a3ab0ee4eabb22240de99690603 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:13:01.133378 30696 leader_election.cc:302] T 2e511a3ab0ee4eabb22240de99690603 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 6d1d7de19459435fbde1999e3dfd226b; no voters:
I20240503 08:13:01.133677 32517 raft_consensus.cc:2781] T 2e511a3ab0ee4eabb22240de99690603 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:01.133817 32517 raft_consensus.cc:482] T 2e511a3ab0ee4eabb22240de99690603 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:01.133920 32517 raft_consensus.cc:3037] T 2e511a3ab0ee4eabb22240de99690603 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:01.136466 32517 raft_consensus.cc:504] T 2e511a3ab0ee4eabb22240de99690603 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:01.136974 32517 leader_election.cc:288] T 2e511a3ab0ee4eabb22240de99690603 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Requested vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:13:01.137264 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e511a3ab0ee4eabb22240de99690603" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:13:01.137362 30894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e511a3ab0ee4eabb22240de99690603" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:13:01.137435 28132 raft_consensus.cc:3037] T 2e511a3ab0ee4eabb22240de99690603 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:01.137542 30894 raft_consensus.cc:3037] T 2e511a3ab0ee4eabb22240de99690603 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:01.139866 30894 raft_consensus.cc:2454] T 2e511a3ab0ee4eabb22240de99690603 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:13:01.140008 28132 raft_consensus.cc:2454] T 2e511a3ab0ee4eabb22240de99690603 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:13:01.140475 30694 leader_election.cc:302] T 2e511a3ab0ee4eabb22240de99690603 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:13:01.140919 32517 raft_consensus.cc:2781] T 2e511a3ab0ee4eabb22240de99690603 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:01.141062 32517 raft_consensus.cc:686] T 2e511a3ab0ee4eabb22240de99690603 P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Becoming Leader. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Running, Role: LEADER
I20240503 08:13:01.141316 32517 consensus_queue.cc:237] T 2e511a3ab0ee4eabb22240de99690603 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:01.145162 27854 catalog_manager.cc:5484] T 2e511a3ab0ee4eabb22240de99690603 P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69). New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:01.166985 32534 raft_consensus.cc:482] T 2c595805729d48fd974d761ed8a60158 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:01.167161 32534 raft_consensus.cc:504] T 2c595805729d48fd974d761ed8a60158 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:01.168089 32534 leader_election.cc:288] T 2c595805729d48fd974d761ed8a60158 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:13:01.168485 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c595805729d48fd974d761ed8a60158" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:13:01.168458 28261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c595805729d48fd974d761ed8a60158" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:13:01.168707 28261 raft_consensus.cc:2454] T 2c595805729d48fd974d761ed8a60158 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 0.
I20240503 08:13:01.168805 28132 raft_consensus.cc:2454] T 2c595805729d48fd974d761ed8a60158 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 0.
I20240503 08:13:01.169139 31095 leader_election.cc:302] T 2c595805729d48fd974d761ed8a60158 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, b00f244c69df401997ea99d492218a98; no voters:
I20240503 08:13:01.169663 32534 raft_consensus.cc:2781] T 2c595805729d48fd974d761ed8a60158 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:01.170229 32534 raft_consensus.cc:482] T 2c595805729d48fd974d761ed8a60158 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:01.170333 32534 raft_consensus.cc:3037] T 2c595805729d48fd974d761ed8a60158 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:01.172364 32534 raft_consensus.cc:504] T 2c595805729d48fd974d761ed8a60158 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:01.172895 32534 leader_election.cc:288] T 2c595805729d48fd974d761ed8a60158 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:13:01.173271 28261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c595805729d48fd974d761ed8a60158" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:13:01.173334 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c595805729d48fd974d761ed8a60158" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:13:01.173455 28261 raft_consensus.cc:3037] T 2c595805729d48fd974d761ed8a60158 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:01.173550 28132 raft_consensus.cc:3037] T 2c595805729d48fd974d761ed8a60158 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:01.175612 28261 raft_consensus.cc:2454] T 2c595805729d48fd974d761ed8a60158 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 1.
I20240503 08:13:01.175948 31095 leader_election.cc:302] T 2c595805729d48fd974d761ed8a60158 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, b00f244c69df401997ea99d492218a98; no voters:
I20240503 08:13:01.176220 32534 raft_consensus.cc:2781] T 2c595805729d48fd974d761ed8a60158 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:01.176457 28132 raft_consensus.cc:2454] T 2c595805729d48fd974d761ed8a60158 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 1.
I20240503 08:13:01.176529 32534 raft_consensus.cc:686] T 2c595805729d48fd974d761ed8a60158 P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Becoming Leader. State: Replica: b00f244c69df401997ea99d492218a98, State: Running, Role: LEADER
I20240503 08:13:01.176836 32534 consensus_queue.cc:237] T 2c595805729d48fd974d761ed8a60158 P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:01.180502 27854 catalog_manager.cc:5484] T 2c595805729d48fd974d761ed8a60158 P b00f244c69df401997ea99d492218a98 reported cstate change: term changed from 0 to 1, leader changed from <none> to b00f244c69df401997ea99d492218a98 (127.20.230.72). New cstate: current_term: 1 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:01.231608 27854 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:13:01.231866 27854 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:13:01.235579 27854 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet df973fab80774584b29be6802dd5ca77
I20240503 08:13:01.236325 30741 tablet_service.cc:1514] Processing DeleteTablet for tablet df973fab80774584b29be6802dd5ca77 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:01 UTC) from {username='slave'} at 127.0.0.1:46794
I20240503 08:13:01.236564 28111 tablet_service.cc:1514] Processing DeleteTablet for tablet df973fab80774584b29be6802dd5ca77 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:01 UTC) from {username='slave'} at 127.0.0.1:57912
I20240503 08:13:01.237473 32568 tablet_replica.cc:330] T df973fab80774584b29be6802dd5ca77 P 1b411c6bb25549368041773d7b710d48: stopping tablet replica
I20240503 08:13:01.237744 32568 raft_consensus.cc:2229] T df973fab80774584b29be6802dd5ca77 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:01.237998 32568 raft_consensus.cc:2258] T df973fab80774584b29be6802dd5ca77 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:01.238526 32567 tablet_replica.cc:330] T df973fab80774584b29be6802dd5ca77 P 6d1d7de19459435fbde1999e3dfd226b: stopping tablet replica
I20240503 08:13:01.238476 27981 tablet_service.cc:1514] Processing DeleteTablet for tablet df973fab80774584b29be6802dd5ca77 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:01 UTC) from {username='slave'} at 127.0.0.1:55610
I20240503 08:13:01.238778 32568 ts_tablet_manager.cc:1875] T df973fab80774584b29be6802dd5ca77 P 1b411c6bb25549368041773d7b710d48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:01.239073 32567 raft_consensus.cc:2229] T df973fab80774584b29be6802dd5ca77 P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:13:01.240322 32567 raft_consensus.cc:2258] T df973fab80774584b29be6802dd5ca77 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:01.241077 32567 ts_tablet_manager.cc:1875] T df973fab80774584b29be6802dd5ca77 P 6d1d7de19459435fbde1999e3dfd226b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:01.244031 32568 ts_tablet_manager.cc:1888] T df973fab80774584b29be6802dd5ca77 P 1b411c6bb25549368041773d7b710d48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:13:01.244246 32568 log.cc:1196] T df973fab80774584b29be6802dd5ca77 P 1b411c6bb25549368041773d7b710d48: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/df973fab80774584b29be6802dd5ca77
I20240503 08:13:01.244587 32568 ts_tablet_manager.cc:1909] T df973fab80774584b29be6802dd5ca77 P 1b411c6bb25549368041773d7b710d48: Deleting consensus metadata
I20240503 08:13:01.246122 27848 catalog_manager.cc:4830] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): tablet df973fab80774584b29be6802dd5ca77 (table rebalancer_dynamic_table_new_name [id=749808d425844a80bc7fa108a2e26403]) successfully deleted
I20240503 08:13:01.247669 32567 ts_tablet_manager.cc:1888] T df973fab80774584b29be6802dd5ca77 P 6d1d7de19459435fbde1999e3dfd226b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:13:01.247793 32567 log.cc:1196] T df973fab80774584b29be6802dd5ca77 P 6d1d7de19459435fbde1999e3dfd226b: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal/wals/df973fab80774584b29be6802dd5ca77
I20240503 08:13:01.248478 32567 ts_tablet_manager.cc:1909] T df973fab80774584b29be6802dd5ca77 P 6d1d7de19459435fbde1999e3dfd226b: Deleting consensus metadata
I20240503 08:13:01.249541 27848 catalog_manager.cc:4830] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): tablet df973fab80774584b29be6802dd5ca77 (table rebalancer_dynamic_table_new_name [id=749808d425844a80bc7fa108a2e26403]) successfully deleted
I20240503 08:13:01.249593 32569 tablet_replica.cc:330] T df973fab80774584b29be6802dd5ca77 P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:13:01.250907 32569 raft_consensus.cc:2229] T df973fab80774584b29be6802dd5ca77 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:01.251087 32569 raft_consensus.cc:2258] T df973fab80774584b29be6802dd5ca77 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:01.251753 32569 ts_tablet_manager.cc:1875] T df973fab80774584b29be6802dd5ca77 P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:01.256676 32569 ts_tablet_manager.cc:1888] T df973fab80774584b29be6802dd5ca77 P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:13:01.256798 32569 log.cc:1196] T df973fab80774584b29be6802dd5ca77 P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/df973fab80774584b29be6802dd5ca77
I20240503 08:13:01.257239 32569 ts_tablet_manager.cc:1909] T df973fab80774584b29be6802dd5ca77 P 833582ff68c94269be364ebd187a75db: Deleting consensus metadata
I20240503 08:13:01.258092 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet df973fab80774584b29be6802dd5ca77 (table rebalancer_dynamic_table_new_name [id=749808d425844a80bc7fa108a2e26403]) successfully deleted
I20240503 08:13:01.338255 27854 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:13:01.344710 30873 tablet_service.cc:1467] Processing CreateTablet for tablet 5cb4a32781d54e8492fcf19bfc85e106 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=689a3a35a67b405ca3b34fc92ec69ab1]), partition=
I20240503 08:13:01.345060 27981 tablet_service.cc:1467] Processing CreateTablet for tablet 5cb4a32781d54e8492fcf19bfc85e106 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=689a3a35a67b405ca3b34fc92ec69ab1]), partition=
I20240503 08:13:01.345118 30873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cb4a32781d54e8492fcf19bfc85e106. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:01.345428 27981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cb4a32781d54e8492fcf19bfc85e106. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:01.345468 28372 tablet_service.cc:1467] Processing CreateTablet for tablet 5cb4a32781d54e8492fcf19bfc85e106 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=689a3a35a67b405ca3b34fc92ec69ab1]), partition=
I20240503 08:13:01.345835 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cb4a32781d54e8492fcf19bfc85e106. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:01.350978 32552 tablet_bootstrap.cc:492] T 5cb4a32781d54e8492fcf19bfc85e106 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:13:01.351120 32528 tablet_bootstrap.cc:492] T 5cb4a32781d54e8492fcf19bfc85e106 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:13:01.352360 32552 tablet_bootstrap.cc:654] T 5cb4a32781d54e8492fcf19bfc85e106 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:01.353127 32528 tablet_bootstrap.cc:654] T 5cb4a32781d54e8492fcf19bfc85e106 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:01.356738 32528 tablet_bootstrap.cc:492] T 5cb4a32781d54e8492fcf19bfc85e106 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:13:01.356974 32528 ts_tablet_manager.cc:1367] T 5cb4a32781d54e8492fcf19bfc85e106 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 08:13:01.357959 32528 raft_consensus.cc:348] T 5cb4a32781d54e8492fcf19bfc85e106 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:01.358994 32528 raft_consensus.cc:374] T 5cb4a32781d54e8492fcf19bfc85e106 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:01.359735 32528 raft_consensus.cc:729] T 5cb4a32781d54e8492fcf19bfc85e106 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:13:01.360069 32528 consensus_queue.cc:260] T 5cb4a32781d54e8492fcf19bfc85e106 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:01.360656 32528 ts_tablet_manager.cc:1398] T 5cb4a32781d54e8492fcf19bfc85e106 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 08:13:01.363791 32552 tablet_bootstrap.cc:492] T 5cb4a32781d54e8492fcf19bfc85e106 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:13:01.363996 32552 ts_tablet_manager.cc:1367] T 5cb4a32781d54e8492fcf19bfc85e106 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.001s
I20240503 08:13:01.364843 32552 raft_consensus.cc:348] T 5cb4a32781d54e8492fcf19bfc85e106 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:01.365058 32552 raft_consensus.cc:374] T 5cb4a32781d54e8492fcf19bfc85e106 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:01.365131 32552 raft_consensus.cc:729] T 5cb4a32781d54e8492fcf19bfc85e106 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:13:01.365325 32571 tablet_bootstrap.cc:492] T 5cb4a32781d54e8492fcf19bfc85e106 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:13:01.365360 32552 consensus_queue.cc:260] T 5cb4a32781d54e8492fcf19bfc85e106 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:01.365959 32552 ts_tablet_manager.cc:1398] T 5cb4a32781d54e8492fcf19bfc85e106 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:13:01.366832 32571 tablet_bootstrap.cc:654] T 5cb4a32781d54e8492fcf19bfc85e106 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:01.368497 32571 tablet_bootstrap.cc:492] T 5cb4a32781d54e8492fcf19bfc85e106 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:13:01.368633 32571 ts_tablet_manager.cc:1367] T 5cb4a32781d54e8492fcf19bfc85e106 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:13:01.369257 32571 raft_consensus.cc:348] T 5cb4a32781d54e8492fcf19bfc85e106 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:01.369679 32571 raft_consensus.cc:374] T 5cb4a32781d54e8492fcf19bfc85e106 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:01.369817 32571 raft_consensus.cc:729] T 5cb4a32781d54e8492fcf19bfc85e106 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:13:01.370069 32571 consensus_queue.cc:260] T 5cb4a32781d54e8492fcf19bfc85e106 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:01.370604 32571 ts_tablet_manager.cc:1398] T 5cb4a32781d54e8492fcf19bfc85e106 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:01.543166 32536 consensus_queue.cc:1035] T c8ca21ffe16e41ab80f5693dc4dbb682 P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:01.552630 32496 consensus_queue.cc:1035] T c8ca21ffe16e41ab80f5693dc4dbb682 P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:01.590888 32493 consensus_queue.cc:1035] T 2c595805729d48fd974d761ed8a60158 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:01.594758 32493 consensus_queue.cc:1035] T 2c595805729d48fd974d761ed8a60158 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:01.612149 32486 raft_consensus.cc:482] T 5cb4a32781d54e8492fcf19bfc85e106 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:01.612356 32486 raft_consensus.cc:504] T 5cb4a32781d54e8492fcf19bfc85e106 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:01.613179 32486 leader_election.cc:288] T 5cb4a32781d54e8492fcf19bfc85e106 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:13:01.613512 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cb4a32781d54e8492fcf19bfc85e106" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:13:01.613739 30895 raft_consensus.cc:2454] T 5cb4a32781d54e8492fcf19bfc85e106 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:13:01.613818 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cb4a32781d54e8492fcf19bfc85e106" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:13:01.614048 28003 raft_consensus.cc:2454] T 5cb4a32781d54e8492fcf19bfc85e106 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:13:01.614265 28326 leader_election.cc:302] T 5cb4a32781d54e8492fcf19bfc85e106 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:13:01.614537 32486 raft_consensus.cc:2781] T 5cb4a32781d54e8492fcf19bfc85e106 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:01.614631 32486 raft_consensus.cc:482] T 5cb4a32781d54e8492fcf19bfc85e106 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:01.614730 32486 raft_consensus.cc:3037] T 5cb4a32781d54e8492fcf19bfc85e106 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:01.617111 32486 raft_consensus.cc:504] T 5cb4a32781d54e8492fcf19bfc85e106 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:01.617739 32486 leader_election.cc:288] T 5cb4a32781d54e8492fcf19bfc85e106 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Requested vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:13:01.618091 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cb4a32781d54e8492fcf19bfc85e106" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:13:01.618278 30895 raft_consensus.cc:3037] T 5cb4a32781d54e8492fcf19bfc85e106 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:01.620329 30895 raft_consensus.cc:2454] T 5cb4a32781d54e8492fcf19bfc85e106 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:13:01.620743 28326 leader_election.cc:302] T 5cb4a32781d54e8492fcf19bfc85e106 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:13:01.621006 32486 raft_consensus.cc:2781] T 5cb4a32781d54e8492fcf19bfc85e106 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:01.621148 32486 raft_consensus.cc:686] T 5cb4a32781d54e8492fcf19bfc85e106 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Becoming Leader. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Running, Role: LEADER
I20240503 08:13:01.621519 32486 consensus_queue.cc:237] T 5cb4a32781d54e8492fcf19bfc85e106 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:01.623311 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cb4a32781d54e8492fcf19bfc85e106" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:13:01.623530 28003 raft_consensus.cc:3037] T 5cb4a32781d54e8492fcf19bfc85e106 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:01.627379 28003 raft_consensus.cc:2454] T 5cb4a32781d54e8492fcf19bfc85e106 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:13:01.632900 27854 catalog_manager.cc:5484] T 5cb4a32781d54e8492fcf19bfc85e106 P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68). New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:01.651453 32517 consensus_queue.cc:1035] T 2e511a3ab0ee4eabb22240de99690603 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:01.674253 32452 consensus_queue.cc:1035] T 2e511a3ab0ee4eabb22240de99690603 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:01.716557 27854 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_extra_table_48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:13:01.725683 31006 tablet_service.cc:1467] Processing CreateTablet for tablet b52e4b59153d40378e2a37809b04a2c8 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=65fb43133ebb48fa9b6c1fe9b9dd6f27]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:01.725735 30873 tablet_service.cc:1467] Processing CreateTablet for tablet b52e4b59153d40378e2a37809b04a2c8 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=65fb43133ebb48fa9b6c1fe9b9dd6f27]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:01.726168 31006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b52e4b59153d40378e2a37809b04a2c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:01.726194 30873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b52e4b59153d40378e2a37809b04a2c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:01.727403 30874 tablet_service.cc:1467] Processing CreateTablet for tablet 620b0cdb874d4efe8d49c971a48ee924 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=65fb43133ebb48fa9b6c1fe9b9dd6f27]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:01.727813 30874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 620b0cdb874d4efe8d49c971a48ee924. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:01.730867 28243 tablet_service.cc:1467] Processing CreateTablet for tablet 620b0cdb874d4efe8d49c971a48ee924 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=65fb43133ebb48fa9b6c1fe9b9dd6f27]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:01.731014 31007 tablet_service.cc:1467] Processing CreateTablet for tablet 4ebf45bdc432451e87b7c98d435a76d2 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=65fb43133ebb48fa9b6c1fe9b9dd6f27]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:01.731467 31007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ebf45bdc432451e87b7c98d435a76d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:01.731590 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 620b0cdb874d4efe8d49c971a48ee924. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:01.731657 30741 tablet_service.cc:1467] Processing CreateTablet for tablet 4ebf45bdc432451e87b7c98d435a76d2 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=65fb43133ebb48fa9b6c1fe9b9dd6f27]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:01.732033 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ebf45bdc432451e87b7c98d435a76d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:01.734424 32552 tablet_bootstrap.cc:492] T 620b0cdb874d4efe8d49c971a48ee924 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:13:01.734601 28111 tablet_service.cc:1467] Processing CreateTablet for tablet b52e4b59153d40378e2a37809b04a2c8 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=65fb43133ebb48fa9b6c1fe9b9dd6f27]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:01.734984 28111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b52e4b59153d40378e2a37809b04a2c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:01.735893 32552 tablet_bootstrap.cc:654] T 620b0cdb874d4efe8d49c971a48ee924 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:01.742206 32591 tablet_bootstrap.cc:492] T 4ebf45bdc432451e87b7c98d435a76d2 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:13:01.742506 31140 tablet_service.cc:1467] Processing CreateTablet for tablet 620b0cdb874d4efe8d49c971a48ee924 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=65fb43133ebb48fa9b6c1fe9b9dd6f27]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:01.742563 31141 tablet_service.cc:1467] Processing CreateTablet for tablet 4ebf45bdc432451e87b7c98d435a76d2 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=65fb43133ebb48fa9b6c1fe9b9dd6f27]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:01.743400 31141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ebf45bdc432451e87b7c98d435a76d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:01.743721 27856 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:13:01.746995 31140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 620b0cdb874d4efe8d49c971a48ee924. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:01.747164 32592 tablet_bootstrap.cc:492] T 4ebf45bdc432451e87b7c98d435a76d2 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:13:01.747356 32590 tablet_bootstrap.cc:492] T b52e4b59153d40378e2a37809b04a2c8 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:13:01.748849 32591 tablet_bootstrap.cc:654] T 4ebf45bdc432451e87b7c98d435a76d2 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:01.748991 32590 tablet_bootstrap.cc:654] T b52e4b59153d40378e2a37809b04a2c8 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:01.752403 32552 tablet_bootstrap.cc:492] T 620b0cdb874d4efe8d49c971a48ee924 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:13:01.752434 32592 tablet_bootstrap.cc:654] T 4ebf45bdc432451e87b7c98d435a76d2 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:01.752625 32552 ts_tablet_manager.cc:1367] T 620b0cdb874d4efe8d49c971a48ee924 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.005s
I20240503 08:13:01.753556 32552 raft_consensus.cc:348] T 620b0cdb874d4efe8d49c971a48ee924 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:01.753851 32552 raft_consensus.cc:374] T 620b0cdb874d4efe8d49c971a48ee924 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:01.754087 32552 raft_consensus.cc:729] T 620b0cdb874d4efe8d49c971a48ee924 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:13:01.754441 32552 consensus_queue.cc:260] T 620b0cdb874d4efe8d49c971a48ee924 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:01.755146 32552 ts_tablet_manager.cc:1398] T 620b0cdb874d4efe8d49c971a48ee924 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:13:01.756222 32589 tablet_bootstrap.cc:492] T 620b0cdb874d4efe8d49c971a48ee924 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:13:01.758026 32589 tablet_bootstrap.cc:654] T 620b0cdb874d4efe8d49c971a48ee924 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:01.758234 32591 tablet_bootstrap.cc:492] T 4ebf45bdc432451e87b7c98d435a76d2 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:13:01.758450 32591 ts_tablet_manager.cc:1367] T 4ebf45bdc432451e87b7c98d435a76d2 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20240503 08:13:01.759796 32591 raft_consensus.cc:348] T 4ebf45bdc432451e87b7c98d435a76d2 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:01.760061 32591 raft_consensus.cc:374] T 4ebf45bdc432451e87b7c98d435a76d2 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:01.760200 32591 raft_consensus.cc:729] T 4ebf45bdc432451e87b7c98d435a76d2 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:13:01.760505 32591 consensus_queue.cc:260] T 4ebf45bdc432451e87b7c98d435a76d2 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:01.763701 32591 ts_tablet_manager.cc:1398] T 4ebf45bdc432451e87b7c98d435a76d2 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:13:01.764017 32591 tablet_bootstrap.cc:492] T b52e4b59153d40378e2a37809b04a2c8 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:13:01.772295 32552 tablet_bootstrap.cc:492] T b52e4b59153d40378e2a37809b04a2c8 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:13:01.772852 32591 tablet_bootstrap.cc:654] T b52e4b59153d40378e2a37809b04a2c8 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:01.774039 32552 tablet_bootstrap.cc:654] T b52e4b59153d40378e2a37809b04a2c8 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:01.775255 32598 tablet_bootstrap.cc:492] T 4ebf45bdc432451e87b7c98d435a76d2 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:13:01.777796 32598 tablet_bootstrap.cc:654] T 4ebf45bdc432451e87b7c98d435a76d2 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:01.781505 32591 tablet_bootstrap.cc:492] T b52e4b59153d40378e2a37809b04a2c8 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:13:01.781745 32456 raft_consensus.cc:482] T 4ebf45bdc432451e87b7c98d435a76d2 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:01.781883 32456 raft_consensus.cc:504] T 4ebf45bdc432451e87b7c98d435a76d2 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:01.782335 32589 tablet_bootstrap.cc:492] T 620b0cdb874d4efe8d49c971a48ee924 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:13:01.782488 32589 ts_tablet_manager.cc:1367] T 620b0cdb874d4efe8d49c971a48ee924 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.009s
I20240503 08:13:01.782573 32456 leader_election.cc:288] T 4ebf45bdc432451e87b7c98d435a76d2 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:13:01.783058 31160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ebf45bdc432451e87b7c98d435a76d2" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:13:01.781659 32591 ts_tablet_manager.cc:1367] T b52e4b59153d40378e2a37809b04a2c8 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20240503 08:13:01.783228 32589 raft_consensus.cc:348] T 620b0cdb874d4efe8d49c971a48ee924 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:01.783440 32589 raft_consensus.cc:374] T 620b0cdb874d4efe8d49c971a48ee924 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:01.783952 32591 raft_consensus.cc:348] T b52e4b59153d40378e2a37809b04a2c8 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:01.784794 32598 tablet_bootstrap.cc:492] T 4ebf45bdc432451e87b7c98d435a76d2 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:13:01.784930 32589 raft_consensus.cc:729] T 620b0cdb874d4efe8d49c971a48ee924 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:13:01.784978 32598 ts_tablet_manager.cc:1367] T 4ebf45bdc432451e87b7c98d435a76d2 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.005s
I20240503 08:13:01.785328 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ebf45bdc432451e87b7c98d435a76d2" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:13:01.785517 28003 raft_consensus.cc:1261] T 5cb4a32781d54e8492fcf19bfc85e106 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:01.785701 32598 raft_consensus.cc:348] T 4ebf45bdc432451e87b7c98d435a76d2 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:01.785990 32598 raft_consensus.cc:374] T 4ebf45bdc432451e87b7c98d435a76d2 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:01.786131 32598 raft_consensus.cc:729] T 4ebf45bdc432451e87b7c98d435a76d2 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:13:01.786123 32486 consensus_queue.cc:1035] T 5cb4a32781d54e8492fcf19bfc85e106 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:01.786427 32598 consensus_queue.cc:260] T 4ebf45bdc432451e87b7c98d435a76d2 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:01.786509 32589 consensus_queue.cc:260] T 620b0cdb874d4efe8d49c971a48ee924 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:01.789373 32589 ts_tablet_manager.cc:1398] T 620b0cdb874d4efe8d49c971a48ee924 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240503 08:13:01.789376 32598 ts_tablet_manager.cc:1398] T 4ebf45bdc432451e87b7c98d435a76d2 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 08:13:01.789733 32598 tablet_bootstrap.cc:492] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:13:01.791416 32598 tablet_bootstrap.cc:654] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:01.794411 32590 tablet_bootstrap.cc:492] T b52e4b59153d40378e2a37809b04a2c8 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:13:01.797472 32590 ts_tablet_manager.cc:1367] T b52e4b59153d40378e2a37809b04a2c8 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.047s user 0.001s sys 0.005s
I20240503 08:13:01.798381 32590 raft_consensus.cc:348] T b52e4b59153d40378e2a37809b04a2c8 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:01.799305 32590 raft_consensus.cc:374] T b52e4b59153d40378e2a37809b04a2c8 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:01.799456 32552 tablet_bootstrap.cc:492] T b52e4b59153d40378e2a37809b04a2c8 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:13:01.799453 32590 raft_consensus.cc:729] T b52e4b59153d40378e2a37809b04a2c8 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:13:01.799635 32552 ts_tablet_manager.cc:1367] T b52e4b59153d40378e2a37809b04a2c8 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.027s user 0.011s sys 0.001s
I20240503 08:13:01.800432 32552 raft_consensus.cc:348] T b52e4b59153d40378e2a37809b04a2c8 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:01.800660 32552 raft_consensus.cc:374] T b52e4b59153d40378e2a37809b04a2c8 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:01.800727 32552 raft_consensus.cc:729] T b52e4b59153d40378e2a37809b04a2c8 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:13:01.801007 32552 consensus_queue.cc:260] T b52e4b59153d40378e2a37809b04a2c8 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:01.801672 32552 ts_tablet_manager.cc:1398] T b52e4b59153d40378e2a37809b04a2c8 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:13:01.801884 32590 consensus_queue.cc:260] T b52e4b59153d40378e2a37809b04a2c8 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:01.802630 32590 ts_tablet_manager.cc:1398] T b52e4b59153d40378e2a37809b04a2c8 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:13:01.806818 27855 catalog_manager.cc:5576] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081) does not have the latest schema for tablet 5cb4a32781d54e8492fcf19bfc85e106 (table rebalancer_dynamic_table [id=689a3a35a67b405ca3b34fc92ec69ab1]). Expected version 1 got 0
I20240503 08:13:01.807235 32598 tablet_bootstrap.cc:492] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:13:01.807385 32598 ts_tablet_manager.cc:1367] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.001s
I20240503 08:13:01.808691 32591 raft_consensus.cc:374] T b52e4b59153d40378e2a37809b04a2c8 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:01.808851 32591 raft_consensus.cc:729] T b52e4b59153d40378e2a37809b04a2c8 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:13:01.809063 30895 raft_consensus.cc:1261] T 5cb4a32781d54e8492fcf19bfc85e106 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:01.808106 32598 raft_consensus.cc:348] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
W20240503 08:13:01.785851 30959 leader_election.cc:341] T 4ebf45bdc432451e87b7c98d435a76d2 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:13:01.809191 32591 consensus_queue.cc:260] T b52e4b59153d40378e2a37809b04a2c8 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:01.809269 32598 raft_consensus.cc:374] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:01.809687 32598 raft_consensus.cc:729] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
W20240503 08:13:01.809715 30961 leader_election.cc:341] T 4ebf45bdc432451e87b7c98d435a76d2 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:13:01.809938 30961 leader_election.cc:302] T 4ebf45bdc432451e87b7c98d435a76d2 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8; no voters: 6d1d7de19459435fbde1999e3dfd226b, b00f244c69df401997ea99d492218a98
I20240503 08:13:01.809984 32598 consensus_queue.cc:260] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:01.810267 32535 raft_consensus.cc:2726] T 4ebf45bdc432451e87b7c98d435a76d2 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:13:01.811082 32592 tablet_bootstrap.cc:492] T 4ebf45bdc432451e87b7c98d435a76d2 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:13:01.811292 32592 ts_tablet_manager.cc:1367] T 4ebf45bdc432451e87b7c98d435a76d2 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.064s user 0.002s sys 0.008s
I20240503 08:13:01.811990 32486 consensus_queue.cc:1035] T 5cb4a32781d54e8492fcf19bfc85e106 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:01.813095 32592 raft_consensus.cc:348] T 4ebf45bdc432451e87b7c98d435a76d2 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:01.813295 32592 raft_consensus.cc:374] T 4ebf45bdc432451e87b7c98d435a76d2 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:01.813395 32592 raft_consensus.cc:729] T 4ebf45bdc432451e87b7c98d435a76d2 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:13:01.813616 32592 consensus_queue.cc:260] T 4ebf45bdc432451e87b7c98d435a76d2 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:01.814329 32598 ts_tablet_manager.cc:1398] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.007s user 0.001s sys 0.002s
I20240503 08:13:01.816180 32607 tablet.cc:1722] T 5cb4a32781d54e8492fcf19bfc85e106 P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:13:01.816704 32607 tablet.cc:1620] T 5cb4a32781d54e8492fcf19bfc85e106 P ce1496a47d084365ad786cd09e472a2c: MemRowSet was empty: no flush needed.
I20240503 08:13:01.817505 32592 ts_tablet_manager.cc:1398] T 4ebf45bdc432451e87b7c98d435a76d2 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:13:01.820806 27856 catalog_manager.cc:5576] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747) does not have the latest schema for tablet 5cb4a32781d54e8492fcf19bfc85e106 (table rebalancer_dynamic_table [id=689a3a35a67b405ca3b34fc92ec69ab1]). Expected version 1 got 0
I20240503 08:13:01.805183 32604 tablet.cc:1722] T 5cb4a32781d54e8492fcf19bfc85e106 P d09a11a03989421b9f28019dc1c14cc2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:13:01.822441 32604 tablet.cc:1620] T 5cb4a32781d54e8492fcf19bfc85e106 P d09a11a03989421b9f28019dc1c14cc2: MemRowSet was empty: no flush needed.
I20240503 08:13:01.826264 27854 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=689a3a35a67b405ca3b34fc92ec69ab1] alter complete (version 1)
I20240503 08:13:01.830222 32591 ts_tablet_manager.cc:1398] T b52e4b59153d40378e2a37809b04a2c8 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.047s user 0.001s sys 0.000s
I20240503 08:13:01.830672 32605 tablet.cc:1722] T 5cb4a32781d54e8492fcf19bfc85e106 P 833582ff68c94269be364ebd187a75db: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:13:01.831218 32605 tablet.cc:1620] T 5cb4a32781d54e8492fcf19bfc85e106 P 833582ff68c94269be364ebd187a75db: MemRowSet was empty: no flush needed.
I20240503 08:13:01.833218 27856 catalog_manager.cc:5576] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081) does not have the latest schema for tablet 5cb4a32781d54e8492fcf19bfc85e106 (table rebalancer_dynamic_table [id=689a3a35a67b405ca3b34fc92ec69ab1]). Expected version 1 got 0
I20240503 08:13:01.852502 32493 raft_consensus.cc:482] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:01.852716 32493 raft_consensus.cc:504] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:01.853500 32493 leader_election.cc:288] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:13:01.854197 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "620b0cdb874d4efe8d49c971a48ee924" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:13:01.854450 30895 raft_consensus.cc:2454] T 620b0cdb874d4efe8d49c971a48ee924 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 0.
I20240503 08:13:01.854853 31094 leader_election.cc:302] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:13:01.855204 32493 raft_consensus.cc:2781] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:01.855312 32493 raft_consensus.cc:482] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:01.855410 32493 raft_consensus.cc:3037] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:01.856438 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "620b0cdb874d4efe8d49c971a48ee924" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:13:01.856637 28262 raft_consensus.cc:2454] T 620b0cdb874d4efe8d49c971a48ee924 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 0.
I20240503 08:13:01.858422 32493 raft_consensus.cc:504] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:01.858963 32493 leader_election.cc:288] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 election: Requested vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:13:01.859249 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "620b0cdb874d4efe8d49c971a48ee924" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:13:01.859395 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "620b0cdb874d4efe8d49c971a48ee924" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:13:01.859462 30895 raft_consensus.cc:3037] T 620b0cdb874d4efe8d49c971a48ee924 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:01.859560 28262 raft_consensus.cc:3037] T 620b0cdb874d4efe8d49c971a48ee924 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:01.862622 30895 raft_consensus.cc:2454] T 620b0cdb874d4efe8d49c971a48ee924 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 1.
I20240503 08:13:01.862982 31094 leader_election.cc:302] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:13:01.864470 28262 raft_consensus.cc:2454] T 620b0cdb874d4efe8d49c971a48ee924 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 1.
I20240503 08:13:01.865505 32493 raft_consensus.cc:2781] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:01.865718 32493 raft_consensus.cc:686] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Becoming Leader. State: Replica: b00f244c69df401997ea99d492218a98, State: Running, Role: LEADER
I20240503 08:13:01.866189 32493 consensus_queue.cc:237] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:01.869666 27854 catalog_manager.cc:5484] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98 reported cstate change: term changed from 0 to 1, leader changed from <none> to b00f244c69df401997ea99d492218a98 (127.20.230.72). New cstate: current_term: 1 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:13:01.910995 32566 raft_consensus.cc:482] T b52e4b59153d40378e2a37809b04a2c8 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:01.911173 32566 raft_consensus.cc:504] T b52e4b59153d40378e2a37809b04a2c8 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:01.911998 32566 leader_election.cc:288] T b52e4b59153d40378e2a37809b04a2c8 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:13:01.912364 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b52e4b59153d40378e2a37809b04a2c8" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:13:01.912379 31026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b52e4b59153d40378e2a37809b04a2c8" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:13:01.912552 30895 raft_consensus.cc:2454] T b52e4b59153d40378e2a37809b04a2c8 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:13:01.912566 31026 raft_consensus.cc:2454] T b52e4b59153d40378e2a37809b04a2c8 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:13:01.912909 28066 leader_election.cc:302] T b52e4b59153d40378e2a37809b04a2c8 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:13:01.913178 32566 raft_consensus.cc:2781] T b52e4b59153d40378e2a37809b04a2c8 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:01.913293 32566 raft_consensus.cc:482] T b52e4b59153d40378e2a37809b04a2c8 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:01.913376 32566 raft_consensus.cc:3037] T b52e4b59153d40378e2a37809b04a2c8 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:01.915436 32566 raft_consensus.cc:504] T b52e4b59153d40378e2a37809b04a2c8 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:01.916118 32566 leader_election.cc:288] T b52e4b59153d40378e2a37809b04a2c8 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Requested vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:13:01.916545 31026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b52e4b59153d40378e2a37809b04a2c8" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:13:01.916700 31026 raft_consensus.cc:3037] T b52e4b59153d40378e2a37809b04a2c8 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:01.919082 31026 raft_consensus.cc:2454] T b52e4b59153d40378e2a37809b04a2c8 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:13:01.919435 28067 leader_election.cc:302] T b52e4b59153d40378e2a37809b04a2c8 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:13:01.919400 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b52e4b59153d40378e2a37809b04a2c8" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:13:01.919581 30895 raft_consensus.cc:3037] T b52e4b59153d40378e2a37809b04a2c8 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:01.919699 32566 raft_consensus.cc:2781] T b52e4b59153d40378e2a37809b04a2c8 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:01.919836 32566 raft_consensus.cc:686] T b52e4b59153d40378e2a37809b04a2c8 P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Becoming Leader. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Running, Role: LEADER
I20240503 08:13:01.920080 32566 consensus_queue.cc:237] T b52e4b59153d40378e2a37809b04a2c8 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:01.921986 30895 raft_consensus.cc:2454] T b52e4b59153d40378e2a37809b04a2c8 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:13:01.924415 27856 catalog_manager.cc:5484] T b52e4b59153d40378e2a37809b04a2c8 P 1b411c6bb25549368041773d7b710d48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b411c6bb25549368041773d7b710d48 (127.20.230.66). New cstate: current_term: 1 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:01.932559 27856 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:13:01.940301 32604 tablet.cc:1722] T 5cb4a32781d54e8492fcf19bfc85e106 P d09a11a03989421b9f28019dc1c14cc2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:13:01.940788 32604 tablet.cc:1620] T 5cb4a32781d54e8492fcf19bfc85e106 P d09a11a03989421b9f28019dc1c14cc2: MemRowSet was empty: no flush needed.
I20240503 08:13:01.941048 32607 tablet.cc:1722] T 5cb4a32781d54e8492fcf19bfc85e106 P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:13:01.941407 32607 tablet.cc:1620] T 5cb4a32781d54e8492fcf19bfc85e106 P ce1496a47d084365ad786cd09e472a2c: MemRowSet was empty: no flush needed.
I20240503 08:13:01.941488 32605 tablet.cc:1722] T 5cb4a32781d54e8492fcf19bfc85e106 P 833582ff68c94269be364ebd187a75db: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:13:01.941951 32605 tablet.cc:1620] T 5cb4a32781d54e8492fcf19bfc85e106 P 833582ff68c94269be364ebd187a75db: MemRowSet was empty: no flush needed.
I20240503 08:13:01.949183 32517 raft_consensus.cc:482] T 4ebf45bdc432451e87b7c98d435a76d2 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:01.949362 32517 raft_consensus.cc:504] T 4ebf45bdc432451e87b7c98d435a76d2 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:01.950079 32517 leader_election.cc:288] T 4ebf45bdc432451e87b7c98d435a76d2 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:13:01.950425 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ebf45bdc432451e87b7c98d435a76d2" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:13:01.950649 31027 raft_consensus.cc:2454] T 4ebf45bdc432451e87b7c98d435a76d2 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:13:01.951239 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ebf45bdc432451e87b7c98d435a76d2" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:13:01.951437 31159 raft_consensus.cc:2454] T 4ebf45bdc432451e87b7c98d435a76d2 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:13:01.951994 30694 leader_election.cc:302] T 4ebf45bdc432451e87b7c98d435a76d2 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, b00f244c69df401997ea99d492218a98; no voters:
I20240503 08:13:01.952270 32517 raft_consensus.cc:2781] T 4ebf45bdc432451e87b7c98d435a76d2 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:01.952378 32517 raft_consensus.cc:482] T 4ebf45bdc432451e87b7c98d435a76d2 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:01.952430 32517 raft_consensus.cc:3037] T 4ebf45bdc432451e87b7c98d435a76d2 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:01.954412 27856 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=689a3a35a67b405ca3b34fc92ec69ab1] alter complete (version 2)
I20240503 08:13:01.954669 32517 raft_consensus.cc:504] T 4ebf45bdc432451e87b7c98d435a76d2 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:01.955318 32517 leader_election.cc:288] T 4ebf45bdc432451e87b7c98d435a76d2 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Requested vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:13:01.956211 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ebf45bdc432451e87b7c98d435a76d2" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:13:01.956269 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ebf45bdc432451e87b7c98d435a76d2" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:13:01.956410 31027 raft_consensus.cc:3037] T 4ebf45bdc432451e87b7c98d435a76d2 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:01.956446 31159 raft_consensus.cc:3037] T 4ebf45bdc432451e87b7c98d435a76d2 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:01.958706 31159 raft_consensus.cc:2454] T 4ebf45bdc432451e87b7c98d435a76d2 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:13:01.958746 31027 raft_consensus.cc:2454] T 4ebf45bdc432451e87b7c98d435a76d2 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:13:01.959131 30694 leader_election.cc:302] T 4ebf45bdc432451e87b7c98d435a76d2 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, b00f244c69df401997ea99d492218a98, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:13:01.959420 32517 raft_consensus.cc:2781] T 4ebf45bdc432451e87b7c98d435a76d2 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:01.959551 32517 raft_consensus.cc:686] T 4ebf45bdc432451e87b7c98d435a76d2 P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Becoming Leader. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Running, Role: LEADER
I20240503 08:13:01.959988 32517 consensus_queue.cc:237] T 4ebf45bdc432451e87b7c98d435a76d2 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:01.962833 27856 catalog_manager.cc:5484] T 4ebf45bdc432451e87b7c98d435a76d2 P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69). New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:13:02.060065 27856 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:13:02.067739 32604 tablet.cc:1722] T 5cb4a32781d54e8492fcf19bfc85e106 P d09a11a03989421b9f28019dc1c14cc2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:13:02.068447 32607 tablet.cc:1722] T 5cb4a32781d54e8492fcf19bfc85e106 P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:13:02.076501 32605 tablet.cc:1722] T 5cb4a32781d54e8492fcf19bfc85e106 P 833582ff68c94269be364ebd187a75db: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:13:02.077982 27856 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table_new_name [id=689a3a35a67b405ca3b34fc92ec69ab1] alter complete (version 3)
I20240503 08:13:02.180932 27855 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:13:02.181149 27855 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:13:02.184214 27855 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet 5cb4a32781d54e8492fcf19bfc85e106
I20240503 08:13:02.185104 30873 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cb4a32781d54e8492fcf19bfc85e106 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:02 UTC) from {username='slave'} at 127.0.0.1:34426
I20240503 08:13:02.185264 28372 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cb4a32781d54e8492fcf19bfc85e106 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:02 UTC) from {username='slave'} at 127.0.0.1:34786
I20240503 08:13:02.185719 32612 tablet_replica.cc:330] T 5cb4a32781d54e8492fcf19bfc85e106 P ce1496a47d084365ad786cd09e472a2c: stopping tablet replica
I20240503 08:13:02.185787 27981 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cb4a32781d54e8492fcf19bfc85e106 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:02 UTC) from {username='slave'} at 127.0.0.1:55610
I20240503 08:13:02.185948 32612 raft_consensus.cc:2229] T 5cb4a32781d54e8492fcf19bfc85e106 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:02.186164 32612 raft_consensus.cc:2258] T 5cb4a32781d54e8492fcf19bfc85e106 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:02.186395 32614 tablet_replica.cc:330] T 5cb4a32781d54e8492fcf19bfc85e106 P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:13:02.186606 32614 raft_consensus.cc:2229] T 5cb4a32781d54e8492fcf19bfc85e106 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:02.186832 32614 raft_consensus.cc:2258] T 5cb4a32781d54e8492fcf19bfc85e106 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:02.187090 32612 ts_tablet_manager.cc:1875] T 5cb4a32781d54e8492fcf19bfc85e106 P ce1496a47d084365ad786cd09e472a2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:02.187783 32614 ts_tablet_manager.cc:1875] T 5cb4a32781d54e8492fcf19bfc85e106 P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:02.190130 32613 tablet_replica.cc:330] T 5cb4a32781d54e8492fcf19bfc85e106 P d09a11a03989421b9f28019dc1c14cc2: stopping tablet replica
I20240503 08:13:02.190423 32613 raft_consensus.cc:2229] T 5cb4a32781d54e8492fcf19bfc85e106 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:13:02.190796 32613 raft_consensus.cc:2258] T 5cb4a32781d54e8492fcf19bfc85e106 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:02.191474 32613 ts_tablet_manager.cc:1875] T 5cb4a32781d54e8492fcf19bfc85e106 P d09a11a03989421b9f28019dc1c14cc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:02.192742 32612 ts_tablet_manager.cc:1888] T 5cb4a32781d54e8492fcf19bfc85e106 P ce1496a47d084365ad786cd09e472a2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:13:02.192890 32612 log.cc:1196] T 5cb4a32781d54e8492fcf19bfc85e106 P ce1496a47d084365ad786cd09e472a2c: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal/wals/5cb4a32781d54e8492fcf19bfc85e106
I20240503 08:13:02.193364 32612 ts_tablet_manager.cc:1909] T 5cb4a32781d54e8492fcf19bfc85e106 P ce1496a47d084365ad786cd09e472a2c: Deleting consensus metadata
I20240503 08:13:02.194378 27846 catalog_manager.cc:4830] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): tablet 5cb4a32781d54e8492fcf19bfc85e106 (table rebalancer_dynamic_table_new_name [id=689a3a35a67b405ca3b34fc92ec69ab1]) successfully deleted
I20240503 08:13:02.194792 32614 ts_tablet_manager.cc:1888] T 5cb4a32781d54e8492fcf19bfc85e106 P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:13:02.194891 32614 log.cc:1196] T 5cb4a32781d54e8492fcf19bfc85e106 P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/5cb4a32781d54e8492fcf19bfc85e106
I20240503 08:13:02.195246 32614 ts_tablet_manager.cc:1909] T 5cb4a32781d54e8492fcf19bfc85e106 P 833582ff68c94269be364ebd187a75db: Deleting consensus metadata
I20240503 08:13:02.195997 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet 5cb4a32781d54e8492fcf19bfc85e106 (table rebalancer_dynamic_table_new_name [id=689a3a35a67b405ca3b34fc92ec69ab1]) successfully deleted
I20240503 08:13:02.196861 32613 ts_tablet_manager.cc:1888] T 5cb4a32781d54e8492fcf19bfc85e106 P d09a11a03989421b9f28019dc1c14cc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:13:02.196978 32613 log.cc:1196] T 5cb4a32781d54e8492fcf19bfc85e106 P d09a11a03989421b9f28019dc1c14cc2: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/wals/5cb4a32781d54e8492fcf19bfc85e106
I20240503 08:13:02.197264 32613 ts_tablet_manager.cc:1909] T 5cb4a32781d54e8492fcf19bfc85e106 P d09a11a03989421b9f28019dc1c14cc2: Deleting consensus metadata
I20240503 08:13:02.198057 27849 catalog_manager.cc:4830] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): tablet 5cb4a32781d54e8492fcf19bfc85e106 (table rebalancer_dynamic_table_new_name [id=689a3a35a67b405ca3b34fc92ec69ab1]) successfully deleted
I20240503 08:13:02.286499 27855 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:13:02.294173 28243 tablet_service.cc:1467] Processing CreateTablet for tablet d9761e1466cd478d9af8049536ddd6fb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6e2e9373551b4f30a1a4081f42994787]), partition=
I20240503 08:13:02.294373 31006 tablet_service.cc:1467] Processing CreateTablet for tablet d9761e1466cd478d9af8049536ddd6fb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6e2e9373551b4f30a1a4081f42994787]), partition=
I20240503 08:13:02.294554 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9761e1466cd478d9af8049536ddd6fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:02.294742 31006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9761e1466cd478d9af8049536ddd6fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:02.298048 28372 tablet_service.cc:1467] Processing CreateTablet for tablet d9761e1466cd478d9af8049536ddd6fb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6e2e9373551b4f30a1a4081f42994787]), partition=
I20240503 08:13:02.298311 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9761e1466cd478d9af8049536ddd6fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:02.300805 32591 tablet_bootstrap.cc:492] T d9761e1466cd478d9af8049536ddd6fb P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:13:02.302208 32591 tablet_bootstrap.cc:654] T d9761e1466cd478d9af8049536ddd6fb P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:02.304811 32591 tablet_bootstrap.cc:492] T d9761e1466cd478d9af8049536ddd6fb P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:13:02.304968 32591 ts_tablet_manager.cc:1367] T d9761e1466cd478d9af8049536ddd6fb P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:13:02.305606 32591 raft_consensus.cc:348] T d9761e1466cd478d9af8049536ddd6fb P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:02.305781 32591 raft_consensus.cc:374] T d9761e1466cd478d9af8049536ddd6fb P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:02.305873 32591 raft_consensus.cc:729] T d9761e1466cd478d9af8049536ddd6fb P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:13:02.306063 32591 consensus_queue.cc:260] T d9761e1466cd478d9af8049536ddd6fb P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:02.306458 32591 ts_tablet_manager.cc:1398] T d9761e1466cd478d9af8049536ddd6fb P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 08:13:02.306845 32616 tablet_bootstrap.cc:492] T d9761e1466cd478d9af8049536ddd6fb P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:13:02.308605 32616 tablet_bootstrap.cc:654] T d9761e1466cd478d9af8049536ddd6fb P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:02.315263 32617 tablet_bootstrap.cc:492] T d9761e1466cd478d9af8049536ddd6fb P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:13:02.316036 32616 tablet_bootstrap.cc:492] T d9761e1466cd478d9af8049536ddd6fb P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:13:02.316144 32616 ts_tablet_manager.cc:1367] T d9761e1466cd478d9af8049536ddd6fb P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20240503 08:13:02.316668 32617 tablet_bootstrap.cc:654] T d9761e1466cd478d9af8049536ddd6fb P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:02.316875 32616 raft_consensus.cc:348] T d9761e1466cd478d9af8049536ddd6fb P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:02.317054 32616 raft_consensus.cc:374] T d9761e1466cd478d9af8049536ddd6fb P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:02.317142 32616 raft_consensus.cc:729] T d9761e1466cd478d9af8049536ddd6fb P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:13:02.317329 32616 consensus_queue.cc:260] T d9761e1466cd478d9af8049536ddd6fb P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:02.317987 32616 ts_tablet_manager.cc:1398] T d9761e1466cd478d9af8049536ddd6fb P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:13:02.319232 32617 tablet_bootstrap.cc:492] T d9761e1466cd478d9af8049536ddd6fb P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:13:02.319432 32617 ts_tablet_manager.cc:1367] T d9761e1466cd478d9af8049536ddd6fb P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:13:02.320132 32617 raft_consensus.cc:348] T d9761e1466cd478d9af8049536ddd6fb P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:02.320349 32617 raft_consensus.cc:374] T d9761e1466cd478d9af8049536ddd6fb P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:02.320508 32617 raft_consensus.cc:729] T d9761e1466cd478d9af8049536ddd6fb P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:13:02.320757 32617 consensus_queue.cc:260] T d9761e1466cd478d9af8049536ddd6fb P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:02.323525 32617 ts_tablet_manager.cc:1398] T d9761e1466cd478d9af8049536ddd6fb P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:13:02.343824 32452 consensus_queue.cc:1035] T 4ebf45bdc432451e87b7c98d435a76d2 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:02.350935 32452 consensus_queue.cc:1035] T 4ebf45bdc432451e87b7c98d435a76d2 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:02.376380 32493 consensus_queue.cc:1035] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:02.380290 32493 consensus_queue.cc:1035] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:02.448866 32566 consensus_queue.cc:1035] T b52e4b59153d40378e2a37809b04a2c8 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:02.455832 32631 consensus_queue.cc:1035] T b52e4b59153d40378e2a37809b04a2c8 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:02.481148 32535 raft_consensus.cc:482] T d9761e1466cd478d9af8049536ddd6fb P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:02.481318 32535 raft_consensus.cc:504] T d9761e1466cd478d9af8049536ddd6fb P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:02.481993 32535 leader_election.cc:288] T d9761e1466cd478d9af8049536ddd6fb P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:13:02.482383 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9761e1466cd478d9af8049536ddd6fb" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:13:02.482443 28393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9761e1466cd478d9af8049536ddd6fb" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:13:02.482614 28262 raft_consensus.cc:2454] T d9761e1466cd478d9af8049536ddd6fb P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 0.
I20240503 08:13:02.482679 28393 raft_consensus.cc:2454] T d9761e1466cd478d9af8049536ddd6fb P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 0.
I20240503 08:13:02.483048 30960 leader_election.cc:302] T d9761e1466cd478d9af8049536ddd6fb P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:13:02.483320 32535 raft_consensus.cc:2781] T d9761e1466cd478d9af8049536ddd6fb P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:02.483423 32535 raft_consensus.cc:482] T d9761e1466cd478d9af8049536ddd6fb P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:02.483520 32535 raft_consensus.cc:3037] T d9761e1466cd478d9af8049536ddd6fb P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:02.485729 32535 raft_consensus.cc:504] T d9761e1466cd478d9af8049536ddd6fb P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:02.486358 32535 leader_election.cc:288] T d9761e1466cd478d9af8049536ddd6fb P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:13:02.486644 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9761e1466cd478d9af8049536ddd6fb" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:13:02.486790 28393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9761e1466cd478d9af8049536ddd6fb" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:13:02.486845 28262 raft_consensus.cc:3037] T d9761e1466cd478d9af8049536ddd6fb P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:02.486991 28393 raft_consensus.cc:3037] T d9761e1466cd478d9af8049536ddd6fb P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:02.489506 28262 raft_consensus.cc:2454] T d9761e1466cd478d9af8049536ddd6fb P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 1.
I20240503 08:13:02.489637 28393 raft_consensus.cc:2454] T d9761e1466cd478d9af8049536ddd6fb P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 1.
I20240503 08:13:02.490100 30962 leader_election.cc:302] T d9761e1466cd478d9af8049536ddd6fb P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:13:02.491183 32535 raft_consensus.cc:2781] T d9761e1466cd478d9af8049536ddd6fb P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:02.491384 32535 raft_consensus.cc:686] T d9761e1466cd478d9af8049536ddd6fb P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 LEADER]: Becoming Leader. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Running, Role: LEADER
I20240503 08:13:02.491780 32535 consensus_queue.cc:237] T d9761e1466cd478d9af8049536ddd6fb P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:02.496100 27855 catalog_manager.cc:5484] T d9761e1466cd478d9af8049536ddd6fb P ced21ca983b749d0b4d4f6a0e2426bf8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71). New cstate: current_term: 1 leader_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:02.507356 27855 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_extra_table_50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:13:02.515606 27981 tablet_service.cc:1467] Processing CreateTablet for tablet de5dfccb0359462dbb7da776293fa679 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=c50031f182e6429695396372dc04b4e9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:02.515954 28111 tablet_service.cc:1467] Processing CreateTablet for tablet de5dfccb0359462dbb7da776293fa679 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=c50031f182e6429695396372dc04b4e9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:02.516055 27981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de5dfccb0359462dbb7da776293fa679. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:02.516168 28243 tablet_service.cc:1467] Processing CreateTablet for tablet de5dfccb0359462dbb7da776293fa679 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=c50031f182e6429695396372dc04b4e9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:02.516356 28111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de5dfccb0359462dbb7da776293fa679. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:02.516566 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de5dfccb0359462dbb7da776293fa679. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:02.527498 30873 tablet_service.cc:1467] Processing CreateTablet for tablet e8bc656988c44600a198850c604e0b64 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=c50031f182e6429695396372dc04b4e9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:02.527925 30873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8bc656988c44600a198850c604e0b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:02.530879 30741 tablet_service.cc:1467] Processing CreateTablet for tablet e8bc656988c44600a198850c604e0b64 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=c50031f182e6429695396372dc04b4e9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:02.531317 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8bc656988c44600a198850c604e0b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:02.534162 30742 tablet_service.cc:1467] Processing CreateTablet for tablet fa714cf4cc254703a2d356d2a536db8d (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=c50031f182e6429695396372dc04b4e9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:02.534330 32617 tablet_bootstrap.cc:492] T de5dfccb0359462dbb7da776293fa679 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:13:02.534520 30742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa714cf4cc254703a2d356d2a536db8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:02.535048 32634 tablet_bootstrap.cc:492] T de5dfccb0359462dbb7da776293fa679 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:13:02.535818 32617 tablet_bootstrap.cc:654] T de5dfccb0359462dbb7da776293fa679 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:02.537814 32634 tablet_bootstrap.cc:654] T de5dfccb0359462dbb7da776293fa679 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:02.540174 31141 tablet_service.cc:1467] Processing CreateTablet for tablet e8bc656988c44600a198850c604e0b64 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=c50031f182e6429695396372dc04b4e9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:02.540596 31141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8bc656988c44600a198850c604e0b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:02.546110 32617 tablet_bootstrap.cc:492] T de5dfccb0359462dbb7da776293fa679 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:13:02.546257 32617 ts_tablet_manager.cc:1367] T de5dfccb0359462dbb7da776293fa679 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20240503 08:13:02.547024 32617 raft_consensus.cc:348] T de5dfccb0359462dbb7da776293fa679 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:02.547227 32617 raft_consensus.cc:374] T de5dfccb0359462dbb7da776293fa679 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:02.547298 32617 raft_consensus.cc:729] T de5dfccb0359462dbb7da776293fa679 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:13:02.547523 32617 consensus_queue.cc:260] T de5dfccb0359462dbb7da776293fa679 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:02.548106 32617 ts_tablet_manager.cc:1398] T de5dfccb0359462dbb7da776293fa679 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:13:02.550258 32634 tablet_bootstrap.cc:492] T de5dfccb0359462dbb7da776293fa679 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:13:02.550491 32634 ts_tablet_manager.cc:1367] T de5dfccb0359462dbb7da776293fa679 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.008s
I20240503 08:13:02.551328 32634 raft_consensus.cc:348] T de5dfccb0359462dbb7da776293fa679 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:02.552618 32634 raft_consensus.cc:374] T de5dfccb0359462dbb7da776293fa679 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:02.552771 32634 raft_consensus.cc:729] T de5dfccb0359462dbb7da776293fa679 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:13:02.553066 32634 consensus_queue.cc:260] T de5dfccb0359462dbb7da776293fa679 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:02.555243 32634 ts_tablet_manager.cc:1398] T de5dfccb0359462dbb7da776293fa679 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 08:13:02.556496 28111 tablet_service.cc:1467] Processing CreateTablet for tablet fa714cf4cc254703a2d356d2a536db8d (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=c50031f182e6429695396372dc04b4e9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:02.556978 28111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa714cf4cc254703a2d356d2a536db8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:02.561043 27983 tablet_service.cc:1467] Processing CreateTablet for tablet fa714cf4cc254703a2d356d2a536db8d (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=c50031f182e6429695396372dc04b4e9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:02.561475 27983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa714cf4cc254703a2d356d2a536db8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:02.568152 32639 tablet_bootstrap.cc:492] T e8bc656988c44600a198850c604e0b64 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:13:02.570078 32639 tablet_bootstrap.cc:654] T e8bc656988c44600a198850c604e0b64 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:02.570324 32640 tablet_bootstrap.cc:492] T e8bc656988c44600a198850c604e0b64 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:13:02.572940 32640 tablet_bootstrap.cc:654] T e8bc656988c44600a198850c604e0b64 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:02.578691 32634 tablet_bootstrap.cc:492] T fa714cf4cc254703a2d356d2a536db8d P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:13:02.580192 32634 tablet_bootstrap.cc:654] T fa714cf4cc254703a2d356d2a536db8d P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:02.584275 32638 tablet_bootstrap.cc:492] T e8bc656988c44600a198850c604e0b64 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:13:02.586091 32638 tablet_bootstrap.cc:654] T e8bc656988c44600a198850c604e0b64 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:02.595669 32635 tablet_bootstrap.cc:492] T de5dfccb0359462dbb7da776293fa679 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:13:02.597559 32635 tablet_bootstrap.cc:654] T de5dfccb0359462dbb7da776293fa679 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:02.598385 32634 tablet_bootstrap.cc:492] T fa714cf4cc254703a2d356d2a536db8d P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:13:02.598568 32634 ts_tablet_manager.cc:1367] T fa714cf4cc254703a2d356d2a536db8d P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.006s
I20240503 08:13:02.599319 32634 raft_consensus.cc:348] T fa714cf4cc254703a2d356d2a536db8d P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:02.599542 32634 raft_consensus.cc:374] T fa714cf4cc254703a2d356d2a536db8d P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:02.599709 32634 raft_consensus.cc:729] T fa714cf4cc254703a2d356d2a536db8d P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:13:02.600011 32634 consensus_queue.cc:260] T fa714cf4cc254703a2d356d2a536db8d P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:02.600519 32638 tablet_bootstrap.cc:492] T e8bc656988c44600a198850c604e0b64 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:13:02.600698 32638 ts_tablet_manager.cc:1367] T e8bc656988c44600a198850c604e0b64 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.007s
I20240503 08:13:02.601450 32638 raft_consensus.cc:348] T e8bc656988c44600a198850c604e0b64 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:02.601692 32638 raft_consensus.cc:374] T e8bc656988c44600a198850c604e0b64 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:02.601824 32638 raft_consensus.cc:729] T e8bc656988c44600a198850c604e0b64 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:13:02.602080 32638 consensus_queue.cc:260] T e8bc656988c44600a198850c604e0b64 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:02.603955 32638 ts_tablet_manager.cc:1398] T e8bc656988c44600a198850c604e0b64 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 08:13:02.604595 32631 raft_consensus.cc:482] T fa714cf4cc254703a2d356d2a536db8d P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:02.604753 32631 raft_consensus.cc:504] T fa714cf4cc254703a2d356d2a536db8d P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:02.605559 32631 leader_election.cc:288] T fa714cf4cc254703a2d356d2a536db8d P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:13:02.606134 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa714cf4cc254703a2d356d2a536db8d" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
W20240503 08:13:02.606784 28069 leader_election.cc:341] T fa714cf4cc254703a2d356d2a536db8d P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:13:02.607146 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa714cf4cc254703a2d356d2a536db8d" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:13:02.608690 32634 ts_tablet_manager.cc:1398] T fa714cf4cc254703a2d356d2a536db8d P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
W20240503 08:13:02.608844 28068 leader_election.cc:341] T fa714cf4cc254703a2d356d2a536db8d P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:13:02.609010 28068 leader_election.cc:302] T fa714cf4cc254703a2d356d2a536db8d P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b411c6bb25549368041773d7b710d48; no voters: 6d1d7de19459435fbde1999e3dfd226b, 833582ff68c94269be364ebd187a75db
I20240503 08:13:02.610707 32631 raft_consensus.cc:2726] T fa714cf4cc254703a2d356d2a536db8d P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:13:02.611479 32639 tablet_bootstrap.cc:492] T e8bc656988c44600a198850c604e0b64 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:13:02.611673 32639 ts_tablet_manager.cc:1367] T e8bc656988c44600a198850c604e0b64 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.009s
I20240503 08:13:02.612656 32639 raft_consensus.cc:348] T e8bc656988c44600a198850c604e0b64 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:02.612869 32639 raft_consensus.cc:374] T e8bc656988c44600a198850c604e0b64 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:02.612985 32639 raft_consensus.cc:729] T e8bc656988c44600a198850c604e0b64 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:13:02.613269 32639 consensus_queue.cc:260] T e8bc656988c44600a198850c604e0b64 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:02.613721 32635 tablet_bootstrap.cc:492] T de5dfccb0359462dbb7da776293fa679 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:13:02.613873 32635 ts_tablet_manager.cc:1367] T de5dfccb0359462dbb7da776293fa679 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.003s
I20240503 08:13:02.614544 32635 raft_consensus.cc:348] T de5dfccb0359462dbb7da776293fa679 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:02.614686 32635 raft_consensus.cc:374] T de5dfccb0359462dbb7da776293fa679 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:02.614739 32635 raft_consensus.cc:729] T de5dfccb0359462dbb7da776293fa679 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:13:02.614908 32635 consensus_queue.cc:260] T de5dfccb0359462dbb7da776293fa679 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:02.615371 32635 ts_tablet_manager.cc:1398] T de5dfccb0359462dbb7da776293fa679 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:13:02.615629 32635 tablet_bootstrap.cc:492] T fa714cf4cc254703a2d356d2a536db8d P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:13:02.617879 32635 tablet_bootstrap.cc:654] T fa714cf4cc254703a2d356d2a536db8d P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:02.611614 32640 tablet_bootstrap.cc:492] T e8bc656988c44600a198850c604e0b64 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:13:02.630014 32640 ts_tablet_manager.cc:1367] T e8bc656988c44600a198850c604e0b64 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.060s user 0.005s sys 0.010s
I20240503 08:13:02.630800 32640 raft_consensus.cc:348] T e8bc656988c44600a198850c604e0b64 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:02.631100 32640 raft_consensus.cc:374] T e8bc656988c44600a198850c604e0b64 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:02.631194 32640 raft_consensus.cc:729] T e8bc656988c44600a198850c604e0b64 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:13:02.631470 32640 consensus_queue.cc:260] T e8bc656988c44600a198850c604e0b64 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:02.632203 32640 ts_tablet_manager.cc:1398] T e8bc656988c44600a198850c604e0b64 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:13:02.635015 32639 ts_tablet_manager.cc:1398] T e8bc656988c44600a198850c604e0b64 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.023s user 0.000s sys 0.001s
I20240503 08:13:02.644224 32639 tablet_bootstrap.cc:492] T fa714cf4cc254703a2d356d2a536db8d P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:13:02.646142 32639 tablet_bootstrap.cc:654] T fa714cf4cc254703a2d356d2a536db8d P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:02.656787 32639 tablet_bootstrap.cc:492] T fa714cf4cc254703a2d356d2a536db8d P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:13:02.656970 32639 ts_tablet_manager.cc:1367] T fa714cf4cc254703a2d356d2a536db8d P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.001s
I20240503 08:13:02.658097 32639 raft_consensus.cc:348] T fa714cf4cc254703a2d356d2a536db8d P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:02.658282 32639 raft_consensus.cc:374] T fa714cf4cc254703a2d356d2a536db8d P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:02.658367 32639 raft_consensus.cc:729] T fa714cf4cc254703a2d356d2a536db8d P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:13:02.658638 32639 consensus_queue.cc:260] T fa714cf4cc254703a2d356d2a536db8d P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:02.659279 32639 ts_tablet_manager.cc:1398] T fa714cf4cc254703a2d356d2a536db8d P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 08:13:02.660303 32635 tablet_bootstrap.cc:492] T fa714cf4cc254703a2d356d2a536db8d P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:13:02.660449 32635 ts_tablet_manager.cc:1367] T fa714cf4cc254703a2d356d2a536db8d P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.045s user 0.016s sys 0.017s
I20240503 08:13:02.661181 32635 raft_consensus.cc:348] T fa714cf4cc254703a2d356d2a536db8d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:02.661394 32635 raft_consensus.cc:374] T fa714cf4cc254703a2d356d2a536db8d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:02.661459 32635 raft_consensus.cc:729] T fa714cf4cc254703a2d356d2a536db8d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:13:02.661712 32635 consensus_queue.cc:260] T fa714cf4cc254703a2d356d2a536db8d P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:02.662436 32635 ts_tablet_manager.cc:1398] T fa714cf4cc254703a2d356d2a536db8d P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:02.667480 27854 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:13:02.674403 28392 raft_consensus.cc:1261] T d9761e1466cd478d9af8049536ddd6fb P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer ced21ca983b749d0b4d4f6a0e2426bf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:02.674757 28262 raft_consensus.cc:1261] T d9761e1466cd478d9af8049536ddd6fb P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer ced21ca983b749d0b4d4f6a0e2426bf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:02.675302 32535 consensus_queue.cc:1035] T d9761e1466cd478d9af8049536ddd6fb P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:02.675510 32456 consensus_queue.cc:1035] T d9761e1466cd478d9af8049536ddd6fb P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:02.684985 27854 catalog_manager.cc:5576] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211) does not have the latest schema for tablet d9761e1466cd478d9af8049536ddd6fb (table rebalancer_dynamic_table [id=6e2e9373551b4f30a1a4081f42994787]). Expected version 1 got 0
I20240503 08:13:02.684988 32651 tablet.cc:1722] T d9761e1466cd478d9af8049536ddd6fb P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:13:02.687168 32651 tablet.cc:1620] T d9761e1466cd478d9af8049536ddd6fb P ced21ca983b749d0b4d4f6a0e2426bf8: MemRowSet was empty: no flush needed.
I20240503 08:13:02.688452 32653 tablet.cc:1722] T d9761e1466cd478d9af8049536ddd6fb P aae860d7ae2b4002bf7794c278f9cc56: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:13:02.689160 32653 tablet.cc:1620] T d9761e1466cd478d9af8049536ddd6fb P aae860d7ae2b4002bf7794c278f9cc56: MemRowSet was empty: no flush needed.
I20240503 08:13:02.689985 27854 catalog_manager.cc:5576] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919) does not have the latest schema for tablet d9761e1466cd478d9af8049536ddd6fb (table rebalancer_dynamic_table [id=6e2e9373551b4f30a1a4081f42994787]). Expected version 1 got 0
I20240503 08:13:02.695142 32652 tablet.cc:1722] T d9761e1466cd478d9af8049536ddd6fb P d09a11a03989421b9f28019dc1c14cc2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:13:02.695729 32652 tablet.cc:1620] T d9761e1466cd478d9af8049536ddd6fb P d09a11a03989421b9f28019dc1c14cc2: MemRowSet was empty: no flush needed.
I20240503 08:13:02.697520 27855 catalog_manager.cc:5576] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211) does not have the latest schema for tablet d9761e1466cd478d9af8049536ddd6fb (table rebalancer_dynamic_table [id=6e2e9373551b4f30a1a4081f42994787]). Expected version 1 got 0
I20240503 08:13:02.709964 27854 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=6e2e9373551b4f30a1a4081f42994787] alter complete (version 1)
I20240503 08:13:02.754330 32560 raft_consensus.cc:482] T de5dfccb0359462dbb7da776293fa679 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:02.754505 32560 raft_consensus.cc:504] T de5dfccb0359462dbb7da776293fa679 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:02.755256 32560 leader_election.cc:288] T de5dfccb0359462dbb7da776293fa679 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:13:02.755649 28131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de5dfccb0359462dbb7da776293fa679" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:13:02.755707 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de5dfccb0359462dbb7da776293fa679" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:13:02.755877 28131 raft_consensus.cc:2454] T de5dfccb0359462dbb7da776293fa679 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:13:02.755918 28002 raft_consensus.cc:2454] T de5dfccb0359462dbb7da776293fa679 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:13:02.756275 28198 leader_election.cc:302] T de5dfccb0359462dbb7da776293fa679 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:13:02.756541 32560 raft_consensus.cc:2781] T de5dfccb0359462dbb7da776293fa679 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:02.756660 32560 raft_consensus.cc:482] T de5dfccb0359462dbb7da776293fa679 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:02.756744 32560 raft_consensus.cc:3037] T de5dfccb0359462dbb7da776293fa679 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:02.759305 32560 raft_consensus.cc:504] T de5dfccb0359462dbb7da776293fa679 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:02.759940 32560 leader_election.cc:288] T de5dfccb0359462dbb7da776293fa679 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Requested vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:13:02.760401 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de5dfccb0359462dbb7da776293fa679" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:13:02.760553 28002 raft_consensus.cc:3037] T de5dfccb0359462dbb7da776293fa679 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:02.760679 28131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de5dfccb0359462dbb7da776293fa679" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:13:02.760854 28131 raft_consensus.cc:3037] T de5dfccb0359462dbb7da776293fa679 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:02.762439 28002 raft_consensus.cc:2454] T de5dfccb0359462dbb7da776293fa679 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:13:02.762776 28131 raft_consensus.cc:2454] T de5dfccb0359462dbb7da776293fa679 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:13:02.763108 28198 leader_election.cc:302] T de5dfccb0359462dbb7da776293fa679 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:13:02.763401 32560 raft_consensus.cc:2781] T de5dfccb0359462dbb7da776293fa679 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:02.763536 32560 raft_consensus.cc:686] T de5dfccb0359462dbb7da776293fa679 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Becoming Leader. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Running, Role: LEADER
I20240503 08:13:02.763769 32560 consensus_queue.cc:237] T de5dfccb0359462dbb7da776293fa679 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:02.767728 27856 catalog_manager.cc:5484] T de5dfccb0359462dbb7da776293fa679 P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: term changed from 0 to 1, leader changed from <none> to aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67). New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:02.814538 27856 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:13:02.821826 32651 tablet.cc:1722] T d9761e1466cd478d9af8049536ddd6fb P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:13:02.822273 32651 tablet.cc:1620] T d9761e1466cd478d9af8049536ddd6fb P ced21ca983b749d0b4d4f6a0e2426bf8: MemRowSet was empty: no flush needed.
I20240503 08:13:02.822916 32653 tablet.cc:1722] T d9761e1466cd478d9af8049536ddd6fb P aae860d7ae2b4002bf7794c278f9cc56: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:13:02.823359 32653 tablet.cc:1620] T d9761e1466cd478d9af8049536ddd6fb P aae860d7ae2b4002bf7794c278f9cc56: MemRowSet was empty: no flush needed.
I20240503 08:13:02.826639 32652 tablet.cc:1722] T d9761e1466cd478d9af8049536ddd6fb P d09a11a03989421b9f28019dc1c14cc2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:13:02.827060 32652 tablet.cc:1620] T d9761e1466cd478d9af8049536ddd6fb P d09a11a03989421b9f28019dc1c14cc2: MemRowSet was empty: no flush needed.
I20240503 08:13:02.831053 27855 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=6e2e9373551b4f30a1a4081f42994787] alter complete (version 2)
I20240503 08:13:02.841265 32534 raft_consensus.cc:482] T e8bc656988c44600a198850c604e0b64 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:02.841423 32534 raft_consensus.cc:504] T e8bc656988c44600a198850c604e0b64 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:02.842170 32534 leader_election.cc:288] T e8bc656988c44600a198850c604e0b64 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:13:02.842787 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8bc656988c44600a198850c604e0b64" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:13:02.842878 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8bc656988c44600a198850c604e0b64" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:13:02.843026 30895 raft_consensus.cc:2454] T e8bc656988c44600a198850c604e0b64 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 0.
I20240503 08:13:02.843086 30762 raft_consensus.cc:2454] T e8bc656988c44600a198850c604e0b64 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 0.
I20240503 08:13:02.843463 31094 leader_election.cc:302] T e8bc656988c44600a198850c604e0b64 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:13:02.843752 32534 raft_consensus.cc:2781] T e8bc656988c44600a198850c604e0b64 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:02.843843 32534 raft_consensus.cc:482] T e8bc656988c44600a198850c604e0b64 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:02.843956 32534 raft_consensus.cc:3037] T e8bc656988c44600a198850c604e0b64 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:02.845708 32534 raft_consensus.cc:504] T e8bc656988c44600a198850c604e0b64 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:02.846298 32534 leader_election.cc:288] T e8bc656988c44600a198850c604e0b64 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 election: Requested vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:13:02.846554 30894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8bc656988c44600a198850c604e0b64" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:13:02.846746 30894 raft_consensus.cc:3037] T e8bc656988c44600a198850c604e0b64 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:02.846724 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8bc656988c44600a198850c604e0b64" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:13:02.846889 30762 raft_consensus.cc:3037] T e8bc656988c44600a198850c604e0b64 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:02.848699 30894 raft_consensus.cc:2454] T e8bc656988c44600a198850c604e0b64 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 1.
I20240503 08:13:02.848785 30762 raft_consensus.cc:2454] T e8bc656988c44600a198850c604e0b64 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 1.
I20240503 08:13:02.849082 31094 leader_election.cc:302] T e8bc656988c44600a198850c604e0b64 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:13:02.849335 32534 raft_consensus.cc:2781] T e8bc656988c44600a198850c604e0b64 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:02.849507 32534 raft_consensus.cc:686] T e8bc656988c44600a198850c604e0b64 P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Becoming Leader. State: Replica: b00f244c69df401997ea99d492218a98, State: Running, Role: LEADER
I20240503 08:13:02.849740 32534 consensus_queue.cc:237] T e8bc656988c44600a198850c604e0b64 P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:02.852847 27856 catalog_manager.cc:5484] T e8bc656988c44600a198850c604e0b64 P b00f244c69df401997ea99d492218a98 reported cstate change: term changed from 0 to 1, leader changed from <none> to b00f244c69df401997ea99d492218a98 (127.20.230.72). New cstate: current_term: 1 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:13:02.885924 32496 raft_consensus.cc:482] T fa714cf4cc254703a2d356d2a536db8d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:02.886152 32496 raft_consensus.cc:504] T fa714cf4cc254703a2d356d2a536db8d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:02.886996 32496 leader_election.cc:288] T fa714cf4cc254703a2d356d2a536db8d P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:13:02.887436 28131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa714cf4cc254703a2d356d2a536db8d" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:13:02.887526 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa714cf4cc254703a2d356d2a536db8d" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:13:02.887696 28131 raft_consensus.cc:2454] T fa714cf4cc254703a2d356d2a536db8d P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:13:02.887742 30762 raft_consensus.cc:2454] T fa714cf4cc254703a2d356d2a536db8d P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:13:02.888098 27938 leader_election.cc:302] T fa714cf4cc254703a2d356d2a536db8d P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, 833582ff68c94269be364ebd187a75db; no voters:
I20240503 08:13:02.888362 32496 raft_consensus.cc:2781] T fa714cf4cc254703a2d356d2a536db8d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:02.888471 32496 raft_consensus.cc:482] T fa714cf4cc254703a2d356d2a536db8d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:02.888576 32496 raft_consensus.cc:3037] T fa714cf4cc254703a2d356d2a536db8d P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:02.890774 32496 raft_consensus.cc:504] T fa714cf4cc254703a2d356d2a536db8d P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:02.891713 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa714cf4cc254703a2d356d2a536db8d" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:13:02.891860 32496 leader_election.cc:288] T fa714cf4cc254703a2d356d2a536db8d P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Requested vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:13:02.891808 28131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa714cf4cc254703a2d356d2a536db8d" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:13:02.891932 30762 raft_consensus.cc:3037] T fa714cf4cc254703a2d356d2a536db8d P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:02.892040 28131 raft_consensus.cc:3037] T fa714cf4cc254703a2d356d2a536db8d P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:02.894752 30762 raft_consensus.cc:2454] T fa714cf4cc254703a2d356d2a536db8d P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:13:02.894784 28131 raft_consensus.cc:2454] T fa714cf4cc254703a2d356d2a536db8d P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:13:02.895125 27938 leader_election.cc:302] T fa714cf4cc254703a2d356d2a536db8d P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 833582ff68c94269be364ebd187a75db; no voters:
I20240503 08:13:02.895395 32496 raft_consensus.cc:2781] T fa714cf4cc254703a2d356d2a536db8d P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:02.895557 32496 raft_consensus.cc:686] T fa714cf4cc254703a2d356d2a536db8d P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Becoming Leader. State: Replica: 833582ff68c94269be364ebd187a75db, State: Running, Role: LEADER
I20240503 08:13:02.895823 32496 consensus_queue.cc:237] T fa714cf4cc254703a2d356d2a536db8d P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:02.899354 27856 catalog_manager.cc:5484] T fa714cf4cc254703a2d356d2a536db8d P 833582ff68c94269be364ebd187a75db reported cstate change: term changed from 0 to 1, leader changed from <none> to 833582ff68c94269be364ebd187a75db (127.20.230.65). New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:02.935514 27856 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:13:02.942018 32651 tablet.cc:1722] T d9761e1466cd478d9af8049536ddd6fb P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:13:02.942859 32652 tablet.cc:1722] T d9761e1466cd478d9af8049536ddd6fb P d09a11a03989421b9f28019dc1c14cc2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:13:02.943499 32653 tablet.cc:1722] T d9761e1466cd478d9af8049536ddd6fb P aae860d7ae2b4002bf7794c278f9cc56: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:13:02.951205 27855 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table_new_name [id=6e2e9373551b4f30a1a4081f42994787] alter complete (version 3)
I20240503 08:13:03.054955 27862 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:13:03.055248 27862 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:13:03.058482 27862 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet d9761e1466cd478d9af8049536ddd6fb
I20240503 08:13:03.059965 31006 tablet_service.cc:1514] Processing DeleteTablet for tablet d9761e1466cd478d9af8049536ddd6fb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:03 UTC) from {username='slave'} at 127.0.0.1:47020
I20240503 08:13:03.060143 28372 tablet_service.cc:1514] Processing DeleteTablet for tablet d9761e1466cd478d9af8049536ddd6fb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:03 UTC) from {username='slave'} at 127.0.0.1:34786
I20240503 08:13:03.060530 28243 tablet_service.cc:1514] Processing DeleteTablet for tablet d9761e1466cd478d9af8049536ddd6fb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:03 UTC) from {username='slave'} at 127.0.0.1:41960
I20240503 08:13:03.071422 32658 tablet_replica.cc:330] T d9761e1466cd478d9af8049536ddd6fb P d09a11a03989421b9f28019dc1c14cc2: stopping tablet replica
I20240503 08:13:03.071717 32658 raft_consensus.cc:2229] T d9761e1466cd478d9af8049536ddd6fb P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:03.071996 32658 raft_consensus.cc:2258] T d9761e1466cd478d9af8049536ddd6fb P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:03.073103 32658 ts_tablet_manager.cc:1875] T d9761e1466cd478d9af8049536ddd6fb P d09a11a03989421b9f28019dc1c14cc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:03.073647 32657 tablet_replica.cc:330] T d9761e1466cd478d9af8049536ddd6fb P ced21ca983b749d0b4d4f6a0e2426bf8: stopping tablet replica
I20240503 08:13:03.073958 32657 raft_consensus.cc:2229] T d9761e1466cd478d9af8049536ddd6fb P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:13:03.074313 32657 raft_consensus.cc:2258] T d9761e1466cd478d9af8049536ddd6fb P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:03.075184 32657 ts_tablet_manager.cc:1875] T d9761e1466cd478d9af8049536ddd6fb P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:03.080356 32658 ts_tablet_manager.cc:1888] T d9761e1466cd478d9af8049536ddd6fb P d09a11a03989421b9f28019dc1c14cc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:13:03.080511 32658 log.cc:1196] T d9761e1466cd478d9af8049536ddd6fb P d09a11a03989421b9f28019dc1c14cc2: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/wals/d9761e1466cd478d9af8049536ddd6fb
I20240503 08:13:03.080971 32658 ts_tablet_manager.cc:1909] T d9761e1466cd478d9af8049536ddd6fb P d09a11a03989421b9f28019dc1c14cc2: Deleting consensus metadata
I20240503 08:13:03.081887 32659 tablet_replica.cc:330] T d9761e1466cd478d9af8049536ddd6fb P aae860d7ae2b4002bf7794c278f9cc56: stopping tablet replica
I20240503 08:13:03.081964 27849 catalog_manager.cc:4830] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): tablet d9761e1466cd478d9af8049536ddd6fb (table rebalancer_dynamic_table_new_name [id=6e2e9373551b4f30a1a4081f42994787]) successfully deleted
I20240503 08:13:03.082680 32659 raft_consensus.cc:2229] T d9761e1466cd478d9af8049536ddd6fb P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:03.082944 32659 raft_consensus.cc:2258] T d9761e1466cd478d9af8049536ddd6fb P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:03.083595 32659 ts_tablet_manager.cc:1875] T d9761e1466cd478d9af8049536ddd6fb P aae860d7ae2b4002bf7794c278f9cc56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:03.084167 32657 ts_tablet_manager.cc:1888] T d9761e1466cd478d9af8049536ddd6fb P ced21ca983b749d0b4d4f6a0e2426bf8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:13:03.084275 32657 log.cc:1196] T d9761e1466cd478d9af8049536ddd6fb P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal/wals/d9761e1466cd478d9af8049536ddd6fb
I20240503 08:13:03.084651 32657 ts_tablet_manager.cc:1909] T d9761e1466cd478d9af8049536ddd6fb P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting consensus metadata
I20240503 08:13:03.085561 27847 catalog_manager.cc:4830] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): tablet d9761e1466cd478d9af8049536ddd6fb (table rebalancer_dynamic_table_new_name [id=6e2e9373551b4f30a1a4081f42994787]) successfully deleted
I20240503 08:13:03.088168 32659 ts_tablet_manager.cc:1888] T d9761e1466cd478d9af8049536ddd6fb P aae860d7ae2b4002bf7794c278f9cc56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:13:03.088276 32659 log.cc:1196] T d9761e1466cd478d9af8049536ddd6fb P aae860d7ae2b4002bf7794c278f9cc56: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal/wals/d9761e1466cd478d9af8049536ddd6fb
I20240503 08:13:03.088655 32659 ts_tablet_manager.cc:1909] T d9761e1466cd478d9af8049536ddd6fb P aae860d7ae2b4002bf7794c278f9cc56: Deleting consensus metadata
I20240503 08:13:03.089651 27847 catalog_manager.cc:4830] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): tablet d9761e1466cd478d9af8049536ddd6fb (table rebalancer_dynamic_table_new_name [id=6e2e9373551b4f30a1a4081f42994787]) successfully deleted
I20240503 08:13:03.156951 32560 consensus_queue.cc:1035] T de5dfccb0359462dbb7da776293fa679 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:03.161540 32560 consensus_queue.cc:1035] T de5dfccb0359462dbb7da776293fa679 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:03.162032 27856 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:13:03.174590 27981 tablet_service.cc:1467] Processing CreateTablet for tablet 4a31fe0fbee142bdb90f4e8c7a66c8de (DEFAULT_TABLE table=rebalancer_dynamic_table [id=924c17d4e8a645a2b093b33e05af5507]), partition=
I20240503 08:13:03.174991 27981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a31fe0fbee142bdb90f4e8c7a66c8de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:03.175427 30873 tablet_service.cc:1467] Processing CreateTablet for tablet 4a31fe0fbee142bdb90f4e8c7a66c8de (DEFAULT_TABLE table=rebalancer_dynamic_table [id=924c17d4e8a645a2b093b33e05af5507]), partition=
I20240503 08:13:03.175779 30873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a31fe0fbee142bdb90f4e8c7a66c8de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:03.181447 28243 tablet_service.cc:1467] Processing CreateTablet for tablet 4a31fe0fbee142bdb90f4e8c7a66c8de (DEFAULT_TABLE table=rebalancer_dynamic_table [id=924c17d4e8a645a2b093b33e05af5507]), partition=
I20240503 08:13:03.181830 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a31fe0fbee142bdb90f4e8c7a66c8de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:03.191932 32667 tablet_bootstrap.cc:492] T 4a31fe0fbee142bdb90f4e8c7a66c8de P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:13:03.193957 32667 tablet_bootstrap.cc:654] T 4a31fe0fbee142bdb90f4e8c7a66c8de P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:03.198618 32666 tablet_bootstrap.cc:492] T 4a31fe0fbee142bdb90f4e8c7a66c8de P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:13:03.200088 32666 tablet_bootstrap.cc:654] T 4a31fe0fbee142bdb90f4e8c7a66c8de P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:03.201496 32669 tablet_bootstrap.cc:492] T 4a31fe0fbee142bdb90f4e8c7a66c8de P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:13:03.203019 32669 tablet_bootstrap.cc:654] T 4a31fe0fbee142bdb90f4e8c7a66c8de P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:03.205669 32666 tablet_bootstrap.cc:492] T 4a31fe0fbee142bdb90f4e8c7a66c8de P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:13:03.206604 32669 tablet_bootstrap.cc:492] T 4a31fe0fbee142bdb90f4e8c7a66c8de P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:13:03.206604 32666 ts_tablet_manager.cc:1367] T 4a31fe0fbee142bdb90f4e8c7a66c8de P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240503 08:13:03.206800 32669 ts_tablet_manager.cc:1367] T 4a31fe0fbee142bdb90f4e8c7a66c8de P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 08:13:03.207492 32669 raft_consensus.cc:348] T 4a31fe0fbee142bdb90f4e8c7a66c8de P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:03.207505 32666 raft_consensus.cc:348] T 4a31fe0fbee142bdb90f4e8c7a66c8de P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:03.207717 32666 raft_consensus.cc:374] T 4a31fe0fbee142bdb90f4e8c7a66c8de P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:03.207804 32666 raft_consensus.cc:729] T 4a31fe0fbee142bdb90f4e8c7a66c8de P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:13:03.207824 32669 raft_consensus.cc:374] T 4a31fe0fbee142bdb90f4e8c7a66c8de P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:03.208055 32666 consensus_queue.cc:260] T 4a31fe0fbee142bdb90f4e8c7a66c8de P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:03.208173 32669 raft_consensus.cc:729] T 4a31fe0fbee142bdb90f4e8c7a66c8de P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:13:03.208487 32669 consensus_queue.cc:260] T 4a31fe0fbee142bdb90f4e8c7a66c8de P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:03.208595 32666 ts_tablet_manager.cc:1398] T 4a31fe0fbee142bdb90f4e8c7a66c8de P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:13:03.209056 32669 ts_tablet_manager.cc:1398] T 4a31fe0fbee142bdb90f4e8c7a66c8de P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:03.239188 32667 tablet_bootstrap.cc:492] T 4a31fe0fbee142bdb90f4e8c7a66c8de P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:13:03.239356 32667 ts_tablet_manager.cc:1367] T 4a31fe0fbee142bdb90f4e8c7a66c8de P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.048s user 0.000s sys 0.006s
I20240503 08:13:03.240016 32667 raft_consensus.cc:348] T 4a31fe0fbee142bdb90f4e8c7a66c8de P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:03.240180 32667 raft_consensus.cc:374] T 4a31fe0fbee142bdb90f4e8c7a66c8de P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:03.240235 32667 raft_consensus.cc:729] T 4a31fe0fbee142bdb90f4e8c7a66c8de P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:13:03.240463 32667 consensus_queue.cc:260] T 4a31fe0fbee142bdb90f4e8c7a66c8de P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:03.240908 32667 ts_tablet_manager.cc:1398] T 4a31fe0fbee142bdb90f4e8c7a66c8de P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:13:03.272735 32534 consensus_queue.cc:1035] T e8bc656988c44600a198850c604e0b64 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:03.276825 32534 consensus_queue.cc:1035] T e8bc656988c44600a198850c604e0b64 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:03.357791 32496 consensus_queue.cc:1035] T fa714cf4cc254703a2d356d2a536db8d P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:03.361326 32611 consensus_queue.cc:1035] T fa714cf4cc254703a2d356d2a536db8d P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:03.397809 32448 raft_consensus.cc:482] T 4a31fe0fbee142bdb90f4e8c7a66c8de P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:03.397997 32448 raft_consensus.cc:504] T 4a31fe0fbee142bdb90f4e8c7a66c8de P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:03.398630 32448 leader_election.cc:288] T 4a31fe0fbee142bdb90f4e8c7a66c8de P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:13:03.399086 28263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a31fe0fbee142bdb90f4e8c7a66c8de" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:13:03.399072 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a31fe0fbee142bdb90f4e8c7a66c8de" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:13:03.399307 28263 raft_consensus.cc:2454] T 4a31fe0fbee142bdb90f4e8c7a66c8de P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:13:03.399327 28003 raft_consensus.cc:2454] T 4a31fe0fbee142bdb90f4e8c7a66c8de P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:13:03.399690 30830 leader_election.cc:302] T 4a31fe0fbee142bdb90f4e8c7a66c8de P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:13:03.399957 32448 raft_consensus.cc:2781] T 4a31fe0fbee142bdb90f4e8c7a66c8de P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:03.400030 32448 raft_consensus.cc:482] T 4a31fe0fbee142bdb90f4e8c7a66c8de P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:03.400099 32448 raft_consensus.cc:3037] T 4a31fe0fbee142bdb90f4e8c7a66c8de P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:03.402228 32448 raft_consensus.cc:504] T 4a31fe0fbee142bdb90f4e8c7a66c8de P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:03.402726 32448 leader_election.cc:288] T 4a31fe0fbee142bdb90f4e8c7a66c8de P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Requested vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:13:03.403052 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a31fe0fbee142bdb90f4e8c7a66c8de" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:13:03.403227 28003 raft_consensus.cc:3037] T 4a31fe0fbee142bdb90f4e8c7a66c8de P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:03.405115 28003 raft_consensus.cc:2454] T 4a31fe0fbee142bdb90f4e8c7a66c8de P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:13:03.405409 28263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a31fe0fbee142bdb90f4e8c7a66c8de" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:13:03.405462 30830 leader_election.cc:302] T 4a31fe0fbee142bdb90f4e8c7a66c8de P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:13:03.405575 28263 raft_consensus.cc:3037] T 4a31fe0fbee142bdb90f4e8c7a66c8de P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:03.405673 32448 raft_consensus.cc:2781] T 4a31fe0fbee142bdb90f4e8c7a66c8de P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:03.405841 32448 raft_consensus.cc:686] T 4a31fe0fbee142bdb90f4e8c7a66c8de P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Becoming Leader. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Running, Role: LEADER
I20240503 08:13:03.406229 32448 consensus_queue.cc:237] T 4a31fe0fbee142bdb90f4e8c7a66c8de P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:03.407554 28263 raft_consensus.cc:2454] T 4a31fe0fbee142bdb90f4e8c7a66c8de P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:13:03.409610 27856 catalog_manager.cc:5484] T 4a31fe0fbee142bdb90f4e8c7a66c8de P ce1496a47d084365ad786cd09e472a2c reported cstate change: term changed from 0 to 1, leader changed from <none> to ce1496a47d084365ad786cd09e472a2c (127.20.230.70). New cstate: current_term: 1 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:03.443404 27856 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_extra_table_52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:13:03.451438 31141 tablet_service.cc:1467] Processing CreateTablet for tablet 644bb089a4064ecfb88438d8b859bf01 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=40dc8c7f01804911a18e5c62a4b6be48]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:03.451668 28372 tablet_service.cc:1467] Processing CreateTablet for tablet 644bb089a4064ecfb88438d8b859bf01 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=40dc8c7f01804911a18e5c62a4b6be48]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:03.451936 31141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 644bb089a4064ecfb88438d8b859bf01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:03.451951 31006 tablet_service.cc:1467] Processing CreateTablet for tablet 644bb089a4064ecfb88438d8b859bf01 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=40dc8c7f01804911a18e5c62a4b6be48]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:03.452054 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 644bb089a4064ecfb88438d8b859bf01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:03.452350 31006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 644bb089a4064ecfb88438d8b859bf01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:03.458647 30873 tablet_service.cc:1467] Processing CreateTablet for tablet d8c0fcef461446eabc1e55f434a2ce9f (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=40dc8c7f01804911a18e5c62a4b6be48]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:03.459064 30873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8c0fcef461446eabc1e55f434a2ce9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:03.460418 32683 tablet_bootstrap.cc:492] T 644bb089a4064ecfb88438d8b859bf01 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:13:03.462065 32683 tablet_bootstrap.cc:654] T 644bb089a4064ecfb88438d8b859bf01 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:03.463243 30741 tablet_service.cc:1467] Processing CreateTablet for tablet d8c0fcef461446eabc1e55f434a2ce9f (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=40dc8c7f01804911a18e5c62a4b6be48]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:03.463783 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8c0fcef461446eabc1e55f434a2ce9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:03.464329 28373 tablet_service.cc:1467] Processing CreateTablet for tablet d8c0fcef461446eabc1e55f434a2ce9f (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=40dc8c7f01804911a18e5c62a4b6be48]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:03.464740 28373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8c0fcef461446eabc1e55f434a2ce9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:03.466912 32682 tablet_bootstrap.cc:492] T 644bb089a4064ecfb88438d8b859bf01 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:13:03.468328 31007 tablet_service.cc:1467] Processing CreateTablet for tablet 17c6e3293aad4591a9c1395ebffff2e6 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=40dc8c7f01804911a18e5c62a4b6be48]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:03.468720 31007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17c6e3293aad4591a9c1395ebffff2e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:03.468930 32682 tablet_bootstrap.cc:654] T 644bb089a4064ecfb88438d8b859bf01 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:03.471205 32683 tablet_bootstrap.cc:492] T 644bb089a4064ecfb88438d8b859bf01 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:13:03.471324 32683 ts_tablet_manager.cc:1367] T 644bb089a4064ecfb88438d8b859bf01 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240503 08:13:03.472028 32683 raft_consensus.cc:348] T 644bb089a4064ecfb88438d8b859bf01 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:03.472209 32683 raft_consensus.cc:374] T 644bb089a4064ecfb88438d8b859bf01 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:03.472345 32683 raft_consensus.cc:729] T 644bb089a4064ecfb88438d8b859bf01 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:13:03.472573 32683 consensus_queue.cc:260] T 644bb089a4064ecfb88438d8b859bf01 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:03.473193 32683 ts_tablet_manager.cc:1398] T 644bb089a4064ecfb88438d8b859bf01 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:03.482270 28111 tablet_service.cc:1467] Processing CreateTablet for tablet 17c6e3293aad4591a9c1395ebffff2e6 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=40dc8c7f01804911a18e5c62a4b6be48]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:03.482689 28111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17c6e3293aad4591a9c1395ebffff2e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:03.485231 31141 tablet_service.cc:1467] Processing CreateTablet for tablet 17c6e3293aad4591a9c1395ebffff2e6 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=40dc8c7f01804911a18e5c62a4b6be48]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:03.485705 31141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17c6e3293aad4591a9c1395ebffff2e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:03.490422 32667 tablet_bootstrap.cc:492] T d8c0fcef461446eabc1e55f434a2ce9f P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:13:03.491925 32667 tablet_bootstrap.cc:654] T d8c0fcef461446eabc1e55f434a2ce9f P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:03.500592 32681 tablet_bootstrap.cc:492] T 644bb089a4064ecfb88438d8b859bf01 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:13:03.502278 32686 tablet_bootstrap.cc:492] T 17c6e3293aad4591a9c1395ebffff2e6 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:13:03.502606 32667 tablet_bootstrap.cc:492] T d8c0fcef461446eabc1e55f434a2ce9f P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:13:03.502775 32667 ts_tablet_manager.cc:1367] T d8c0fcef461446eabc1e55f434a2ce9f P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.006s
I20240503 08:13:03.503700 32667 raft_consensus.cc:348] T d8c0fcef461446eabc1e55f434a2ce9f P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:03.503937 32667 raft_consensus.cc:374] T d8c0fcef461446eabc1e55f434a2ce9f P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:03.504009 32667 raft_consensus.cc:729] T d8c0fcef461446eabc1e55f434a2ce9f P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:13:03.506345 32681 tablet_bootstrap.cc:654] T 644bb089a4064ecfb88438d8b859bf01 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:03.504261 32667 consensus_queue.cc:260] T d8c0fcef461446eabc1e55f434a2ce9f P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:03.507506 32667 ts_tablet_manager.cc:1398] T d8c0fcef461446eabc1e55f434a2ce9f P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 08:13:03.507644 32686 tablet_bootstrap.cc:654] T 17c6e3293aad4591a9c1395ebffff2e6 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:03.509956 32682 tablet_bootstrap.cc:492] T 644bb089a4064ecfb88438d8b859bf01 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:13:03.510008 32688 tablet_bootstrap.cc:492] T d8c0fcef461446eabc1e55f434a2ce9f P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:13:03.510115 32682 ts_tablet_manager.cc:1367] T 644bb089a4064ecfb88438d8b859bf01 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.043s user 0.026s sys 0.015s
I20240503 08:13:03.510913 32682 raft_consensus.cc:348] T 644bb089a4064ecfb88438d8b859bf01 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:03.511132 32682 raft_consensus.cc:374] T 644bb089a4064ecfb88438d8b859bf01 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:03.511258 32682 raft_consensus.cc:729] T 644bb089a4064ecfb88438d8b859bf01 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:13:03.511529 32682 consensus_queue.cc:260] T 644bb089a4064ecfb88438d8b859bf01 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:03.512252 32682 ts_tablet_manager.cc:1398] T 644bb089a4064ecfb88438d8b859bf01 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:03.512410 32683 tablet_bootstrap.cc:492] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:13:03.512917 32681 tablet_bootstrap.cc:492] T 644bb089a4064ecfb88438d8b859bf01 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:13:03.513060 32681 ts_tablet_manager.cc:1367] T 644bb089a4064ecfb88438d8b859bf01 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.002s
I20240503 08:13:03.513796 32686 tablet_bootstrap.cc:492] T 17c6e3293aad4591a9c1395ebffff2e6 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:13:03.513931 32686 ts_tablet_manager.cc:1367] T 17c6e3293aad4591a9c1395ebffff2e6 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.001s
I20240503 08:13:03.514712 32686 raft_consensus.cc:348] T 17c6e3293aad4591a9c1395ebffff2e6 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:03.515185 32686 raft_consensus.cc:374] T 17c6e3293aad4591a9c1395ebffff2e6 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:03.515328 32686 raft_consensus.cc:729] T 17c6e3293aad4591a9c1395ebffff2e6 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:13:03.516065 32686 consensus_queue.cc:260] T 17c6e3293aad4591a9c1395ebffff2e6 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:03.516940 32681 raft_consensus.cc:348] T 644bb089a4064ecfb88438d8b859bf01 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:03.517369 32686 ts_tablet_manager.cc:1398] T 17c6e3293aad4591a9c1395ebffff2e6 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:13:03.517596 32681 raft_consensus.cc:374] T 644bb089a4064ecfb88438d8b859bf01 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:03.519359 32681 raft_consensus.cc:729] T 644bb089a4064ecfb88438d8b859bf01 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:13:03.519587 32681 consensus_queue.cc:260] T 644bb089a4064ecfb88438d8b859bf01 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:03.521186 32681 ts_tablet_manager.cc:1398] T 644bb089a4064ecfb88438d8b859bf01 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240503 08:13:03.521514 32681 tablet_bootstrap.cc:492] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:13:03.522879 27856 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:13:03.523698 32681 tablet_bootstrap.cc:654] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:03.529608 32534 raft_consensus.cc:482] T 644bb089a4064ecfb88438d8b859bf01 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:03.529785 32534 raft_consensus.cc:504] T 644bb089a4064ecfb88438d8b859bf01 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:03.530601 32534 leader_election.cc:288] T 644bb089a4064ecfb88438d8b859bf01 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:13:03.532578 28263 raft_consensus.cc:1261] T 4a31fe0fbee142bdb90f4e8c7a66c8de P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:03.533164 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "644bb089a4064ecfb88438d8b859bf01" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:13:03.533147 32448 consensus_queue.cc:1035] T 4a31fe0fbee142bdb90f4e8c7a66c8de P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:03.533386 31027 raft_consensus.cc:2454] T 644bb089a4064ecfb88438d8b859bf01 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 0.
I20240503 08:13:03.533660 32681 tablet_bootstrap.cc:492] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:13:03.533882 32681 ts_tablet_manager.cc:1367] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20240503 08:13:03.533878 31095 leader_election.cc:302] T 644bb089a4064ecfb88438d8b859bf01 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:13:03.534165 32534 raft_consensus.cc:2781] T 644bb089a4064ecfb88438d8b859bf01 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:03.534281 32534 raft_consensus.cc:482] T 644bb089a4064ecfb88438d8b859bf01 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:03.534368 32534 raft_consensus.cc:3037] T 644bb089a4064ecfb88438d8b859bf01 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:03.534969 32681 raft_consensus.cc:348] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:03.535228 32681 raft_consensus.cc:374] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:03.535347 32681 raft_consensus.cc:729] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:13:03.535706 32681 consensus_queue.cc:260] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:03.535866 28003 raft_consensus.cc:1261] T 4a31fe0fbee142bdb90f4e8c7a66c8de P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:03.536393 32681 ts_tablet_manager.cc:1398] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:03.536978 32683 tablet_bootstrap.cc:654] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:03.537689 32448 consensus_queue.cc:1035] T 4a31fe0fbee142bdb90f4e8c7a66c8de P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:03.538596 32534 raft_consensus.cc:504] T 644bb089a4064ecfb88438d8b859bf01 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:03.540113 32534 leader_election.cc:288] T 644bb089a4064ecfb88438d8b859bf01 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 election: Requested vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:13:03.540217 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "644bb089a4064ecfb88438d8b859bf01" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:13:03.540438 31027 raft_consensus.cc:3037] T 644bb089a4064ecfb88438d8b859bf01 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:03.543807 31027 raft_consensus.cc:2454] T 644bb089a4064ecfb88438d8b859bf01 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 1.
I20240503 08:13:03.545102 31095 leader_election.cc:302] T 644bb089a4064ecfb88438d8b859bf01 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:13:03.545364 32534 raft_consensus.cc:2781] T 644bb089a4064ecfb88438d8b859bf01 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:03.545486 32534 raft_consensus.cc:686] T 644bb089a4064ecfb88438d8b859bf01 P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Becoming Leader. State: Replica: b00f244c69df401997ea99d492218a98, State: Running, Role: LEADER
I20240503 08:13:03.545792 32534 consensus_queue.cc:237] T 644bb089a4064ecfb88438d8b859bf01 P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:03.546689 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "644bb089a4064ecfb88438d8b859bf01" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:13:03.546905 28391 raft_consensus.cc:2454] T 644bb089a4064ecfb88438d8b859bf01 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 0.
I20240503 08:13:03.547200 28390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "644bb089a4064ecfb88438d8b859bf01" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:13:03.547387 28390 raft_consensus.cc:3037] T 644bb089a4064ecfb88438d8b859bf01 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:03.550565 27856 catalog_manager.cc:5576] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919) does not have the latest schema for tablet 4a31fe0fbee142bdb90f4e8c7a66c8de (table rebalancer_dynamic_table [id=924c17d4e8a645a2b093b33e05af5507]). Expected version 1 got 0
I20240503 08:13:03.551662 27862 catalog_manager.cc:5576] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081) does not have the latest schema for tablet 4a31fe0fbee142bdb90f4e8c7a66c8de (table rebalancer_dynamic_table [id=924c17d4e8a645a2b093b33e05af5507]). Expected version 1 got 0
I20240503 08:13:03.553651 28390 raft_consensus.cc:2454] T 644bb089a4064ecfb88438d8b859bf01 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 1.
I20240503 08:13:03.556743 32683 tablet_bootstrap.cc:492] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:13:03.556951 32683 ts_tablet_manager.cc:1367] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.000s
I20240503 08:13:03.558427 32683 raft_consensus.cc:348] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:03.558928 32682 tablet_bootstrap.cc:492] T d8c0fcef461446eabc1e55f434a2ce9f P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:13:03.561208 32682 tablet_bootstrap.cc:654] T d8c0fcef461446eabc1e55f434a2ce9f P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:03.561853 32683 raft_consensus.cc:374] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:03.561993 32683 raft_consensus.cc:729] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:13:03.563190 32683 consensus_queue.cc:260] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:03.563227 32698 tablet.cc:1722] T 4a31fe0fbee142bdb90f4e8c7a66c8de P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:13:03.563831 32698 tablet.cc:1620] T 4a31fe0fbee142bdb90f4e8c7a66c8de P ce1496a47d084365ad786cd09e472a2c: MemRowSet was empty: no flush needed.
I20240503 08:13:03.562693 32688 tablet_bootstrap.cc:654] T d8c0fcef461446eabc1e55f434a2ce9f P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:03.568753 32683 ts_tablet_manager.cc:1398] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20240503 08:13:03.573060 27856 catalog_manager.cc:5576] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919) does not have the latest schema for tablet 4a31fe0fbee142bdb90f4e8c7a66c8de (table rebalancer_dynamic_table [id=924c17d4e8a645a2b093b33e05af5507]). Expected version 1 got 0
I20240503 08:13:03.574690 32697 tablet.cc:1722] T 4a31fe0fbee142bdb90f4e8c7a66c8de P 833582ff68c94269be364ebd187a75db: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:13:03.575287 32697 tablet.cc:1620] T 4a31fe0fbee142bdb90f4e8c7a66c8de P 833582ff68c94269be364ebd187a75db: MemRowSet was empty: no flush needed.
I20240503 08:13:03.576185 27862 catalog_manager.cc:5484] T 644bb089a4064ecfb88438d8b859bf01 P b00f244c69df401997ea99d492218a98 reported cstate change: term changed from 0 to 1, leader changed from <none> to b00f244c69df401997ea99d492218a98 (127.20.230.72). New cstate: current_term: 1 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:13:03.578006 32696 tablet.cc:1722] T 4a31fe0fbee142bdb90f4e8c7a66c8de P aae860d7ae2b4002bf7794c278f9cc56: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:13:03.578552 32688 tablet_bootstrap.cc:492] T d8c0fcef461446eabc1e55f434a2ce9f P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:13:03.578740 32696 tablet.cc:1620] T 4a31fe0fbee142bdb90f4e8c7a66c8de P aae860d7ae2b4002bf7794c278f9cc56: MemRowSet was empty: no flush needed.
I20240503 08:13:03.578830 32688 ts_tablet_manager.cc:1367] T d8c0fcef461446eabc1e55f434a2ce9f P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.069s user 0.002s sys 0.002s
I20240503 08:13:03.581365 32688 raft_consensus.cc:348] T d8c0fcef461446eabc1e55f434a2ce9f P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:03.581647 32688 raft_consensus.cc:374] T d8c0fcef461446eabc1e55f434a2ce9f P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:03.581810 32688 raft_consensus.cc:729] T d8c0fcef461446eabc1e55f434a2ce9f P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:13:03.582121 32688 consensus_queue.cc:260] T d8c0fcef461446eabc1e55f434a2ce9f P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:03.582751 32688 ts_tablet_manager.cc:1398] T d8c0fcef461446eabc1e55f434a2ce9f P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:13:03.585079 27855 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=924c17d4e8a645a2b093b33e05af5507] alter complete (version 1)
I20240503 08:13:03.590014 32682 tablet_bootstrap.cc:492] T d8c0fcef461446eabc1e55f434a2ce9f P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:13:03.590162 32682 ts_tablet_manager.cc:1367] T d8c0fcef461446eabc1e55f434a2ce9f P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.031s user 0.006s sys 0.007s
I20240503 08:13:03.592871 32682 raft_consensus.cc:348] T d8c0fcef461446eabc1e55f434a2ce9f P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:03.593082 32682 raft_consensus.cc:374] T d8c0fcef461446eabc1e55f434a2ce9f P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:03.593150 32682 raft_consensus.cc:729] T d8c0fcef461446eabc1e55f434a2ce9f P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:13:03.594036 32682 consensus_queue.cc:260] T d8c0fcef461446eabc1e55f434a2ce9f P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:03.594671 32682 ts_tablet_manager.cc:1398] T d8c0fcef461446eabc1e55f434a2ce9f P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 08:13:03.605226 27856 catalog_manager.cc:5576] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081) does not have the latest schema for tablet 4a31fe0fbee142bdb90f4e8c7a66c8de (table rebalancer_dynamic_table [id=924c17d4e8a645a2b093b33e05af5507]). Expected version 1 got 0
I20240503 08:13:03.704774 27856 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:13:03.713809 32698 tablet.cc:1722] T 4a31fe0fbee142bdb90f4e8c7a66c8de P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:13:03.714286 32698 tablet.cc:1620] T 4a31fe0fbee142bdb90f4e8c7a66c8de P ce1496a47d084365ad786cd09e472a2c: MemRowSet was empty: no flush needed.
I20240503 08:13:03.714512 32696 tablet.cc:1722] T 4a31fe0fbee142bdb90f4e8c7a66c8de P aae860d7ae2b4002bf7794c278f9cc56: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:13:03.714957 32696 tablet.cc:1620] T 4a31fe0fbee142bdb90f4e8c7a66c8de P aae860d7ae2b4002bf7794c278f9cc56: MemRowSet was empty: no flush needed.
I20240503 08:13:03.725275 27856 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=924c17d4e8a645a2b093b33e05af5507] alter complete (version 2)
I20240503 08:13:03.725648 32697 tablet.cc:1722] T 4a31fe0fbee142bdb90f4e8c7a66c8de P 833582ff68c94269be364ebd187a75db: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:13:03.726073 32697 tablet.cc:1620] T 4a31fe0fbee142bdb90f4e8c7a66c8de P 833582ff68c94269be364ebd187a75db: MemRowSet was empty: no flush needed.
I20240503 08:13:03.769210 32677 raft_consensus.cc:482] T d8c0fcef461446eabc1e55f434a2ce9f P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:03.769349 32677 raft_consensus.cc:504] T d8c0fcef461446eabc1e55f434a2ce9f P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:03.770025 32677 leader_election.cc:288] T d8c0fcef461446eabc1e55f434a2ce9f P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:13:03.770287 28390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8c0fcef461446eabc1e55f434a2ce9f" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:13:03.770485 28390 raft_consensus.cc:2454] T d8c0fcef461446eabc1e55f434a2ce9f P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:13:03.770427 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8c0fcef461446eabc1e55f434a2ce9f" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:13:03.770632 30761 raft_consensus.cc:2454] T d8c0fcef461446eabc1e55f434a2ce9f P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:13:03.770893 30830 leader_election.cc:302] T d8c0fcef461446eabc1e55f434a2ce9f P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:13:03.771147 32677 raft_consensus.cc:2781] T d8c0fcef461446eabc1e55f434a2ce9f P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:03.771257 32677 raft_consensus.cc:482] T d8c0fcef461446eabc1e55f434a2ce9f P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:03.771349 32677 raft_consensus.cc:3037] T d8c0fcef461446eabc1e55f434a2ce9f P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:03.773826 32677 raft_consensus.cc:504] T d8c0fcef461446eabc1e55f434a2ce9f P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:03.774350 32677 leader_election.cc:288] T d8c0fcef461446eabc1e55f434a2ce9f P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Requested vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:13:03.774596 28390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8c0fcef461446eabc1e55f434a2ce9f" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:13:03.774796 28390 raft_consensus.cc:3037] T d8c0fcef461446eabc1e55f434a2ce9f P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:03.774817 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8c0fcef461446eabc1e55f434a2ce9f" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:13:03.774983 30761 raft_consensus.cc:3037] T d8c0fcef461446eabc1e55f434a2ce9f P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:03.777482 28390 raft_consensus.cc:2454] T d8c0fcef461446eabc1e55f434a2ce9f P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:13:03.777514 30761 raft_consensus.cc:2454] T d8c0fcef461446eabc1e55f434a2ce9f P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:13:03.777916 30830 leader_election.cc:302] T d8c0fcef461446eabc1e55f434a2ce9f P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ce1496a47d084365ad786cd09e472a2c, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:13:03.778152 32677 raft_consensus.cc:2781] T d8c0fcef461446eabc1e55f434a2ce9f P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:03.778266 32677 raft_consensus.cc:686] T d8c0fcef461446eabc1e55f434a2ce9f P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Becoming Leader. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Running, Role: LEADER
I20240503 08:13:03.778504 32677 consensus_queue.cc:237] T d8c0fcef461446eabc1e55f434a2ce9f P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:03.781682 27862 catalog_manager.cc:5484] T d8c0fcef461446eabc1e55f434a2ce9f P ce1496a47d084365ad786cd09e472a2c reported cstate change: term changed from 0 to 1, leader changed from <none> to ce1496a47d084365ad786cd09e472a2c (127.20.230.70). New cstate: current_term: 1 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:03.809273 32534 raft_consensus.cc:482] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:03.809841 32534 raft_consensus.cc:504] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:03.810406 32534 leader_election.cc:288] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:13:03.810856 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17c6e3293aad4591a9c1395ebffff2e6" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:13:03.810899 28131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17c6e3293aad4591a9c1395ebffff2e6" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:13:03.811048 31027 raft_consensus.cc:2454] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 0.
I20240503 08:13:03.811165 28131 raft_consensus.cc:2454] T 17c6e3293aad4591a9c1395ebffff2e6 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 0.
I20240503 08:13:03.811733 31095 leader_election.cc:302] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:13:03.811957 32534 raft_consensus.cc:2781] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:03.812062 32534 raft_consensus.cc:482] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:03.812166 32534 raft_consensus.cc:3037] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:03.814635 32534 raft_consensus.cc:504] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:03.815168 32534 leader_election.cc:288] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 election: Requested vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:13:03.815532 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17c6e3293aad4591a9c1395ebffff2e6" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:13:03.815564 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17c6e3293aad4591a9c1395ebffff2e6" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:13:03.815723 28132 raft_consensus.cc:3037] T 17c6e3293aad4591a9c1395ebffff2e6 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:03.815733 31027 raft_consensus.cc:3037] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:03.818329 28132 raft_consensus.cc:2454] T 17c6e3293aad4591a9c1395ebffff2e6 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 1.
I20240503 08:13:03.818329 31027 raft_consensus.cc:2454] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 1.
I20240503 08:13:03.818679 31095 leader_election.cc:302] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:13:03.820086 32534 raft_consensus.cc:2781] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:03.820515 32534 raft_consensus.cc:686] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Becoming Leader. State: Replica: b00f244c69df401997ea99d492218a98, State: Running, Role: LEADER
I20240503 08:13:03.820773 32534 consensus_queue.cc:237] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:03.824833 27862 catalog_manager.cc:5484] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98 reported cstate change: term changed from 0 to 1, leader changed from <none> to b00f244c69df401997ea99d492218a98 (127.20.230.72). New cstate: current_term: 1 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:13:03.834070 27862 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:13:03.840454 32698 tablet.cc:1722] T 4a31fe0fbee142bdb90f4e8c7a66c8de P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:13:03.841255 32696 tablet.cc:1722] T 4a31fe0fbee142bdb90f4e8c7a66c8de P aae860d7ae2b4002bf7794c278f9cc56: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:13:03.844497 32697 tablet.cc:1722] T 4a31fe0fbee142bdb90f4e8c7a66c8de P 833582ff68c94269be364ebd187a75db: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:13:03.853418 27856 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table_new_name [id=924c17d4e8a645a2b093b33e05af5507] alter complete (version 3)
I20240503 08:13:03.955080 27859 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:13:03.955327 27859 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:13:03.958154 27859 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet 4a31fe0fbee142bdb90f4e8c7a66c8de
I20240503 08:13:03.959071 27981 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a31fe0fbee142bdb90f4e8c7a66c8de with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:03 UTC) from {username='slave'} at 127.0.0.1:55610
I20240503 08:13:03.959101 30873 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a31fe0fbee142bdb90f4e8c7a66c8de with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:03 UTC) from {username='slave'} at 127.0.0.1:34426
I20240503 08:13:03.959504 28243 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a31fe0fbee142bdb90f4e8c7a66c8de with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:03 UTC) from {username='slave'} at 127.0.0.1:41960
I20240503 08:13:03.960116 32708 tablet_replica.cc:330] T 4a31fe0fbee142bdb90f4e8c7a66c8de P ce1496a47d084365ad786cd09e472a2c: stopping tablet replica
I20240503 08:13:03.960338 32708 raft_consensus.cc:2229] T 4a31fe0fbee142bdb90f4e8c7a66c8de P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:13:03.960691 32708 raft_consensus.cc:2258] T 4a31fe0fbee142bdb90f4e8c7a66c8de P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:03.961828 32709 tablet_replica.cc:330] T 4a31fe0fbee142bdb90f4e8c7a66c8de P aae860d7ae2b4002bf7794c278f9cc56: stopping tablet replica
I20240503 08:13:03.962054 32708 ts_tablet_manager.cc:1875] T 4a31fe0fbee142bdb90f4e8c7a66c8de P ce1496a47d084365ad786cd09e472a2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:03.962138 32709 raft_consensus.cc:2229] T 4a31fe0fbee142bdb90f4e8c7a66c8de P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:03.962448 32709 raft_consensus.cc:2258] T 4a31fe0fbee142bdb90f4e8c7a66c8de P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:03.963858 32709 ts_tablet_manager.cc:1875] T 4a31fe0fbee142bdb90f4e8c7a66c8de P aae860d7ae2b4002bf7794c278f9cc56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:03.966163 32707 tablet_replica.cc:330] T 4a31fe0fbee142bdb90f4e8c7a66c8de P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:13:03.966367 32707 raft_consensus.cc:2229] T 4a31fe0fbee142bdb90f4e8c7a66c8de P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:03.966593 32707 raft_consensus.cc:2258] T 4a31fe0fbee142bdb90f4e8c7a66c8de P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:03.967329 32708 ts_tablet_manager.cc:1888] T 4a31fe0fbee142bdb90f4e8c7a66c8de P ce1496a47d084365ad786cd09e472a2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:13:03.967322 32707 ts_tablet_manager.cc:1875] T 4a31fe0fbee142bdb90f4e8c7a66c8de P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:03.967444 32708 log.cc:1196] T 4a31fe0fbee142bdb90f4e8c7a66c8de P ce1496a47d084365ad786cd09e472a2c: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal/wals/4a31fe0fbee142bdb90f4e8c7a66c8de
I20240503 08:13:03.967886 32708 ts_tablet_manager.cc:1909] T 4a31fe0fbee142bdb90f4e8c7a66c8de P ce1496a47d084365ad786cd09e472a2c: Deleting consensus metadata
I20240503 08:13:03.969679 32709 ts_tablet_manager.cc:1888] T 4a31fe0fbee142bdb90f4e8c7a66c8de P aae860d7ae2b4002bf7794c278f9cc56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:13:03.969666 27846 catalog_manager.cc:4830] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): tablet 4a31fe0fbee142bdb90f4e8c7a66c8de (table rebalancer_dynamic_table_new_name [id=924c17d4e8a645a2b093b33e05af5507]) successfully deleted
I20240503 08:13:03.969892 32709 log.cc:1196] T 4a31fe0fbee142bdb90f4e8c7a66c8de P aae860d7ae2b4002bf7794c278f9cc56: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal/wals/4a31fe0fbee142bdb90f4e8c7a66c8de
I20240503 08:13:03.970304 32709 ts_tablet_manager.cc:1909] T 4a31fe0fbee142bdb90f4e8c7a66c8de P aae860d7ae2b4002bf7794c278f9cc56: Deleting consensus metadata
I20240503 08:13:03.971040 27847 catalog_manager.cc:4830] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): tablet 4a31fe0fbee142bdb90f4e8c7a66c8de (table rebalancer_dynamic_table_new_name [id=924c17d4e8a645a2b093b33e05af5507]) successfully deleted
I20240503 08:13:03.972520 32707 ts_tablet_manager.cc:1888] T 4a31fe0fbee142bdb90f4e8c7a66c8de P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:13:03.972631 32707 log.cc:1196] T 4a31fe0fbee142bdb90f4e8c7a66c8de P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/4a31fe0fbee142bdb90f4e8c7a66c8de
I20240503 08:13:03.973002 32707 ts_tablet_manager.cc:1909] T 4a31fe0fbee142bdb90f4e8c7a66c8de P 833582ff68c94269be364ebd187a75db: Deleting consensus metadata
I20240503 08:13:03.974210 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet 4a31fe0fbee142bdb90f4e8c7a66c8de (table rebalancer_dynamic_table_new_name [id=924c17d4e8a645a2b093b33e05af5507]) successfully deleted
I20240503 08:13:04.062945 27859 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:13:04.071002 28243 tablet_service.cc:1467] Processing CreateTablet for tablet e52da272af784435ab2355d3fe2df27a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fb12df737cb54402a4cad2fde5a851f0]), partition=
I20240503 08:13:04.071424 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e52da272af784435ab2355d3fe2df27a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:04.074347 30741 tablet_service.cc:1467] Processing CreateTablet for tablet e52da272af784435ab2355d3fe2df27a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fb12df737cb54402a4cad2fde5a851f0]), partition=
I20240503 08:13:04.074718 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e52da272af784435ab2355d3fe2df27a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:04.074776 27981 tablet_service.cc:1467] Processing CreateTablet for tablet e52da272af784435ab2355d3fe2df27a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fb12df737cb54402a4cad2fde5a851f0]), partition=
I20240503 08:13:04.077136 27981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e52da272af784435ab2355d3fe2df27a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:04.080456 32688 tablet_bootstrap.cc:492] T e52da272af784435ab2355d3fe2df27a P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:13:04.082465 32688 tablet_bootstrap.cc:654] T e52da272af784435ab2355d3fe2df27a P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:04.090510 32534 consensus_queue.cc:1035] T 644bb089a4064ecfb88438d8b859bf01 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:04.102456 32710 tablet_bootstrap.cc:492] T e52da272af784435ab2355d3fe2df27a P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:13:04.104398 32710 tablet_bootstrap.cc:654] T e52da272af784435ab2355d3fe2df27a P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:04.106539 32714 tablet_bootstrap.cc:492] T e52da272af784435ab2355d3fe2df27a P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:13:04.108525 32714 tablet_bootstrap.cc:654] T e52da272af784435ab2355d3fe2df27a P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:04.110684 32714 tablet_bootstrap.cc:492] T e52da272af784435ab2355d3fe2df27a P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:13:04.110805 32714 ts_tablet_manager.cc:1367] T e52da272af784435ab2355d3fe2df27a P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240503 08:13:04.111347 32714 raft_consensus.cc:348] T e52da272af784435ab2355d3fe2df27a P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:04.111514 32714 raft_consensus.cc:374] T e52da272af784435ab2355d3fe2df27a P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:04.111594 32714 raft_consensus.cc:729] T e52da272af784435ab2355d3fe2df27a P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:13:04.111819 32714 consensus_queue.cc:260] T e52da272af784435ab2355d3fe2df27a P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:04.112665 32714 ts_tablet_manager.cc:1398] T e52da272af784435ab2355d3fe2df27a P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:13:04.114521 32710 tablet_bootstrap.cc:492] T e52da272af784435ab2355d3fe2df27a P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:13:04.114670 32710 ts_tablet_manager.cc:1367] T e52da272af784435ab2355d3fe2df27a P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 08:13:04.115365 32710 raft_consensus.cc:348] T e52da272af784435ab2355d3fe2df27a P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:04.115573 32710 raft_consensus.cc:374] T e52da272af784435ab2355d3fe2df27a P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:04.115660 32710 raft_consensus.cc:729] T e52da272af784435ab2355d3fe2df27a P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:13:04.115861 32710 consensus_queue.cc:260] T e52da272af784435ab2355d3fe2df27a P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:04.116392 32710 ts_tablet_manager.cc:1398] T e52da272af784435ab2355d3fe2df27a P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:13:04.131742 32493 consensus_queue.cc:1035] T 644bb089a4064ecfb88438d8b859bf01 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:04.134552 32688 tablet_bootstrap.cc:492] T e52da272af784435ab2355d3fe2df27a P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:13:04.134791 32688 ts_tablet_manager.cc:1367] T e52da272af784435ab2355d3fe2df27a P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.054s user 0.007s sys 0.014s
I20240503 08:13:04.135830 32688 raft_consensus.cc:348] T e52da272af784435ab2355d3fe2df27a P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:04.136004 32688 raft_consensus.cc:374] T e52da272af784435ab2355d3fe2df27a P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:04.136075 32688 raft_consensus.cc:729] T e52da272af784435ab2355d3fe2df27a P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:13:04.136341 32688 consensus_queue.cc:260] T e52da272af784435ab2355d3fe2df27a P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:04.136952 32688 ts_tablet_manager.cc:1398] T e52da272af784435ab2355d3fe2df27a P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:13:04.169349 32706 consensus_queue.cc:1035] T d8c0fcef461446eabc1e55f434a2ce9f P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:04.173544 32677 consensus_queue.cc:1035] T d8c0fcef461446eabc1e55f434a2ce9f P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:04.279294 32493 consensus_queue.cc:1035] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:04.283457 32493 consensus_queue.cc:1035] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:04.310371 32517 raft_consensus.cc:482] T e52da272af784435ab2355d3fe2df27a P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:04.310526 32517 raft_consensus.cc:504] T e52da272af784435ab2355d3fe2df27a P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:04.311007 32517 leader_election.cc:288] T e52da272af784435ab2355d3fe2df27a P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:13:04.311434 28263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e52da272af784435ab2355d3fe2df27a" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:13:04.311411 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e52da272af784435ab2355d3fe2df27a" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:13:04.311642 28263 raft_consensus.cc:2454] T e52da272af784435ab2355d3fe2df27a P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:13:04.311658 28002 raft_consensus.cc:2454] T e52da272af784435ab2355d3fe2df27a P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:13:04.312117 30697 leader_election.cc:302] T e52da272af784435ab2355d3fe2df27a P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, 833582ff68c94269be364ebd187a75db; no voters:
I20240503 08:13:04.312453 32517 raft_consensus.cc:2781] T e52da272af784435ab2355d3fe2df27a P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:04.312547 32517 raft_consensus.cc:482] T e52da272af784435ab2355d3fe2df27a P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:04.312669 32517 raft_consensus.cc:3037] T e52da272af784435ab2355d3fe2df27a P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:04.315039 32517 raft_consensus.cc:504] T e52da272af784435ab2355d3fe2df27a P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:04.315577 32517 leader_election.cc:288] T e52da272af784435ab2355d3fe2df27a P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Requested vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:13:04.315989 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e52da272af784435ab2355d3fe2df27a" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:13:04.315965 28263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e52da272af784435ab2355d3fe2df27a" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:13:04.316175 28002 raft_consensus.cc:3037] T e52da272af784435ab2355d3fe2df27a P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:04.316193 28263 raft_consensus.cc:3037] T e52da272af784435ab2355d3fe2df27a P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:04.318801 28002 raft_consensus.cc:2454] T e52da272af784435ab2355d3fe2df27a P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:13:04.318852 28263 raft_consensus.cc:2454] T e52da272af784435ab2355d3fe2df27a P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:13:04.319177 30695 leader_election.cc:302] T e52da272af784435ab2355d3fe2df27a P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:13:04.319428 32517 raft_consensus.cc:2781] T e52da272af784435ab2355d3fe2df27a P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:04.319638 32517 raft_consensus.cc:686] T e52da272af784435ab2355d3fe2df27a P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Becoming Leader. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Running, Role: LEADER
I20240503 08:13:04.319909 32517 consensus_queue.cc:237] T e52da272af784435ab2355d3fe2df27a P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:04.323572 27862 catalog_manager.cc:5484] T e52da272af784435ab2355d3fe2df27a P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69). New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:04.348253 27862 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_extra_table_54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:13:04.356673 27981 tablet_service.cc:1467] Processing CreateTablet for tablet 62ca502a089f4a37999d2252776b741b (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=a50fec2484c34f48867b5f6f7ca7f1e0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:04.357113 27981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62ca502a089f4a37999d2252776b741b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:04.357028 28243 tablet_service.cc:1467] Processing CreateTablet for tablet 62ca502a089f4a37999d2252776b741b (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=a50fec2484c34f48867b5f6f7ca7f1e0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:04.357432 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62ca502a089f4a37999d2252776b741b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:04.357473 27983 tablet_service.cc:1467] Processing CreateTablet for tablet e0b904461c1c44ed93e1a227d93498eb (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=a50fec2484c34f48867b5f6f7ca7f1e0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:04.357578 30741 tablet_service.cc:1467] Processing CreateTablet for tablet e0b904461c1c44ed93e1a227d93498eb (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=a50fec2484c34f48867b5f6f7ca7f1e0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:04.357863 27983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0b904461c1c44ed93e1a227d93498eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:04.357981 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0b904461c1c44ed93e1a227d93498eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:04.363909 32710 tablet_bootstrap.cc:492] T 62ca502a089f4a37999d2252776b741b P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:13:04.364372 32714 tablet_bootstrap.cc:492] T 62ca502a089f4a37999d2252776b741b P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:13:04.365221 32710 tablet_bootstrap.cc:654] T 62ca502a089f4a37999d2252776b741b P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:04.365653 32688 tablet_bootstrap.cc:492] T e0b904461c1c44ed93e1a227d93498eb P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:13:04.365806 32714 tablet_bootstrap.cc:654] T 62ca502a089f4a37999d2252776b741b P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:04.366916 30873 tablet_service.cc:1467] Processing CreateTablet for tablet 62ca502a089f4a37999d2252776b741b (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=a50fec2484c34f48867b5f6f7ca7f1e0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:04.367090 32688 tablet_bootstrap.cc:654] T e0b904461c1c44ed93e1a227d93498eb P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:04.367278 30873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62ca502a089f4a37999d2252776b741b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:04.368242 28373 tablet_service.cc:1467] Processing CreateTablet for tablet 1e8dd968fc6c4416ae57cfcf7d8b4693 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=a50fec2484c34f48867b5f6f7ca7f1e0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:04.368661 28373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e8dd968fc6c4416ae57cfcf7d8b4693. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:04.368542 31006 tablet_service.cc:1467] Processing CreateTablet for tablet 1e8dd968fc6c4416ae57cfcf7d8b4693 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=a50fec2484c34f48867b5f6f7ca7f1e0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:04.368952 31006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e8dd968fc6c4416ae57cfcf7d8b4693. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:04.369146 28243 tablet_service.cc:1467] Processing CreateTablet for tablet e0b904461c1c44ed93e1a227d93498eb (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=a50fec2484c34f48867b5f6f7ca7f1e0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:04.369473 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0b904461c1c44ed93e1a227d93498eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:04.370065 28242 tablet_service.cc:1467] Processing CreateTablet for tablet 1e8dd968fc6c4416ae57cfcf7d8b4693 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=a50fec2484c34f48867b5f6f7ca7f1e0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:04.370363 28242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e8dd968fc6c4416ae57cfcf7d8b4693. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:04.383067 32714 tablet_bootstrap.cc:492] T 62ca502a089f4a37999d2252776b741b P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:13:04.383239 32714 ts_tablet_manager.cc:1367] T 62ca502a089f4a37999d2252776b741b P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.002s
I20240503 08:13:04.384024 32714 raft_consensus.cc:348] T 62ca502a089f4a37999d2252776b741b P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:04.384652 32714 raft_consensus.cc:374] T 62ca502a089f4a37999d2252776b741b P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:04.384735 32714 raft_consensus.cc:729] T 62ca502a089f4a37999d2252776b741b P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:13:04.384964 32714 consensus_queue.cc:260] T 62ca502a089f4a37999d2252776b741b P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:04.385475 32714 ts_tablet_manager.cc:1398] T 62ca502a089f4a37999d2252776b741b P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:04.385823 32688 tablet_bootstrap.cc:492] T e0b904461c1c44ed93e1a227d93498eb P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:13:04.385977 32688 ts_tablet_manager.cc:1367] T e0b904461c1c44ed93e1a227d93498eb P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.003s
I20240503 08:13:04.386682 32688 raft_consensus.cc:348] T e0b904461c1c44ed93e1a227d93498eb P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:04.386866 32688 raft_consensus.cc:374] T e0b904461c1c44ed93e1a227d93498eb P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:04.386955 32688 raft_consensus.cc:729] T e0b904461c1c44ed93e1a227d93498eb P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:13:04.387200 32688 consensus_queue.cc:260] T e0b904461c1c44ed93e1a227d93498eb P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:04.387488 32714 tablet_bootstrap.cc:492] T e0b904461c1c44ed93e1a227d93498eb P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:13:04.389281 32688 ts_tablet_manager.cc:1398] T e0b904461c1c44ed93e1a227d93498eb P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:13:04.389784 32714 tablet_bootstrap.cc:654] T e0b904461c1c44ed93e1a227d93498eb P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:04.389919 32735 tablet_bootstrap.cc:492] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:13:04.391638 32735 tablet_bootstrap.cc:654] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:04.394948 32710 tablet_bootstrap.cc:492] T 62ca502a089f4a37999d2252776b741b P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:13:04.395161 32710 ts_tablet_manager.cc:1367] T 62ca502a089f4a37999d2252776b741b P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.000s
I20240503 08:13:04.395996 32710 raft_consensus.cc:348] T 62ca502a089f4a37999d2252776b741b P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:04.396824 32710 raft_consensus.cc:374] T 62ca502a089f4a37999d2252776b741b P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:04.396967 32710 raft_consensus.cc:729] T 62ca502a089f4a37999d2252776b741b P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:13:04.397274 32710 consensus_queue.cc:260] T 62ca502a089f4a37999d2252776b741b P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:04.397876 32710 ts_tablet_manager.cc:1398] T 62ca502a089f4a37999d2252776b741b P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:13:04.399260 32710 tablet_bootstrap.cc:492] T e0b904461c1c44ed93e1a227d93498eb P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:13:04.400004 32734 tablet_bootstrap.cc:492] T 62ca502a089f4a37999d2252776b741b P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:13:04.401051 32710 tablet_bootstrap.cc:654] T e0b904461c1c44ed93e1a227d93498eb P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:04.402017 32734 tablet_bootstrap.cc:654] T 62ca502a089f4a37999d2252776b741b P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:04.406836 32733 tablet_bootstrap.cc:492] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:13:04.407344 32735 tablet_bootstrap.cc:492] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:13:04.407572 32735 ts_tablet_manager.cc:1367] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.006s
I20240503 08:13:04.408326 32735 raft_consensus.cc:348] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:04.408636 32735 raft_consensus.cc:374] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:04.409483 32733 tablet_bootstrap.cc:654] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:04.409973 32734 tablet_bootstrap.cc:492] T 62ca502a089f4a37999d2252776b741b P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:13:04.410156 32734 ts_tablet_manager.cc:1367] T 62ca502a089f4a37999d2252776b741b P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20240503 08:13:04.410547 32735 raft_consensus.cc:729] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:13:04.410846 32735 consensus_queue.cc:260] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:04.410957 32734 raft_consensus.cc:348] T 62ca502a089f4a37999d2252776b741b P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:04.411132 32734 raft_consensus.cc:374] T 62ca502a089f4a37999d2252776b741b P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:04.411217 32734 raft_consensus.cc:729] T 62ca502a089f4a37999d2252776b741b P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:13:04.411099 32714 tablet_bootstrap.cc:492] T e0b904461c1c44ed93e1a227d93498eb P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:13:04.411461 32714 ts_tablet_manager.cc:1367] T e0b904461c1c44ed93e1a227d93498eb P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.005s
I20240503 08:13:04.411502 32734 consensus_queue.cc:260] T 62ca502a089f4a37999d2252776b741b P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:04.412312 32714 raft_consensus.cc:348] T e0b904461c1c44ed93e1a227d93498eb P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:04.412497 32714 raft_consensus.cc:374] T e0b904461c1c44ed93e1a227d93498eb P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:04.412513 32735 ts_tablet_manager.cc:1398] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 08:13:04.412582 32714 raft_consensus.cc:729] T e0b904461c1c44ed93e1a227d93498eb P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:13:04.412809 32714 consensus_queue.cc:260] T e0b904461c1c44ed93e1a227d93498eb P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:04.413527 32734 ts_tablet_manager.cc:1398] T 62ca502a089f4a37999d2252776b741b P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:13:04.414479 32714 ts_tablet_manager.cc:1398] T e0b904461c1c44ed93e1a227d93498eb P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 08:13:04.415395 32710 tablet_bootstrap.cc:492] T e0b904461c1c44ed93e1a227d93498eb P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:13:04.415560 32710 ts_tablet_manager.cc:1367] T e0b904461c1c44ed93e1a227d93498eb P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.002s
I20240503 08:13:04.416312 32710 raft_consensus.cc:348] T e0b904461c1c44ed93e1a227d93498eb P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:04.416572 32710 raft_consensus.cc:374] T e0b904461c1c44ed93e1a227d93498eb P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:04.416723 32710 raft_consensus.cc:729] T e0b904461c1c44ed93e1a227d93498eb P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:13:04.416999 32710 consensus_queue.cc:260] T e0b904461c1c44ed93e1a227d93498eb P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:04.417580 32710 ts_tablet_manager.cc:1398] T e0b904461c1c44ed93e1a227d93498eb P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:13:04.417927 32710 tablet_bootstrap.cc:492] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:13:04.419951 32677 raft_consensus.cc:482] T 62ca502a089f4a37999d2252776b741b P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:04.420080 32677 raft_consensus.cc:504] T 62ca502a089f4a37999d2252776b741b P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:04.420816 32677 leader_election.cc:288] T 62ca502a089f4a37999d2252776b741b P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:13:04.421370 28263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62ca502a089f4a37999d2252776b741b" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:13:04.421594 28263 raft_consensus.cc:2454] T 62ca502a089f4a37999d2252776b741b P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:13:04.421901 32710 tablet_bootstrap.cc:654] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:04.422005 30828 leader_election.cc:302] T 62ca502a089f4a37999d2252776b741b P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:13:04.422586 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62ca502a089f4a37999d2252776b741b" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:13:04.422787 28002 raft_consensus.cc:2454] T 62ca502a089f4a37999d2252776b741b P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:13:04.423099 32677 raft_consensus.cc:2781] T 62ca502a089f4a37999d2252776b741b P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:04.423226 32677 raft_consensus.cc:482] T 62ca502a089f4a37999d2252776b741b P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:04.423316 32677 raft_consensus.cc:3037] T 62ca502a089f4a37999d2252776b741b P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:04.423864 32733 tablet_bootstrap.cc:492] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:13:04.424120 32733 ts_tablet_manager.cc:1367] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.003s
I20240503 08:13:04.424959 32733 raft_consensus.cc:348] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:04.425176 32733 raft_consensus.cc:374] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:04.425261 32733 raft_consensus.cc:729] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:13:04.425371 32710 tablet_bootstrap.cc:492] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:13:04.425519 32733 consensus_queue.cc:260] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:04.425997 32710 ts_tablet_manager.cc:1367] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.003s
I20240503 08:13:04.426115 32733 ts_tablet_manager.cc:1398] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:04.426602 32710 raft_consensus.cc:348] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:04.426764 32710 raft_consensus.cc:374] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:04.426839 32710 raft_consensus.cc:729] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:13:04.426981 32677 raft_consensus.cc:504] T 62ca502a089f4a37999d2252776b741b P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:04.427047 32710 consensus_queue.cc:260] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:04.427546 32710 ts_tablet_manager.cc:1398] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:13:04.428521 32677 leader_election.cc:288] T 62ca502a089f4a37999d2252776b741b P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Requested vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:13:04.428992 28263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62ca502a089f4a37999d2252776b741b" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:13:04.429172 28263 raft_consensus.cc:3037] T 62ca502a089f4a37999d2252776b741b P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:04.431326 28263 raft_consensus.cc:2454] T 62ca502a089f4a37999d2252776b741b P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:13:04.429672 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62ca502a089f4a37999d2252776b741b" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:13:04.431824 28002 raft_consensus.cc:3037] T 62ca502a089f4a37999d2252776b741b P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:04.433594 30828 leader_election.cc:302] T 62ca502a089f4a37999d2252776b741b P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:13:04.433862 32677 raft_consensus.cc:2781] T 62ca502a089f4a37999d2252776b741b P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:04.434057 32677 raft_consensus.cc:686] T 62ca502a089f4a37999d2252776b741b P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Becoming Leader. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Running, Role: LEADER
I20240503 08:13:04.435587 32677 consensus_queue.cc:237] T 62ca502a089f4a37999d2252776b741b P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:04.436156 28002 raft_consensus.cc:2454] T 62ca502a089f4a37999d2252776b741b P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:13:04.439584 27859 catalog_manager.cc:5484] T 62ca502a089f4a37999d2252776b741b P ce1496a47d084365ad786cd09e472a2c reported cstate change: term changed from 0 to 1, leader changed from <none> to ce1496a47d084365ad786cd09e472a2c (127.20.230.70). New cstate: current_term: 1 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:04.481355 27859 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:13:04.486754 28263 raft_consensus.cc:1261] T e52da272af784435ab2355d3fe2df27a P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:04.486965 28002 raft_consensus.cc:1261] T e52da272af784435ab2355d3fe2df27a P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:04.487300 32517 consensus_queue.cc:1035] T e52da272af784435ab2355d3fe2df27a P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:04.487607 32702 consensus_queue.cc:1035] T e52da272af784435ab2355d3fe2df27a P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:04.507802 27856 catalog_manager.cc:5576] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081) does not have the latest schema for tablet e52da272af784435ab2355d3fe2df27a (table rebalancer_dynamic_table [id=fb12df737cb54402a4cad2fde5a851f0]). Expected version 1 got 0
I20240503 08:13:04.509407 32748 tablet.cc:1722] T e52da272af784435ab2355d3fe2df27a P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:13:04.510046 32748 tablet.cc:1620] T e52da272af784435ab2355d3fe2df27a P 6d1d7de19459435fbde1999e3dfd226b: MemRowSet was empty: no flush needed.
I20240503 08:13:04.513242 27856 catalog_manager.cc:5576] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919) does not have the latest schema for tablet e52da272af784435ab2355d3fe2df27a (table rebalancer_dynamic_table [id=fb12df737cb54402a4cad2fde5a851f0]). Expected version 1 got 0
I20240503 08:13:04.517145 32750 tablet.cc:1722] T e52da272af784435ab2355d3fe2df27a P 833582ff68c94269be364ebd187a75db: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:13:04.517823 32750 tablet.cc:1620] T e52da272af784435ab2355d3fe2df27a P 833582ff68c94269be364ebd187a75db: MemRowSet was empty: no flush needed.
I20240503 08:13:04.522218 32749 tablet.cc:1722] T e52da272af784435ab2355d3fe2df27a P aae860d7ae2b4002bf7794c278f9cc56: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:13:04.522778 32749 tablet.cc:1620] T e52da272af784435ab2355d3fe2df27a P aae860d7ae2b4002bf7794c278f9cc56: MemRowSet was empty: no flush needed.
I20240503 08:13:04.522758 27856 catalog_manager.cc:5576] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081) does not have the latest schema for tablet e52da272af784435ab2355d3fe2df27a (table rebalancer_dynamic_table [id=fb12df737cb54402a4cad2fde5a851f0]). Expected version 1 got 0
I20240503 08:13:04.531703 27859 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=fb12df737cb54402a4cad2fde5a851f0] alter complete (version 1)
I20240503 08:13:04.532400 27862 catalog_manager.cc:5576] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919) does not have the latest schema for tablet e52da272af784435ab2355d3fe2df27a (table rebalancer_dynamic_table [id=fb12df737cb54402a4cad2fde5a851f0]). Expected version 1 got 0
I20240503 08:13:04.533277 32704 raft_consensus.cc:482] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:04.533643 32704 raft_consensus.cc:504] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:04.534581 32704 leader_election.cc:288] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:13:04.535133 28390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e8dd968fc6c4416ae57cfcf7d8b4693" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:13:04.535132 28263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e8dd968fc6c4416ae57cfcf7d8b4693" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:13:04.535333 28390 raft_consensus.cc:2454] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 0.
I20240503 08:13:04.535346 28263 raft_consensus.cc:2454] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 0.
I20240503 08:13:04.535789 30962 leader_election.cc:302] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:13:04.536038 32704 raft_consensus.cc:2781] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:04.536151 32704 raft_consensus.cc:482] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:04.536235 32704 raft_consensus.cc:3037] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:04.543141 32704 raft_consensus.cc:504] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:04.544173 28390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e8dd968fc6c4416ae57cfcf7d8b4693" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:13:04.544342 28390 raft_consensus.cc:3037] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:04.544685 28263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e8dd968fc6c4416ae57cfcf7d8b4693" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:13:04.544901 28263 raft_consensus.cc:3037] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:04.547093 28390 raft_consensus.cc:2454] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 1.
I20240503 08:13:04.547425 30962 leader_election.cc:302] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:13:04.543912 32704 leader_election.cc:288] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:13:04.547549 28263 raft_consensus.cc:2454] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 1.
I20240503 08:13:04.547705 32743 raft_consensus.cc:2781] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:04.547828 32743 raft_consensus.cc:686] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 LEADER]: Becoming Leader. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Running, Role: LEADER
I20240503 08:13:04.548094 32743 consensus_queue.cc:237] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:04.551604 27859 catalog_manager.cc:5484] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P ced21ca983b749d0b4d4f6a0e2426bf8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71). New cstate: current_term: 1 leader_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:13:04.642204 27859 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:13:04.651429 32748 tablet.cc:1722] T e52da272af784435ab2355d3fe2df27a P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:13:04.651937 32748 tablet.cc:1620] T e52da272af784435ab2355d3fe2df27a P 6d1d7de19459435fbde1999e3dfd226b: MemRowSet was empty: no flush needed.
I20240503 08:13:04.652930 32750 tablet.cc:1722] T e52da272af784435ab2355d3fe2df27a P 833582ff68c94269be364ebd187a75db: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:13:04.653357 32750 tablet.cc:1620] T e52da272af784435ab2355d3fe2df27a P 833582ff68c94269be364ebd187a75db: MemRowSet was empty: no flush needed.
I20240503 08:13:04.654491 32611 raft_consensus.cc:482] T e0b904461c1c44ed93e1a227d93498eb P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:04.654677 32611 raft_consensus.cc:504] T e0b904461c1c44ed93e1a227d93498eb P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:04.656261 32611 leader_election.cc:288] T e0b904461c1c44ed93e1a227d93498eb P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:13:04.656569 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0b904461c1c44ed93e1a227d93498eb" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:13:04.656800 30761 raft_consensus.cc:2454] T e0b904461c1c44ed93e1a227d93498eb P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:13:04.657276 27938 leader_election.cc:302] T e0b904461c1c44ed93e1a227d93498eb P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, 833582ff68c94269be364ebd187a75db; no voters:
I20240503 08:13:04.657552 32611 raft_consensus.cc:2781] T e0b904461c1c44ed93e1a227d93498eb P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:04.657665 32611 raft_consensus.cc:482] T e0b904461c1c44ed93e1a227d93498eb P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:04.657786 32611 raft_consensus.cc:3037] T e0b904461c1c44ed93e1a227d93498eb P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:04.659265 28261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0b904461c1c44ed93e1a227d93498eb" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:13:04.659508 28261 raft_consensus.cc:2454] T e0b904461c1c44ed93e1a227d93498eb P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:13:04.660315 32611 raft_consensus.cc:504] T e0b904461c1c44ed93e1a227d93498eb P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:04.660913 32611 leader_election.cc:288] T e0b904461c1c44ed93e1a227d93498eb P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Requested vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:13:04.661784 28261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0b904461c1c44ed93e1a227d93498eb" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:13:04.661972 28261 raft_consensus.cc:3037] T e0b904461c1c44ed93e1a227d93498eb P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:04.662516 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0b904461c1c44ed93e1a227d93498eb" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:13:04.662712 30761 raft_consensus.cc:3037] T e0b904461c1c44ed93e1a227d93498eb P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:04.664495 28261 raft_consensus.cc:2454] T e0b904461c1c44ed93e1a227d93498eb P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:13:04.665321 30761 raft_consensus.cc:2454] T e0b904461c1c44ed93e1a227d93498eb P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:13:04.667044 27938 leader_election.cc:302] T e0b904461c1c44ed93e1a227d93498eb P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, 833582ff68c94269be364ebd187a75db; no voters:
I20240503 08:13:04.668392 32611 raft_consensus.cc:2781] T e0b904461c1c44ed93e1a227d93498eb P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:04.668571 32611 raft_consensus.cc:686] T e0b904461c1c44ed93e1a227d93498eb P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Becoming Leader. State: Replica: 833582ff68c94269be364ebd187a75db, State: Running, Role: LEADER
I20240503 08:13:04.668843 32611 consensus_queue.cc:237] T e0b904461c1c44ed93e1a227d93498eb P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:04.671872 32749 tablet.cc:1722] T e52da272af784435ab2355d3fe2df27a P aae860d7ae2b4002bf7794c278f9cc56: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:13:04.673413 27859 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=fb12df737cb54402a4cad2fde5a851f0] alter complete (version 2)
I20240503 08:13:04.673642 32749 tablet.cc:1620] T e52da272af784435ab2355d3fe2df27a P aae860d7ae2b4002bf7794c278f9cc56: MemRowSet was empty: no flush needed.
I20240503 08:13:04.687351 27859 catalog_manager.cc:5484] T e0b904461c1c44ed93e1a227d93498eb P 833582ff68c94269be364ebd187a75db reported cstate change: term changed from 0 to 1, leader changed from <none> to 833582ff68c94269be364ebd187a75db (127.20.230.65). New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:04.784737 27859 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:13:04.791965 32748 tablet.cc:1722] T e52da272af784435ab2355d3fe2df27a P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:13:04.792598 32749 tablet.cc:1722] T e52da272af784435ab2355d3fe2df27a P aae860d7ae2b4002bf7794c278f9cc56: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:13:04.798861 32750 tablet.cc:1722] T e52da272af784435ab2355d3fe2df27a P 833582ff68c94269be364ebd187a75db: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:13:04.801440 27859 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table_new_name [id=fb12df737cb54402a4cad2fde5a851f0] alter complete (version 3)
I20240503 08:13:04.817462 32706 consensus_queue.cc:1035] T 62ca502a089f4a37999d2252776b741b P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:04.822758 32706 consensus_queue.cc:1035] T 62ca502a089f4a37999d2252776b741b P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:04.902931 27859 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:13:04.903218 27859 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:13:04.906605 27859 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet e52da272af784435ab2355d3fe2df27a
I20240503 08:13:04.907440 28242 tablet_service.cc:1514] Processing DeleteTablet for tablet e52da272af784435ab2355d3fe2df27a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:04 UTC) from {username='slave'} at 127.0.0.1:41960
I20240503 08:13:04.907670 27981 tablet_service.cc:1514] Processing DeleteTablet for tablet e52da272af784435ab2355d3fe2df27a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:04 UTC) from {username='slave'} at 127.0.0.1:55610
I20240503 08:13:04.907842 30741 tablet_service.cc:1514] Processing DeleteTablet for tablet e52da272af784435ab2355d3fe2df27a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:04 UTC) from {username='slave'} at 127.0.0.1:46794
I20240503 08:13:04.918006 32761 tablet_replica.cc:330] T e52da272af784435ab2355d3fe2df27a P 6d1d7de19459435fbde1999e3dfd226b: stopping tablet replica
I20240503 08:13:04.918133 32760 tablet_replica.cc:330] T e52da272af784435ab2355d3fe2df27a P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:13:04.918289 32761 raft_consensus.cc:2229] T e52da272af784435ab2355d3fe2df27a P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:13:04.918387 32760 raft_consensus.cc:2229] T e52da272af784435ab2355d3fe2df27a P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:04.918570 32759 tablet_replica.cc:330] T e52da272af784435ab2355d3fe2df27a P aae860d7ae2b4002bf7794c278f9cc56: stopping tablet replica
I20240503 08:13:04.918651 32760 raft_consensus.cc:2258] T e52da272af784435ab2355d3fe2df27a P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:04.918684 32761 raft_consensus.cc:2258] T e52da272af784435ab2355d3fe2df27a P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:04.918807 32759 raft_consensus.cc:2229] T e52da272af784435ab2355d3fe2df27a P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:04.919039 32759 raft_consensus.cc:2258] T e52da272af784435ab2355d3fe2df27a P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:04.919636 32761 ts_tablet_manager.cc:1875] T e52da272af784435ab2355d3fe2df27a P 6d1d7de19459435fbde1999e3dfd226b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:04.919811 32759 ts_tablet_manager.cc:1875] T e52da272af784435ab2355d3fe2df27a P aae860d7ae2b4002bf7794c278f9cc56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:04.919863 32760 ts_tablet_manager.cc:1875] T e52da272af784435ab2355d3fe2df27a P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:04.924240 32760 ts_tablet_manager.cc:1888] T e52da272af784435ab2355d3fe2df27a P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:13:04.924430 32760 log.cc:1196] T e52da272af784435ab2355d3fe2df27a P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/e52da272af784435ab2355d3fe2df27a
I20240503 08:13:04.924913 32760 ts_tablet_manager.cc:1909] T e52da272af784435ab2355d3fe2df27a P 833582ff68c94269be364ebd187a75db: Deleting consensus metadata
I20240503 08:13:04.926211 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet e52da272af784435ab2355d3fe2df27a (table rebalancer_dynamic_table_new_name [id=fb12df737cb54402a4cad2fde5a851f0]) successfully deleted
I20240503 08:13:04.926710 32759 ts_tablet_manager.cc:1888] T e52da272af784435ab2355d3fe2df27a P aae860d7ae2b4002bf7794c278f9cc56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:13:04.926909 32759 log.cc:1196] T e52da272af784435ab2355d3fe2df27a P aae860d7ae2b4002bf7794c278f9cc56: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal/wals/e52da272af784435ab2355d3fe2df27a
I20240503 08:13:04.927428 32759 ts_tablet_manager.cc:1909] T e52da272af784435ab2355d3fe2df27a P aae860d7ae2b4002bf7794c278f9cc56: Deleting consensus metadata
I20240503 08:13:04.928138 32761 ts_tablet_manager.cc:1888] T e52da272af784435ab2355d3fe2df27a P 6d1d7de19459435fbde1999e3dfd226b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:13:04.928304 32761 log.cc:1196] T e52da272af784435ab2355d3fe2df27a P 6d1d7de19459435fbde1999e3dfd226b: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal/wals/e52da272af784435ab2355d3fe2df27a
I20240503 08:13:04.928983 27847 catalog_manager.cc:4830] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): tablet e52da272af784435ab2355d3fe2df27a (table rebalancer_dynamic_table_new_name [id=fb12df737cb54402a4cad2fde5a851f0]) successfully deleted
I20240503 08:13:04.929546 32761 ts_tablet_manager.cc:1909] T e52da272af784435ab2355d3fe2df27a P 6d1d7de19459435fbde1999e3dfd226b: Deleting consensus metadata
I20240503 08:13:04.930641 27848 catalog_manager.cc:4830] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): tablet e52da272af784435ab2355d3fe2df27a (table rebalancer_dynamic_table_new_name [id=fb12df737cb54402a4cad2fde5a851f0]) successfully deleted
I20240503 08:13:05.009464 27859 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:13:05.016705 30741 tablet_service.cc:1467] Processing CreateTablet for tablet 4dc36c1ae8294ed5af4a240413bf213a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dfaebba7e5fc482ca5a5a7f70188e755]), partition=
I20240503 08:13:05.017128 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dc36c1ae8294ed5af4a240413bf213a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:05.017400 28111 tablet_service.cc:1467] Processing CreateTablet for tablet 4dc36c1ae8294ed5af4a240413bf213a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dfaebba7e5fc482ca5a5a7f70188e755]), partition=
I20240503 08:13:05.017683 28111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dc36c1ae8294ed5af4a240413bf213a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:05.022351 28373 tablet_service.cc:1467] Processing CreateTablet for tablet 4dc36c1ae8294ed5af4a240413bf213a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dfaebba7e5fc482ca5a5a7f70188e755]), partition=
I20240503 08:13:05.022698 28373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dc36c1ae8294ed5af4a240413bf213a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:05.024423 32764 tablet_bootstrap.cc:492] T 4dc36c1ae8294ed5af4a240413bf213a P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:13:05.026199 32764 tablet_bootstrap.cc:654] T 4dc36c1ae8294ed5af4a240413bf213a P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:05.029568 32764 tablet_bootstrap.cc:492] T 4dc36c1ae8294ed5af4a240413bf213a P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:13:05.029703 32764 ts_tablet_manager.cc:1367] T 4dc36c1ae8294ed5af4a240413bf213a P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 08:13:05.029919 32763 tablet_bootstrap.cc:492] T 4dc36c1ae8294ed5af4a240413bf213a P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:13:05.030431 32764 raft_consensus.cc:348] T 4dc36c1ae8294ed5af4a240413bf213a P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:05.030726 32766 tablet_bootstrap.cc:492] T 4dc36c1ae8294ed5af4a240413bf213a P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:13:05.030731 32764 raft_consensus.cc:374] T 4dc36c1ae8294ed5af4a240413bf213a P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:05.030829 32764 raft_consensus.cc:729] T 4dc36c1ae8294ed5af4a240413bf213a P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:13:05.031138 32764 consensus_queue.cc:260] T 4dc36c1ae8294ed5af4a240413bf213a P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:05.031713 32764 ts_tablet_manager.cc:1398] T 4dc36c1ae8294ed5af4a240413bf213a P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:05.032356 32766 tablet_bootstrap.cc:654] T 4dc36c1ae8294ed5af4a240413bf213a P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:05.033429 32763 tablet_bootstrap.cc:654] T 4dc36c1ae8294ed5af4a240413bf213a P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:05.034260 32766 tablet_bootstrap.cc:492] T 4dc36c1ae8294ed5af4a240413bf213a P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:13:05.034431 32766 ts_tablet_manager.cc:1367] T 4dc36c1ae8294ed5af4a240413bf213a P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:13:05.035214 32766 raft_consensus.cc:348] T 4dc36c1ae8294ed5af4a240413bf213a P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:05.035499 32766 raft_consensus.cc:374] T 4dc36c1ae8294ed5af4a240413bf213a P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:05.035825 32766 raft_consensus.cc:729] T 4dc36c1ae8294ed5af4a240413bf213a P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:13:05.036090 32766 consensus_queue.cc:260] T 4dc36c1ae8294ed5af4a240413bf213a P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:05.037168 32766 ts_tablet_manager.cc:1398] T 4dc36c1ae8294ed5af4a240413bf213a P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:13:05.046387 32763 tablet_bootstrap.cc:492] T 4dc36c1ae8294ed5af4a240413bf213a P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:13:05.046705 32763 ts_tablet_manager.cc:1367] T 4dc36c1ae8294ed5af4a240413bf213a P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.005s
I20240503 08:13:05.047997 32763 raft_consensus.cc:348] T 4dc36c1ae8294ed5af4a240413bf213a P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:05.048174 32763 raft_consensus.cc:374] T 4dc36c1ae8294ed5af4a240413bf213a P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:05.048261 32763 raft_consensus.cc:729] T 4dc36c1ae8294ed5af4a240413bf213a P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:13:05.048518 32763 consensus_queue.cc:260] T 4dc36c1ae8294ed5af4a240413bf213a P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:05.049044 32763 ts_tablet_manager.cc:1398] T 4dc36c1ae8294ed5af4a240413bf213a P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:13:05.103296 32611 consensus_queue.cc:1035] T e0b904461c1c44ed93e1a227d93498eb P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:05.107697 32611 consensus_queue.cc:1035] T e0b904461c1c44ed93e1a227d93498eb P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:05.116987 32743 consensus_queue.cc:1035] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:05.121512 32743 consensus_queue.cc:1035] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:05.195561 32762 raft_consensus.cc:482] T 4dc36c1ae8294ed5af4a240413bf213a P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:05.195704 32762 raft_consensus.cc:504] T 4dc36c1ae8294ed5af4a240413bf213a P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:05.196430 32762 leader_election.cc:288] T 4dc36c1ae8294ed5af4a240413bf213a P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:13:05.196776 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dc36c1ae8294ed5af4a240413bf213a" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:13:05.196843 28389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dc36c1ae8294ed5af4a240413bf213a" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:13:05.196965 30761 raft_consensus.cc:2454] T 4dc36c1ae8294ed5af4a240413bf213a P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:13:05.197062 28389 raft_consensus.cc:2454] T 4dc36c1ae8294ed5af4a240413bf213a P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:13:05.197293 28068 leader_election.cc:302] T 4dc36c1ae8294ed5af4a240413bf213a P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 6d1d7de19459435fbde1999e3dfd226b; no voters:
I20240503 08:13:05.197571 32762 raft_consensus.cc:2781] T 4dc36c1ae8294ed5af4a240413bf213a P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:05.197685 32762 raft_consensus.cc:482] T 4dc36c1ae8294ed5af4a240413bf213a P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:05.197754 32762 raft_consensus.cc:3037] T 4dc36c1ae8294ed5af4a240413bf213a P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:05.199636 32762 raft_consensus.cc:504] T 4dc36c1ae8294ed5af4a240413bf213a P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:05.200130 32762 leader_election.cc:288] T 4dc36c1ae8294ed5af4a240413bf213a P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Requested vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:13:05.200482 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dc36c1ae8294ed5af4a240413bf213a" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:13:05.200624 30761 raft_consensus.cc:3037] T 4dc36c1ae8294ed5af4a240413bf213a P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:05.202813 30761 raft_consensus.cc:2454] T 4dc36c1ae8294ed5af4a240413bf213a P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:13:05.203166 28068 leader_election.cc:302] T 4dc36c1ae8294ed5af4a240413bf213a P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 6d1d7de19459435fbde1999e3dfd226b; no voters:
I20240503 08:13:05.203459 32762 raft_consensus.cc:2781] T 4dc36c1ae8294ed5af4a240413bf213a P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:05.203704 32762 raft_consensus.cc:686] T 4dc36c1ae8294ed5af4a240413bf213a P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Becoming Leader. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Running, Role: LEADER
I20240503 08:13:05.203974 32762 consensus_queue.cc:237] T 4dc36c1ae8294ed5af4a240413bf213a P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:05.204449 28389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dc36c1ae8294ed5af4a240413bf213a" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:13:05.204629 28389 raft_consensus.cc:3037] T 4dc36c1ae8294ed5af4a240413bf213a P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:05.207207 28389 raft_consensus.cc:2454] T 4dc36c1ae8294ed5af4a240413bf213a P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:13:05.208287 27859 catalog_manager.cc:5484] T 4dc36c1ae8294ed5af4a240413bf213a P 1b411c6bb25549368041773d7b710d48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b411c6bb25549368041773d7b710d48 (127.20.230.66). New cstate: current_term: 1 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:05.255650 27859 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_extra_table_56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:13:05.264551 31006 tablet_service.cc:1467] Processing CreateTablet for tablet 2401f8f101f94189aa3c0582f08cef88 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=0d91b0f1ea854f178cc99c7e43f0195f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:05.264973 31006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2401f8f101f94189aa3c0582f08cef88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:05.264968 31141 tablet_service.cc:1467] Processing CreateTablet for tablet 2401f8f101f94189aa3c0582f08cef88 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=0d91b0f1ea854f178cc99c7e43f0195f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:05.265390 31141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2401f8f101f94189aa3c0582f08cef88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:05.265456 28242 tablet_service.cc:1467] Processing CreateTablet for tablet d61ad2e11c4544bfbdef7df1e92d2d11 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=0d91b0f1ea854f178cc99c7e43f0195f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:05.265869 28242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d61ad2e11c4544bfbdef7df1e92d2d11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:05.270071 310 tablet_bootstrap.cc:492] T 2401f8f101f94189aa3c0582f08cef88 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:13:05.270687 30873 tablet_service.cc:1467] Processing CreateTablet for tablet 2401f8f101f94189aa3c0582f08cef88 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=0d91b0f1ea854f178cc99c7e43f0195f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:05.271086 30873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2401f8f101f94189aa3c0582f08cef88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:05.271317 30874 tablet_service.cc:1467] Processing CreateTablet for tablet d61ad2e11c4544bfbdef7df1e92d2d11 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=0d91b0f1ea854f178cc99c7e43f0195f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:05.271694 30874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d61ad2e11c4544bfbdef7df1e92d2d11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:05.273795 28373 tablet_service.cc:1467] Processing CreateTablet for tablet d61ad2e11c4544bfbdef7df1e92d2d11 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=0d91b0f1ea854f178cc99c7e43f0195f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:05.274199 28373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d61ad2e11c4544bfbdef7df1e92d2d11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:05.274464 28243 tablet_service.cc:1467] Processing CreateTablet for tablet 2dce5b2aaea045ccb7424cbbd44576c2 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=0d91b0f1ea854f178cc99c7e43f0195f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:05.274830 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dce5b2aaea045ccb7424cbbd44576c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:05.277956 28111 tablet_service.cc:1467] Processing CreateTablet for tablet 2dce5b2aaea045ccb7424cbbd44576c2 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=0d91b0f1ea854f178cc99c7e43f0195f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:05.278347 28111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dce5b2aaea045ccb7424cbbd44576c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:05.281724 313 tablet_bootstrap.cc:492] T d61ad2e11c4544bfbdef7df1e92d2d11 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:13:05.282562 31140 tablet_service.cc:1467] Processing CreateTablet for tablet 2dce5b2aaea045ccb7424cbbd44576c2 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=0d91b0f1ea854f178cc99c7e43f0195f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:05.282972 31140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dce5b2aaea045ccb7424cbbd44576c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:05.283524 313 tablet_bootstrap.cc:654] T d61ad2e11c4544bfbdef7df1e92d2d11 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:05.286813 32766 tablet_bootstrap.cc:492] T d61ad2e11c4544bfbdef7df1e92d2d11 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:13:05.288617 32766 tablet_bootstrap.cc:654] T d61ad2e11c4544bfbdef7df1e92d2d11 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:05.295176 32766 tablet_bootstrap.cc:492] T d61ad2e11c4544bfbdef7df1e92d2d11 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:13:05.295352 32766 ts_tablet_manager.cc:1367] T d61ad2e11c4544bfbdef7df1e92d2d11 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240503 08:13:05.296370 32766 raft_consensus.cc:348] T d61ad2e11c4544bfbdef7df1e92d2d11 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:05.296622 32766 raft_consensus.cc:374] T d61ad2e11c4544bfbdef7df1e92d2d11 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:05.296703 32766 raft_consensus.cc:729] T d61ad2e11c4544bfbdef7df1e92d2d11 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:13:05.296938 32766 consensus_queue.cc:260] T d61ad2e11c4544bfbdef7df1e92d2d11 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:05.297565 32766 ts_tablet_manager.cc:1398] T d61ad2e11c4544bfbdef7df1e92d2d11 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:05.300967 315 tablet_bootstrap.cc:492] T 2401f8f101f94189aa3c0582f08cef88 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:13:05.303057 312 tablet_bootstrap.cc:492] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:13:05.306097 312 tablet_bootstrap.cc:654] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:05.306689 315 tablet_bootstrap.cc:654] T 2401f8f101f94189aa3c0582f08cef88 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:05.310856 32764 tablet_bootstrap.cc:492] T 2dce5b2aaea045ccb7424cbbd44576c2 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:13:05.310914 315 tablet_bootstrap.cc:492] T 2401f8f101f94189aa3c0582f08cef88 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:13:05.311023 315 ts_tablet_manager.cc:1367] T 2401f8f101f94189aa3c0582f08cef88 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240503 08:13:05.311739 315 raft_consensus.cc:348] T 2401f8f101f94189aa3c0582f08cef88 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:05.311937 315 raft_consensus.cc:374] T 2401f8f101f94189aa3c0582f08cef88 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:05.312054 315 raft_consensus.cc:729] T 2401f8f101f94189aa3c0582f08cef88 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:13:05.312250 315 consensus_queue.cc:260] T 2401f8f101f94189aa3c0582f08cef88 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:05.312595 32764 tablet_bootstrap.cc:654] T 2dce5b2aaea045ccb7424cbbd44576c2 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:05.312741 315 ts_tablet_manager.cc:1398] T 2401f8f101f94189aa3c0582f08cef88 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:05.313073 315 tablet_bootstrap.cc:492] T 2dce5b2aaea045ccb7424cbbd44576c2 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:13:05.317481 315 tablet_bootstrap.cc:654] T 2dce5b2aaea045ccb7424cbbd44576c2 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:05.321537 312 tablet_bootstrap.cc:492] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:13:05.321364 310 tablet_bootstrap.cc:654] T 2401f8f101f94189aa3c0582f08cef88 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:05.321794 312 ts_tablet_manager.cc:1367] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.005s
I20240503 08:13:05.322650 312 raft_consensus.cc:348] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:05.322990 312 raft_consensus.cc:374] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:05.323124 312 raft_consensus.cc:729] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:13:05.323393 312 consensus_queue.cc:260] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:05.325425 312 ts_tablet_manager.cc:1398] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:13:05.325479 32764 tablet_bootstrap.cc:492] T 2dce5b2aaea045ccb7424cbbd44576c2 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:13:05.325625 27859 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:13:05.325750 312 tablet_bootstrap.cc:492] T d61ad2e11c4544bfbdef7df1e92d2d11 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:13:05.325691 32764 ts_tablet_manager.cc:1367] T 2dce5b2aaea045ccb7424cbbd44576c2 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.004s
I20240503 08:13:05.327981 32764 raft_consensus.cc:348] T 2dce5b2aaea045ccb7424cbbd44576c2 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:05.328250 32764 raft_consensus.cc:374] T 2dce5b2aaea045ccb7424cbbd44576c2 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:05.328385 32764 raft_consensus.cc:729] T 2dce5b2aaea045ccb7424cbbd44576c2 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:13:05.328675 32764 consensus_queue.cc:260] T 2dce5b2aaea045ccb7424cbbd44576c2 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:05.329241 32764 ts_tablet_manager.cc:1398] T 2dce5b2aaea045ccb7424cbbd44576c2 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:05.334807 32706 raft_consensus.cc:482] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:05.334962 32706 raft_consensus.cc:504] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:05.335793 32706 leader_election.cc:288] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:13:05.340031 31160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401f8f101f94189aa3c0582f08cef88" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:13:05.340219 31160 raft_consensus.cc:2454] T 2401f8f101f94189aa3c0582f08cef88 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:13:05.340350 31026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401f8f101f94189aa3c0582f08cef88" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:13:05.340648 30827 leader_election.cc:302] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:13:05.340946 32706 raft_consensus.cc:2781] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:05.341050 32706 raft_consensus.cc:482] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:05.341147 32706 raft_consensus.cc:3037] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
W20240503 08:13:05.341837 30828 leader_election.cc:341] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:13:05.343931 32706 raft_consensus.cc:504] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:05.344990 31026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401f8f101f94189aa3c0582f08cef88" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
W20240503 08:13:05.345728 30828 leader_election.cc:341] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:13:05.346465 310 tablet_bootstrap.cc:492] T 2401f8f101f94189aa3c0582f08cef88 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:13:05.346477 31160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401f8f101f94189aa3c0582f08cef88" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:13:05.346668 310 ts_tablet_manager.cc:1367] T 2401f8f101f94189aa3c0582f08cef88 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.077s user 0.007s sys 0.000s
I20240503 08:13:05.346673 31160 raft_consensus.cc:3037] T 2401f8f101f94189aa3c0582f08cef88 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:05.347470 310 raft_consensus.cc:348] T 2401f8f101f94189aa3c0582f08cef88 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:05.347664 310 raft_consensus.cc:374] T 2401f8f101f94189aa3c0582f08cef88 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:05.347739 310 raft_consensus.cc:729] T 2401f8f101f94189aa3c0582f08cef88 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:13:05.347934 310 consensus_queue.cc:260] T 2401f8f101f94189aa3c0582f08cef88 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:05.348470 310 ts_tablet_manager.cc:1398] T 2401f8f101f94189aa3c0582f08cef88 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:13:05.349381 31160 raft_consensus.cc:2454] T 2401f8f101f94189aa3c0582f08cef88 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:13:05.349489 313 tablet_bootstrap.cc:492] T d61ad2e11c4544bfbdef7df1e92d2d11 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:13:05.349623 313 ts_tablet_manager.cc:1367] T d61ad2e11c4544bfbdef7df1e92d2d11 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.068s user 0.017s sys 0.023s
I20240503 08:13:05.349831 30827 leader_election.cc:302] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b00f244c69df401997ea99d492218a98, ce1496a47d084365ad786cd09e472a2c; no voters: ced21ca983b749d0b4d4f6a0e2426bf8
I20240503 08:13:05.350173 32677 raft_consensus.cc:2781] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:05.350423 313 raft_consensus.cc:348] T d61ad2e11c4544bfbdef7df1e92d2d11 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:05.350636 313 raft_consensus.cc:374] T d61ad2e11c4544bfbdef7df1e92d2d11 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:05.350713 313 raft_consensus.cc:729] T d61ad2e11c4544bfbdef7df1e92d2d11 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:13:05.350941 313 consensus_queue.cc:260] T d61ad2e11c4544bfbdef7df1e92d2d11 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:05.352653 28389 raft_consensus.cc:1261] T 4dc36c1ae8294ed5af4a240413bf213a P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:05.353317 30761 raft_consensus.cc:1261] T 4dc36c1ae8294ed5af4a240413bf213a P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:05.353746 313 ts_tablet_manager.cc:1398] T d61ad2e11c4544bfbdef7df1e92d2d11 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 08:13:05.354024 313 tablet_bootstrap.cc:492] T 2dce5b2aaea045ccb7424cbbd44576c2 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:13:05.355680 313 tablet_bootstrap.cc:654] T 2dce5b2aaea045ccb7424cbbd44576c2 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:05.358875 32762 consensus_queue.cc:1035] T 4dc36c1ae8294ed5af4a240413bf213a P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:05.359316 32762 consensus_queue.cc:1035] T 4dc36c1ae8294ed5af4a240413bf213a P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:05.370771 307 mvcc.cc:205] Tried to move back new op lower bound from 7023509443977416704 to 7023509443397677056. Current Snapshot: MvccSnapshot[applied={T|T < 7023509443977416704}]
I20240503 08:13:05.372808 27856 catalog_manager.cc:5576] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211) does not have the latest schema for tablet 4dc36c1ae8294ed5af4a240413bf213a (table rebalancer_dynamic_table [id=dfaebba7e5fc482ca5a5a7f70188e755]). Expected version 1 got 0
I20240503 08:13:05.384567 32706 leader_election.cc:288] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Requested vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:13:05.391474 313 tablet_bootstrap.cc:492] T 2dce5b2aaea045ccb7424cbbd44576c2 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:13:05.391716 313 ts_tablet_manager.cc:1367] T 2dce5b2aaea045ccb7424cbbd44576c2 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.038s user 0.002s sys 0.008s
I20240503 08:13:05.392618 313 raft_consensus.cc:348] T 2dce5b2aaea045ccb7424cbbd44576c2 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:05.392951 313 raft_consensus.cc:374] T 2dce5b2aaea045ccb7424cbbd44576c2 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:05.393110 313 raft_consensus.cc:729] T 2dce5b2aaea045ccb7424cbbd44576c2 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:13:05.394837 313 consensus_queue.cc:260] T 2dce5b2aaea045ccb7424cbbd44576c2 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:05.395676 313 ts_tablet_manager.cc:1398] T 2dce5b2aaea045ccb7424cbbd44576c2 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 08:13:05.327353 312 tablet_bootstrap.cc:654] T d61ad2e11c4544bfbdef7df1e92d2d11 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:05.410820 312 tablet_bootstrap.cc:492] T d61ad2e11c4544bfbdef7df1e92d2d11 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:13:05.410995 312 ts_tablet_manager.cc:1367] T d61ad2e11c4544bfbdef7df1e92d2d11 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.085s user 0.001s sys 0.003s
I20240503 08:13:05.411790 312 raft_consensus.cc:348] T d61ad2e11c4544bfbdef7df1e92d2d11 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:05.412027 312 raft_consensus.cc:374] T d61ad2e11c4544bfbdef7df1e92d2d11 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:05.412104 312 raft_consensus.cc:729] T d61ad2e11c4544bfbdef7df1e92d2d11 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:13:05.412366 312 consensus_queue.cc:260] T d61ad2e11c4544bfbdef7df1e92d2d11 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:05.413067 312 ts_tablet_manager.cc:1398] T d61ad2e11c4544bfbdef7df1e92d2d11 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:13:05.413219 331 raft_consensus.cc:482] T 2dce5b2aaea045ccb7424cbbd44576c2 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:05.413360 331 raft_consensus.cc:504] T 2dce5b2aaea045ccb7424cbbd44576c2 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:05.414450 31160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dce5b2aaea045ccb7424cbbd44576c2" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:13:05.414918 329 tablet.cc:1722] T 4dc36c1ae8294ed5af4a240413bf213a P d09a11a03989421b9f28019dc1c14cc2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240503 08:13:05.415042 28066 leader_election.cc:341] T 2dce5b2aaea045ccb7424cbbd44576c2 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:13:05.415694 329 tablet.cc:1620] T 4dc36c1ae8294ed5af4a240413bf213a P d09a11a03989421b9f28019dc1c14cc2: MemRowSet was empty: no flush needed.
I20240503 08:13:05.418238 331 leader_election.cc:288] T 2dce5b2aaea045ccb7424cbbd44576c2 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:13:05.419015 28261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dce5b2aaea045ccb7424cbbd44576c2" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:13:05.419217 28261 raft_consensus.cc:2454] T 2dce5b2aaea045ccb7424cbbd44576c2 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:13:05.419692 28067 leader_election.cc:302] T 2dce5b2aaea045ccb7424cbbd44576c2 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56; no voters: b00f244c69df401997ea99d492218a98
I20240503 08:13:05.419978 331 raft_consensus.cc:2781] T 2dce5b2aaea045ccb7424cbbd44576c2 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:05.420145 331 raft_consensus.cc:482] T 2dce5b2aaea045ccb7424cbbd44576c2 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:05.420284 331 raft_consensus.cc:3037] T 2dce5b2aaea045ccb7424cbbd44576c2 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:05.421099 27862 catalog_manager.cc:5576] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211) does not have the latest schema for tablet 4dc36c1ae8294ed5af4a240413bf213a (table rebalancer_dynamic_table [id=dfaebba7e5fc482ca5a5a7f70188e755]). Expected version 1 got 0
I20240503 08:13:05.422509 328 tablet.cc:1722] T 4dc36c1ae8294ed5af4a240413bf213a P 1b411c6bb25549368041773d7b710d48: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:13:05.423209 328 tablet.cc:1620] T 4dc36c1ae8294ed5af4a240413bf213a P 1b411c6bb25549368041773d7b710d48: MemRowSet was empty: no flush needed.
I20240503 08:13:05.424465 331 raft_consensus.cc:504] T 2dce5b2aaea045ccb7424cbbd44576c2 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:05.425520 31160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dce5b2aaea045ccb7424cbbd44576c2" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
W20240503 08:13:05.426250 28066 leader_election.cc:341] T 2dce5b2aaea045ccb7424cbbd44576c2 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:13:05.427003 28261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dce5b2aaea045ccb7424cbbd44576c2" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:13:05.427209 28261 raft_consensus.cc:3037] T 2dce5b2aaea045ccb7424cbbd44576c2 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
W20240503 08:13:05.427172 28068 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.69:34725, user_credentials={real_user=slave}} blocked reactor thread for 70680.2us
I20240503 08:13:05.430780 28261 raft_consensus.cc:2454] T 2dce5b2aaea045ccb7424cbbd44576c2 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:13:05.431478 28067 leader_election.cc:302] T 2dce5b2aaea045ccb7424cbbd44576c2 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56; no voters: b00f244c69df401997ea99d492218a98
I20240503 08:13:05.431803 331 leader_election.cc:288] T 2dce5b2aaea045ccb7424cbbd44576c2 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Requested vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:13:05.431818 32762 raft_consensus.cc:2781] T 2dce5b2aaea045ccb7424cbbd44576c2 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:05.432125 32762 raft_consensus.cc:686] T 2dce5b2aaea045ccb7424cbbd44576c2 P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Becoming Leader. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Running, Role: LEADER
I20240503 08:13:05.432399 32762 consensus_queue.cc:237] T 2dce5b2aaea045ccb7424cbbd44576c2 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:05.435917 27859 catalog_manager.cc:5484] T 2dce5b2aaea045ccb7424cbbd44576c2 P 1b411c6bb25549368041773d7b710d48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b411c6bb25549368041773d7b710d48 (127.20.230.66). New cstate: current_term: 1 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:05.443137 335 tablet.cc:1722] T 4dc36c1ae8294ed5af4a240413bf213a P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:13:05.443847 335 tablet.cc:1620] T 4dc36c1ae8294ed5af4a240413bf213a P 6d1d7de19459435fbde1999e3dfd226b: MemRowSet was empty: no flush needed.
I20240503 08:13:05.446084 27859 catalog_manager.cc:5576] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725) does not have the latest schema for tablet 4dc36c1ae8294ed5af4a240413bf213a (table rebalancer_dynamic_table [id=dfaebba7e5fc482ca5a5a7f70188e755]). Expected version 1 got 0
I20240503 08:13:05.446943 315 tablet_bootstrap.cc:492] T 2dce5b2aaea045ccb7424cbbd44576c2 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:13:05.447141 315 ts_tablet_manager.cc:1367] T 2dce5b2aaea045ccb7424cbbd44576c2 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.134s user 0.021s sys 0.024s
I20240503 08:13:05.449338 315 raft_consensus.cc:348] T 2dce5b2aaea045ccb7424cbbd44576c2 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:05.449617 315 raft_consensus.cc:374] T 2dce5b2aaea045ccb7424cbbd44576c2 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:05.449734 315 raft_consensus.cc:729] T 2dce5b2aaea045ccb7424cbbd44576c2 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:13:05.450053 315 consensus_queue.cc:260] T 2dce5b2aaea045ccb7424cbbd44576c2 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:05.450762 315 ts_tablet_manager.cc:1398] T 2dce5b2aaea045ccb7424cbbd44576c2 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:13:05.453965 32677 raft_consensus.cc:686] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Becoming Leader. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Running, Role: LEADER
I20240503 08:13:05.455616 32677 consensus_queue.cc:237] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:05.457437 27856 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=dfaebba7e5fc482ca5a5a7f70188e755] alter complete (version 1)
I20240503 08:13:05.460397 27854 catalog_manager.cc:5484] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c reported cstate change: term changed from 0 to 1, leader changed from <none> to ce1496a47d084365ad786cd09e472a2c (127.20.230.70). New cstate: current_term: 1 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:13:05.517590 32677 raft_consensus.cc:482] T d61ad2e11c4544bfbdef7df1e92d2d11 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:05.517819 32677 raft_consensus.cc:504] T d61ad2e11c4544bfbdef7df1e92d2d11 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:05.518626 32677 leader_election.cc:288] T d61ad2e11c4544bfbdef7df1e92d2d11 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:13:05.518975 28261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61ad2e11c4544bfbdef7df1e92d2d11" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:13:05.519205 28261 raft_consensus.cc:2454] T d61ad2e11c4544bfbdef7df1e92d2d11 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:13:05.519261 28390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61ad2e11c4544bfbdef7df1e92d2d11" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:13:05.519492 28390 raft_consensus.cc:2454] T d61ad2e11c4544bfbdef7df1e92d2d11 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:13:05.519582 30828 leader_election.cc:302] T d61ad2e11c4544bfbdef7df1e92d2d11 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:13:05.519856 32677 raft_consensus.cc:2781] T d61ad2e11c4544bfbdef7df1e92d2d11 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:05.519979 32677 raft_consensus.cc:482] T d61ad2e11c4544bfbdef7df1e92d2d11 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:05.520071 32677 raft_consensus.cc:3037] T d61ad2e11c4544bfbdef7df1e92d2d11 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:05.522249 32677 raft_consensus.cc:504] T d61ad2e11c4544bfbdef7df1e92d2d11 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:05.522737 32677 leader_election.cc:288] T d61ad2e11c4544bfbdef7df1e92d2d11 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:13:05.523023 28261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61ad2e11c4544bfbdef7df1e92d2d11" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:13:05.523208 28261 raft_consensus.cc:3037] T d61ad2e11c4544bfbdef7df1e92d2d11 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:05.523725 28390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61ad2e11c4544bfbdef7df1e92d2d11" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:13:05.523907 28390 raft_consensus.cc:3037] T d61ad2e11c4544bfbdef7df1e92d2d11 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:05.525060 28261 raft_consensus.cc:2454] T d61ad2e11c4544bfbdef7df1e92d2d11 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:13:05.525436 30828 leader_election.cc:302] T d61ad2e11c4544bfbdef7df1e92d2d11 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:13:05.525667 32677 raft_consensus.cc:2781] T d61ad2e11c4544bfbdef7df1e92d2d11 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:05.525802 32677 raft_consensus.cc:686] T d61ad2e11c4544bfbdef7df1e92d2d11 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Becoming Leader. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Running, Role: LEADER
I20240503 08:13:05.526029 32677 consensus_queue.cc:237] T d61ad2e11c4544bfbdef7df1e92d2d11 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:05.526662 28390 raft_consensus.cc:2454] T d61ad2e11c4544bfbdef7df1e92d2d11 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:13:05.529542 27854 catalog_manager.cc:5484] T d61ad2e11c4544bfbdef7df1e92d2d11 P ce1496a47d084365ad786cd09e472a2c reported cstate change: term changed from 0 to 1, leader changed from <none> to ce1496a47d084365ad786cd09e472a2c (127.20.230.70). New cstate: current_term: 1 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:05.532910 32493 raft_consensus.cc:482] T 2dce5b2aaea045ccb7424cbbd44576c2 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:05.533077 32493 raft_consensus.cc:504] T 2dce5b2aaea045ccb7424cbbd44576c2 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:05.533634 32493 leader_election.cc:288] T 2dce5b2aaea045ccb7424cbbd44576c2 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:13:05.534060 28261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dce5b2aaea045ccb7424cbbd44576c2" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:13:05.534273 28261 raft_consensus.cc:2379] T 2dce5b2aaea045ccb7424cbbd44576c2 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b00f244c69df401997ea99d492218a98 in current term 1: Already voted for candidate 1b411c6bb25549368041773d7b710d48 in this term.
I20240503 08:13:05.534543 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dce5b2aaea045ccb7424cbbd44576c2" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:13:05.535099 31096 leader_election.cc:302] T 2dce5b2aaea045ccb7424cbbd44576c2 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b00f244c69df401997ea99d492218a98; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:13:05.535470 32493 raft_consensus.cc:3037] T 2dce5b2aaea045ccb7424cbbd44576c2 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:05.538165 32493 raft_consensus.cc:2726] T 2dce5b2aaea045ccb7424cbbd44576c2 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:13:05.575994 27854 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:13:05.584445 335 tablet.cc:1722] T 4dc36c1ae8294ed5af4a240413bf213a P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:13:05.584903 335 tablet.cc:1620] T 4dc36c1ae8294ed5af4a240413bf213a P 6d1d7de19459435fbde1999e3dfd226b: MemRowSet was empty: no flush needed.
I20240503 08:13:05.585959 329 tablet.cc:1722] T 4dc36c1ae8294ed5af4a240413bf213a P d09a11a03989421b9f28019dc1c14cc2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:13:05.586457 329 tablet.cc:1620] T 4dc36c1ae8294ed5af4a240413bf213a P d09a11a03989421b9f28019dc1c14cc2: MemRowSet was empty: no flush needed.
I20240503 08:13:05.588788 328 tablet.cc:1722] T 4dc36c1ae8294ed5af4a240413bf213a P 1b411c6bb25549368041773d7b710d48: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:13:05.589272 328 tablet.cc:1620] T 4dc36c1ae8294ed5af4a240413bf213a P 1b411c6bb25549368041773d7b710d48: MemRowSet was empty: no flush needed.
I20240503 08:13:05.596805 27856 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=dfaebba7e5fc482ca5a5a7f70188e755] alter complete (version 2)
I20240503 08:13:05.699391 27854 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:13:05.706362 328 tablet.cc:1722] T 4dc36c1ae8294ed5af4a240413bf213a P 1b411c6bb25549368041773d7b710d48: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:13:05.707497 329 tablet.cc:1722] T 4dc36c1ae8294ed5af4a240413bf213a P d09a11a03989421b9f28019dc1c14cc2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:13:05.716421 335 tablet.cc:1722] T 4dc36c1ae8294ed5af4a240413bf213a P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:13:05.716866 27854 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table_new_name [id=dfaebba7e5fc482ca5a5a7f70188e755] alter complete (version 3)
I20240503 08:13:05.818825 27854 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:13:05.819044 27854 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:13:05.821738 27854 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet 4dc36c1ae8294ed5af4a240413bf213a
I20240503 08:13:05.822525 30741 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dc36c1ae8294ed5af4a240413bf213a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:05 UTC) from {username='slave'} at 127.0.0.1:46794
I20240503 08:13:05.823091 342 tablet_replica.cc:330] T 4dc36c1ae8294ed5af4a240413bf213a P 6d1d7de19459435fbde1999e3dfd226b: stopping tablet replica
I20240503 08:13:05.823182 28111 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dc36c1ae8294ed5af4a240413bf213a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:05 UTC) from {username='slave'} at 127.0.0.1:57912
I20240503 08:13:05.823346 342 raft_consensus.cc:2229] T 4dc36c1ae8294ed5af4a240413bf213a P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:05.823701 28373 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dc36c1ae8294ed5af4a240413bf213a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:05 UTC) from {username='slave'} at 127.0.0.1:34786
I20240503 08:13:05.824026 343 tablet_replica.cc:330] T 4dc36c1ae8294ed5af4a240413bf213a P 1b411c6bb25549368041773d7b710d48: stopping tablet replica
I20240503 08:13:05.824118 342 raft_consensus.cc:2258] T 4dc36c1ae8294ed5af4a240413bf213a P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:05.824193 344 tablet_replica.cc:330] T 4dc36c1ae8294ed5af4a240413bf213a P d09a11a03989421b9f28019dc1c14cc2: stopping tablet replica
I20240503 08:13:05.824458 343 raft_consensus.cc:2229] T 4dc36c1ae8294ed5af4a240413bf213a P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:13:05.824659 344 raft_consensus.cc:2229] T 4dc36c1ae8294ed5af4a240413bf213a P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:05.824851 343 raft_consensus.cc:2258] T 4dc36c1ae8294ed5af4a240413bf213a P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:05.824916 344 raft_consensus.cc:2258] T 4dc36c1ae8294ed5af4a240413bf213a P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:05.825053 342 ts_tablet_manager.cc:1875] T 4dc36c1ae8294ed5af4a240413bf213a P 6d1d7de19459435fbde1999e3dfd226b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:05.825656 344 ts_tablet_manager.cc:1875] T 4dc36c1ae8294ed5af4a240413bf213a P d09a11a03989421b9f28019dc1c14cc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:05.825906 343 ts_tablet_manager.cc:1875] T 4dc36c1ae8294ed5af4a240413bf213a P 1b411c6bb25549368041773d7b710d48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:05.831019 342 ts_tablet_manager.cc:1888] T 4dc36c1ae8294ed5af4a240413bf213a P 6d1d7de19459435fbde1999e3dfd226b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:13:05.831159 342 log.cc:1196] T 4dc36c1ae8294ed5af4a240413bf213a P 6d1d7de19459435fbde1999e3dfd226b: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal/wals/4dc36c1ae8294ed5af4a240413bf213a
I20240503 08:13:05.831410 344 ts_tablet_manager.cc:1888] T 4dc36c1ae8294ed5af4a240413bf213a P d09a11a03989421b9f28019dc1c14cc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:13:05.831554 344 log.cc:1196] T 4dc36c1ae8294ed5af4a240413bf213a P d09a11a03989421b9f28019dc1c14cc2: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/wals/4dc36c1ae8294ed5af4a240413bf213a
I20240503 08:13:05.831614 342 ts_tablet_manager.cc:1909] T 4dc36c1ae8294ed5af4a240413bf213a P 6d1d7de19459435fbde1999e3dfd226b: Deleting consensus metadata
I20240503 08:13:05.832089 344 ts_tablet_manager.cc:1909] T 4dc36c1ae8294ed5af4a240413bf213a P d09a11a03989421b9f28019dc1c14cc2: Deleting consensus metadata
I20240503 08:13:05.831892 343 ts_tablet_manager.cc:1888] T 4dc36c1ae8294ed5af4a240413bf213a P 1b411c6bb25549368041773d7b710d48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:13:05.832258 343 log.cc:1196] T 4dc36c1ae8294ed5af4a240413bf213a P 1b411c6bb25549368041773d7b710d48: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/4dc36c1ae8294ed5af4a240413bf213a
I20240503 08:13:05.832581 27848 catalog_manager.cc:4830] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): tablet 4dc36c1ae8294ed5af4a240413bf213a (table rebalancer_dynamic_table_new_name [id=dfaebba7e5fc482ca5a5a7f70188e755]) successfully deleted
I20240503 08:13:05.832768 343 ts_tablet_manager.cc:1909] T 4dc36c1ae8294ed5af4a240413bf213a P 1b411c6bb25549368041773d7b710d48: Deleting consensus metadata
I20240503 08:13:05.833182 27849 catalog_manager.cc:4830] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): tablet 4dc36c1ae8294ed5af4a240413bf213a (table rebalancer_dynamic_table_new_name [id=dfaebba7e5fc482ca5a5a7f70188e755]) successfully deleted
I20240503 08:13:05.833938 27848 catalog_manager.cc:4830] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): tablet 4dc36c1ae8294ed5af4a240413bf213a (table rebalancer_dynamic_table_new_name [id=dfaebba7e5fc482ca5a5a7f70188e755]) successfully deleted
I20240503 08:13:05.838449 32743 raft_consensus.cc:482] T 2401f8f101f94189aa3c0582f08cef88 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:05.838614 32743 raft_consensus.cc:504] T 2401f8f101f94189aa3c0582f08cef88 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:05.839263 32743 leader_election.cc:288] T 2401f8f101f94189aa3c0582f08cef88 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:13:05.839627 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401f8f101f94189aa3c0582f08cef88" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:13:05.839686 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401f8f101f94189aa3c0582f08cef88" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:13:05.839910 31159 raft_consensus.cc:2379] T 2401f8f101f94189aa3c0582f08cef88 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ced21ca983b749d0b4d4f6a0e2426bf8 in current term 1: Already voted for candidate ce1496a47d084365ad786cd09e472a2c in this term.
I20240503 08:13:05.840252 30959 leader_election.cc:302] T 2401f8f101f94189aa3c0582f08cef88 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8; no voters: b00f244c69df401997ea99d492218a98, ce1496a47d084365ad786cd09e472a2c
I20240503 08:13:05.840545 32743 raft_consensus.cc:3037] T 2401f8f101f94189aa3c0582f08cef88 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:05.842386 32743 raft_consensus.cc:2726] T 2401f8f101f94189aa3c0582f08cef88 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:13:05.873883 324 consensus_queue.cc:1035] T 2dce5b2aaea045ccb7424cbbd44576c2 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:05.877661 324 consensus_queue.cc:1035] T 2dce5b2aaea045ccb7424cbbd44576c2 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:05.924804 27854 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:13:05.932307 28373 tablet_service.cc:1467] Processing CreateTablet for tablet 4f8fb9f623124e2a956a56a38754b0f3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=76e29da269bb4a6ca00083d58e880fd2]), partition=
I20240503 08:13:05.932330 27981 tablet_service.cc:1467] Processing CreateTablet for tablet 4f8fb9f623124e2a956a56a38754b0f3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=76e29da269bb4a6ca00083d58e880fd2]), partition=
I20240503 08:13:05.932631 31006 tablet_service.cc:1467] Processing CreateTablet for tablet 4f8fb9f623124e2a956a56a38754b0f3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=76e29da269bb4a6ca00083d58e880fd2]), partition=
I20240503 08:13:05.932685 28373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f8fb9f623124e2a956a56a38754b0f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:05.932751 27981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f8fb9f623124e2a956a56a38754b0f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:05.932955 31006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f8fb9f623124e2a956a56a38754b0f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:05.937924 349 tablet_bootstrap.cc:492] T 4f8fb9f623124e2a956a56a38754b0f3 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:13:05.939617 349 tablet_bootstrap.cc:654] T 4f8fb9f623124e2a956a56a38754b0f3 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:05.941332 349 tablet_bootstrap.cc:492] T 4f8fb9f623124e2a956a56a38754b0f3 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:13:05.941458 349 ts_tablet_manager.cc:1367] T 4f8fb9f623124e2a956a56a38754b0f3 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:13:05.942185 349 raft_consensus.cc:348] T 4f8fb9f623124e2a956a56a38754b0f3 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:05.942479 349 raft_consensus.cc:374] T 4f8fb9f623124e2a956a56a38754b0f3 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:05.942618 349 raft_consensus.cc:729] T 4f8fb9f623124e2a956a56a38754b0f3 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:13:05.942895 349 consensus_queue.cc:260] T 4f8fb9f623124e2a956a56a38754b0f3 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:05.945406 349 ts_tablet_manager.cc:1398] T 4f8fb9f623124e2a956a56a38754b0f3 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:13:05.949909 350 tablet_bootstrap.cc:492] T 4f8fb9f623124e2a956a56a38754b0f3 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:13:05.950201 352 tablet_bootstrap.cc:492] T 4f8fb9f623124e2a956a56a38754b0f3 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:13:05.951680 352 tablet_bootstrap.cc:654] T 4f8fb9f623124e2a956a56a38754b0f3 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:05.952756 350 tablet_bootstrap.cc:654] T 4f8fb9f623124e2a956a56a38754b0f3 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:05.955313 350 tablet_bootstrap.cc:492] T 4f8fb9f623124e2a956a56a38754b0f3 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:13:05.955435 350 ts_tablet_manager.cc:1367] T 4f8fb9f623124e2a956a56a38754b0f3 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:13:05.956123 350 raft_consensus.cc:348] T 4f8fb9f623124e2a956a56a38754b0f3 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:05.956300 350 raft_consensus.cc:374] T 4f8fb9f623124e2a956a56a38754b0f3 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:05.956346 350 raft_consensus.cc:729] T 4f8fb9f623124e2a956a56a38754b0f3 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:13:05.956377 352 tablet_bootstrap.cc:492] T 4f8fb9f623124e2a956a56a38754b0f3 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:13:05.956532 352 ts_tablet_manager.cc:1367] T 4f8fb9f623124e2a956a56a38754b0f3 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 08:13:05.956568 350 consensus_queue.cc:260] T 4f8fb9f623124e2a956a56a38754b0f3 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:05.957099 350 ts_tablet_manager.cc:1398] T 4f8fb9f623124e2a956a56a38754b0f3 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:05.957195 352 raft_consensus.cc:348] T 4f8fb9f623124e2a956a56a38754b0f3 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:05.957396 352 raft_consensus.cc:374] T 4f8fb9f623124e2a956a56a38754b0f3 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:05.957481 352 raft_consensus.cc:729] T 4f8fb9f623124e2a956a56a38754b0f3 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:13:05.957682 352 consensus_queue.cc:260] T 4f8fb9f623124e2a956a56a38754b0f3 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:05.958376 352 ts_tablet_manager.cc:1398] T 4f8fb9f623124e2a956a56a38754b0f3 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:06.000253 327 consensus_queue.cc:1035] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:06.007540 327 consensus_queue.cc:1035] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:06.015552 311 raft_consensus.cc:482] T 4f8fb9f623124e2a956a56a38754b0f3 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:06.015677 311 raft_consensus.cc:504] T 4f8fb9f623124e2a956a56a38754b0f3 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:06.016345 311 leader_election.cc:288] T 4f8fb9f623124e2a956a56a38754b0f3 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:13:06.016965 28389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f8fb9f623124e2a956a56a38754b0f3" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:13:06.017055 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f8fb9f623124e2a956a56a38754b0f3" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:13:06.017210 28389 raft_consensus.cc:2454] T 4f8fb9f623124e2a956a56a38754b0f3 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:13:06.017251 31027 raft_consensus.cc:2454] T 4f8fb9f623124e2a956a56a38754b0f3 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:13:06.017570 27939 leader_election.cc:302] T 4f8fb9f623124e2a956a56a38754b0f3 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:13:06.017822 311 raft_consensus.cc:2781] T 4f8fb9f623124e2a956a56a38754b0f3 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:06.017899 311 raft_consensus.cc:482] T 4f8fb9f623124e2a956a56a38754b0f3 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:06.017956 311 raft_consensus.cc:3037] T 4f8fb9f623124e2a956a56a38754b0f3 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:06.019986 311 raft_consensus.cc:504] T 4f8fb9f623124e2a956a56a38754b0f3 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:06.020416 311 leader_election.cc:288] T 4f8fb9f623124e2a956a56a38754b0f3 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Requested vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:13:06.020896 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f8fb9f623124e2a956a56a38754b0f3" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:13:06.020958 28389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f8fb9f623124e2a956a56a38754b0f3" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:13:06.021121 28389 raft_consensus.cc:3037] T 4f8fb9f623124e2a956a56a38754b0f3 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:06.021111 31027 raft_consensus.cc:3037] T 4f8fb9f623124e2a956a56a38754b0f3 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:06.023620 28389 raft_consensus.cc:2454] T 4f8fb9f623124e2a956a56a38754b0f3 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:13:06.023734 31027 raft_consensus.cc:2454] T 4f8fb9f623124e2a956a56a38754b0f3 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:13:06.024076 27939 leader_election.cc:302] T 4f8fb9f623124e2a956a56a38754b0f3 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:13:06.024287 311 raft_consensus.cc:2781] T 4f8fb9f623124e2a956a56a38754b0f3 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:06.024487 311 raft_consensus.cc:686] T 4f8fb9f623124e2a956a56a38754b0f3 P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Becoming Leader. State: Replica: 833582ff68c94269be364ebd187a75db, State: Running, Role: LEADER
I20240503 08:13:06.024729 311 consensus_queue.cc:237] T 4f8fb9f623124e2a956a56a38754b0f3 P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:06.025964 327 consensus_queue.cc:1035] T d61ad2e11c4544bfbdef7df1e92d2d11 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:06.030409 27859 catalog_manager.cc:5484] T 4f8fb9f623124e2a956a56a38754b0f3 P 833582ff68c94269be364ebd187a75db reported cstate change: term changed from 0 to 1, leader changed from <none> to 833582ff68c94269be364ebd187a75db (127.20.230.65). New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:13:06.030629 323 consensus_queue.cc:1035] T d61ad2e11c4544bfbdef7df1e92d2d11 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:06.100752 27859 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_extra_table_58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:13:06.109071 28111 tablet_service.cc:1467] Processing CreateTablet for tablet f463dfe26ee94b5f9447c6dbdf8759c3 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=69e4cfedb64b4938988b9ed5d473eec0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:06.109588 28111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f463dfe26ee94b5f9447c6dbdf8759c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:06.109697 28373 tablet_service.cc:1467] Processing CreateTablet for tablet f463dfe26ee94b5f9447c6dbdf8759c3 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=69e4cfedb64b4938988b9ed5d473eec0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:06.110131 28373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f463dfe26ee94b5f9447c6dbdf8759c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:06.110165 31140 tablet_service.cc:1467] Processing CreateTablet for tablet f463dfe26ee94b5f9447c6dbdf8759c3 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=69e4cfedb64b4938988b9ed5d473eec0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:06.110587 31140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f463dfe26ee94b5f9447c6dbdf8759c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:06.110498 30741 tablet_service.cc:1467] Processing CreateTablet for tablet 7c8c7659181040cabf20ae116fae78c8 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=69e4cfedb64b4938988b9ed5d473eec0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:06.110883 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c8c7659181040cabf20ae116fae78c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:06.114092 27981 tablet_service.cc:1467] Processing CreateTablet for tablet 7c8c7659181040cabf20ae116fae78c8 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=69e4cfedb64b4938988b9ed5d473eec0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:06.114092 27983 tablet_service.cc:1467] Processing CreateTablet for tablet d0a06e419c6c4f3db6705cb2377e9336 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=69e4cfedb64b4938988b9ed5d473eec0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:06.114442 27981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c8c7659181040cabf20ae116fae78c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:06.114641 27983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0a06e419c6c4f3db6705cb2377e9336. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:06.115612 30874 tablet_service.cc:1467] Processing CreateTablet for tablet d0a06e419c6c4f3db6705cb2377e9336 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=69e4cfedb64b4938988b9ed5d473eec0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:06.115993 30874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0a06e419c6c4f3db6705cb2377e9336. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:06.117949 349 tablet_bootstrap.cc:492] T f463dfe26ee94b5f9447c6dbdf8759c3 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:13:06.119302 349 tablet_bootstrap.cc:654] T f463dfe26ee94b5f9447c6dbdf8759c3 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:06.122999 28113 tablet_service.cc:1467] Processing CreateTablet for tablet d0a06e419c6c4f3db6705cb2377e9336 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=69e4cfedb64b4938988b9ed5d473eec0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:06.123414 28113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0a06e419c6c4f3db6705cb2377e9336. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:06.123689 350 tablet_bootstrap.cc:492] T 7c8c7659181040cabf20ae116fae78c8 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:13:06.124917 350 tablet_bootstrap.cc:654] T 7c8c7659181040cabf20ae116fae78c8 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:06.126369 350 tablet_bootstrap.cc:492] T 7c8c7659181040cabf20ae116fae78c8 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:13:06.126475 350 ts_tablet_manager.cc:1367] T 7c8c7659181040cabf20ae116fae78c8 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:13:06.127012 350 raft_consensus.cc:348] T 7c8c7659181040cabf20ae116fae78c8 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:06.127149 350 raft_consensus.cc:374] T 7c8c7659181040cabf20ae116fae78c8 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:06.127190 350 raft_consensus.cc:729] T 7c8c7659181040cabf20ae116fae78c8 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:13:06.127378 350 consensus_queue.cc:260] T 7c8c7659181040cabf20ae116fae78c8 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:06.128121 350 ts_tablet_manager.cc:1398] T 7c8c7659181040cabf20ae116fae78c8 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:06.128751 28243 tablet_service.cc:1467] Processing CreateTablet for tablet 7c8c7659181040cabf20ae116fae78c8 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=69e4cfedb64b4938988b9ed5d473eec0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:06.129343 349 tablet_bootstrap.cc:492] T f463dfe26ee94b5f9447c6dbdf8759c3 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:13:06.129506 349 ts_tablet_manager.cc:1367] T f463dfe26ee94b5f9447c6dbdf8759c3 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240503 08:13:06.129737 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c8c7659181040cabf20ae116fae78c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:06.130200 349 raft_consensus.cc:348] T f463dfe26ee94b5f9447c6dbdf8759c3 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:06.130416 349 raft_consensus.cc:374] T f463dfe26ee94b5f9447c6dbdf8759c3 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:06.130479 349 raft_consensus.cc:729] T f463dfe26ee94b5f9447c6dbdf8759c3 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:13:06.130707 349 consensus_queue.cc:260] T f463dfe26ee94b5f9447c6dbdf8759c3 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:06.131204 349 ts_tablet_manager.cc:1398] T f463dfe26ee94b5f9447c6dbdf8759c3 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:06.141741 27854 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:13:06.142196 367 tablet_bootstrap.cc:492] T 7c8c7659181040cabf20ae116fae78c8 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:13:06.142612 350 tablet_bootstrap.cc:492] T d0a06e419c6c4f3db6705cb2377e9336 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:13:06.143859 367 tablet_bootstrap.cc:654] T 7c8c7659181040cabf20ae116fae78c8 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:06.144896 350 tablet_bootstrap.cc:654] T d0a06e419c6c4f3db6705cb2377e9336 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:06.145663 364 tablet_bootstrap.cc:492] T d0a06e419c6c4f3db6705cb2377e9336 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:13:06.149421 364 tablet_bootstrap.cc:654] T d0a06e419c6c4f3db6705cb2377e9336 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:06.151288 367 tablet_bootstrap.cc:492] T 7c8c7659181040cabf20ae116fae78c8 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:13:06.151484 367 ts_tablet_manager.cc:1367] T 7c8c7659181040cabf20ae116fae78c8 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240503 08:13:06.151588 350 tablet_bootstrap.cc:492] T d0a06e419c6c4f3db6705cb2377e9336 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:13:06.151693 350 ts_tablet_manager.cc:1367] T d0a06e419c6c4f3db6705cb2377e9336 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 08:13:06.152676 367 raft_consensus.cc:348] T 7c8c7659181040cabf20ae116fae78c8 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:06.152921 367 raft_consensus.cc:374] T 7c8c7659181040cabf20ae116fae78c8 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:06.153049 367 raft_consensus.cc:729] T 7c8c7659181040cabf20ae116fae78c8 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:13:06.153193 350 raft_consensus.cc:348] T d0a06e419c6c4f3db6705cb2377e9336 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:06.153348 350 raft_consensus.cc:374] T d0a06e419c6c4f3db6705cb2377e9336 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:06.153406 350 raft_consensus.cc:729] T d0a06e419c6c4f3db6705cb2377e9336 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:13:06.153623 350 consensus_queue.cc:260] T d0a06e419c6c4f3db6705cb2377e9336 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:06.154662 350 ts_tablet_manager.cc:1398] T d0a06e419c6c4f3db6705cb2377e9336 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:13:06.156343 367 consensus_queue.cc:260] T 7c8c7659181040cabf20ae116fae78c8 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:06.157094 367 ts_tablet_manager.cc:1398] T 7c8c7659181040cabf20ae116fae78c8 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:13:06.174391 370 tablet_bootstrap.cc:492] T d0a06e419c6c4f3db6705cb2377e9336 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:13:06.176841 370 tablet_bootstrap.cc:654] T d0a06e419c6c4f3db6705cb2377e9336 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:06.179697 28389 raft_consensus.cc:1261] T 4f8fb9f623124e2a956a56a38754b0f3 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:06.179744 31027 raft_consensus.cc:1261] T 4f8fb9f623124e2a956a56a38754b0f3 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:06.180492 32754 consensus_queue.cc:1035] T 4f8fb9f623124e2a956a56a38754b0f3 P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:06.180935 32754 consensus_queue.cc:1035] T 4f8fb9f623124e2a956a56a38754b0f3 P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:06.186659 311 raft_consensus.cc:482] T d0a06e419c6c4f3db6705cb2377e9336 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:06.186791 311 raft_consensus.cc:504] T d0a06e419c6c4f3db6705cb2377e9336 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:06.187564 311 leader_election.cc:288] T d0a06e419c6c4f3db6705cb2377e9336 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:13:06.188623 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0a06e419c6c4f3db6705cb2377e9336" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:13:06.189103 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0a06e419c6c4f3db6705cb2377e9336" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
W20240503 08:13:06.189153 27936 leader_election.cc:341] T d0a06e419c6c4f3db6705cb2377e9336 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): Illegal state: must be running to vote when last-logged opid is not known
W20240503 08:13:06.189617 27938 leader_election.cc:341] T d0a06e419c6c4f3db6705cb2377e9336 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:13:06.189736 27938 leader_election.cc:302] T d0a06e419c6c4f3db6705cb2377e9336 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 833582ff68c94269be364ebd187a75db; no voters: 1b411c6bb25549368041773d7b710d48, ce1496a47d084365ad786cd09e472a2c
I20240503 08:13:06.190171 311 raft_consensus.cc:2726] T d0a06e419c6c4f3db6705cb2377e9336 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:13:06.191758 27858 catalog_manager.cc:5576] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211) does not have the latest schema for tablet 4f8fb9f623124e2a956a56a38754b0f3 (table rebalancer_dynamic_table [id=76e29da269bb4a6ca00083d58e880fd2]). Expected version 1 got 0
I20240503 08:13:06.193104 329 tablet.cc:1722] T 4f8fb9f623124e2a956a56a38754b0f3 P d09a11a03989421b9f28019dc1c14cc2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:13:06.193629 329 tablet.cc:1620] T 4f8fb9f623124e2a956a56a38754b0f3 P d09a11a03989421b9f28019dc1c14cc2: MemRowSet was empty: no flush needed.
I20240503 08:13:06.203845 27858 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=76e29da269bb4a6ca00083d58e880fd2] alter complete (version 1)
I20240503 08:13:06.210877 373 tablet_bootstrap.cc:492] T 7c8c7659181040cabf20ae116fae78c8 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:13:06.213634 377 tablet.cc:1722] T 4f8fb9f623124e2a956a56a38754b0f3 P 833582ff68c94269be364ebd187a75db: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:13:06.214128 377 tablet.cc:1620] T 4f8fb9f623124e2a956a56a38754b0f3 P 833582ff68c94269be364ebd187a75db: MemRowSet was empty: no flush needed.
I20240503 08:13:06.215116 373 tablet_bootstrap.cc:654] T 7c8c7659181040cabf20ae116fae78c8 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:06.217878 365 tablet_bootstrap.cc:492] T f463dfe26ee94b5f9447c6dbdf8759c3 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:13:06.219404 365 tablet_bootstrap.cc:654] T f463dfe26ee94b5f9447c6dbdf8759c3 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:06.223382 27859 catalog_manager.cc:5576] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915) does not have the latest schema for tablet 4f8fb9f623124e2a956a56a38754b0f3 (table rebalancer_dynamic_table [id=76e29da269bb4a6ca00083d58e880fd2]). Expected version 1 got 0
I20240503 08:13:06.227979 359 mvcc.cc:205] Tried to move back new op lower bound from 7023509447385595904 to 7023509446759800832. Current Snapshot: MvccSnapshot[applied={T|T < 7023509447385595904}]
I20240503 08:13:06.229956 364 tablet_bootstrap.cc:492] T d0a06e419c6c4f3db6705cb2377e9336 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:13:06.230139 364 ts_tablet_manager.cc:1367] T d0a06e419c6c4f3db6705cb2377e9336 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.085s user 0.011s sys 0.010s
I20240503 08:13:06.230959 364 raft_consensus.cc:348] T d0a06e419c6c4f3db6705cb2377e9336 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:06.231186 364 raft_consensus.cc:374] T d0a06e419c6c4f3db6705cb2377e9336 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:06.231302 364 raft_consensus.cc:729] T d0a06e419c6c4f3db6705cb2377e9336 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:13:06.231555 364 consensus_queue.cc:260] T d0a06e419c6c4f3db6705cb2377e9336 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:06.233280 364 ts_tablet_manager.cc:1398] T d0a06e419c6c4f3db6705cb2377e9336 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:13:06.233410 373 tablet_bootstrap.cc:492] T 7c8c7659181040cabf20ae116fae78c8 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:13:06.233567 373 ts_tablet_manager.cc:1367] T 7c8c7659181040cabf20ae116fae78c8 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20240503 08:13:06.233598 370 tablet_bootstrap.cc:492] T d0a06e419c6c4f3db6705cb2377e9336 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:13:06.233733 370 ts_tablet_manager.cc:1367] T d0a06e419c6c4f3db6705cb2377e9336 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.059s user 0.001s sys 0.004s
I20240503 08:13:06.235627 370 raft_consensus.cc:348] T d0a06e419c6c4f3db6705cb2377e9336 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:06.235893 370 raft_consensus.cc:374] T d0a06e419c6c4f3db6705cb2377e9336 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:06.236290 381 tablet.cc:1722] T 4f8fb9f623124e2a956a56a38754b0f3 P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:13:06.234288 373 raft_consensus.cc:348] T 7c8c7659181040cabf20ae116fae78c8 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:06.236573 373 raft_consensus.cc:374] T 7c8c7659181040cabf20ae116fae78c8 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:06.236714 370 raft_consensus.cc:729] T d0a06e419c6c4f3db6705cb2377e9336 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:13:06.236770 373 raft_consensus.cc:729] T 7c8c7659181040cabf20ae116fae78c8 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:13:06.237000 373 consensus_queue.cc:260] T 7c8c7659181040cabf20ae116fae78c8 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:06.237111 370 consensus_queue.cc:260] T d0a06e419c6c4f3db6705cb2377e9336 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:06.237574 373 ts_tablet_manager.cc:1398] T 7c8c7659181040cabf20ae116fae78c8 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 08:13:06.238844 27859 catalog_manager.cc:5576] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915) does not have the latest schema for tablet 4f8fb9f623124e2a956a56a38754b0f3 (table rebalancer_dynamic_table [id=76e29da269bb4a6ca00083d58e880fd2]). Expected version 1 got 0
I20240503 08:13:06.243949 370 ts_tablet_manager.cc:1398] T d0a06e419c6c4f3db6705cb2377e9336 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 08:13:06.244707 381 tablet.cc:1620] T 4f8fb9f623124e2a956a56a38754b0f3 P ced21ca983b749d0b4d4f6a0e2426bf8: MemRowSet was empty: no flush needed.
I20240503 08:13:06.245328 370 tablet_bootstrap.cc:492] T f463dfe26ee94b5f9447c6dbdf8759c3 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:13:06.249289 370 tablet_bootstrap.cc:654] T f463dfe26ee94b5f9447c6dbdf8759c3 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:06.250375 365 tablet_bootstrap.cc:492] T f463dfe26ee94b5f9447c6dbdf8759c3 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:13:06.250552 365 ts_tablet_manager.cc:1367] T f463dfe26ee94b5f9447c6dbdf8759c3 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.002s
I20240503 08:13:06.251245 365 raft_consensus.cc:348] T f463dfe26ee94b5f9447c6dbdf8759c3 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:06.252101 365 raft_consensus.cc:374] T f463dfe26ee94b5f9447c6dbdf8759c3 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:06.252236 365 raft_consensus.cc:729] T f463dfe26ee94b5f9447c6dbdf8759c3 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:13:06.252462 365 consensus_queue.cc:260] T f463dfe26ee94b5f9447c6dbdf8759c3 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:06.253053 365 ts_tablet_manager.cc:1398] T f463dfe26ee94b5f9447c6dbdf8759c3 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:06.255031 370 tablet_bootstrap.cc:492] T f463dfe26ee94b5f9447c6dbdf8759c3 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:13:06.255239 370 ts_tablet_manager.cc:1367] T f463dfe26ee94b5f9447c6dbdf8759c3 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.002s
I20240503 08:13:06.256122 370 raft_consensus.cc:348] T f463dfe26ee94b5f9447c6dbdf8759c3 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:06.256371 370 raft_consensus.cc:374] T f463dfe26ee94b5f9447c6dbdf8759c3 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:06.256481 370 raft_consensus.cc:729] T f463dfe26ee94b5f9447c6dbdf8759c3 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:13:06.257164 370 consensus_queue.cc:260] T f463dfe26ee94b5f9447c6dbdf8759c3 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:06.257685 370 ts_tablet_manager.cc:1398] T f463dfe26ee94b5f9447c6dbdf8759c3 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:13:06.310523 27859 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:13:06.317924 377 tablet.cc:1722] T 4f8fb9f623124e2a956a56a38754b0f3 P 833582ff68c94269be364ebd187a75db: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:13:06.318452 377 tablet.cc:1620] T 4f8fb9f623124e2a956a56a38754b0f3 P 833582ff68c94269be364ebd187a75db: MemRowSet was empty: no flush needed.
I20240503 08:13:06.318511 329 tablet.cc:1722] T 4f8fb9f623124e2a956a56a38754b0f3 P d09a11a03989421b9f28019dc1c14cc2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:13:06.318931 329 tablet.cc:1620] T 4f8fb9f623124e2a956a56a38754b0f3 P d09a11a03989421b9f28019dc1c14cc2: MemRowSet was empty: no flush needed.
I20240503 08:13:06.318970 381 tablet.cc:1722] T 4f8fb9f623124e2a956a56a38754b0f3 P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:13:06.319530 381 tablet.cc:1620] T 4f8fb9f623124e2a956a56a38754b0f3 P ced21ca983b749d0b4d4f6a0e2426bf8: MemRowSet was empty: no flush needed.
I20240503 08:13:06.328505 27859 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=76e29da269bb4a6ca00083d58e880fd2] alter complete (version 2)
I20240503 08:13:06.335922 355 raft_consensus.cc:482] T f463dfe26ee94b5f9447c6dbdf8759c3 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:06.336062 355 raft_consensus.cc:504] T f463dfe26ee94b5f9447c6dbdf8759c3 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:06.336704 355 leader_election.cc:288] T f463dfe26ee94b5f9447c6dbdf8759c3 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:13:06.336992 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f463dfe26ee94b5f9447c6dbdf8759c3" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:13:06.337026 28389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f463dfe26ee94b5f9447c6dbdf8759c3" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:13:06.337185 28389 raft_consensus.cc:2454] T f463dfe26ee94b5f9447c6dbdf8759c3 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 0.
I20240503 08:13:06.337198 28132 raft_consensus.cc:2454] T f463dfe26ee94b5f9447c6dbdf8759c3 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 0.
I20240503 08:13:06.337550 31096 leader_election.cc:302] T f463dfe26ee94b5f9447c6dbdf8759c3 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, b00f244c69df401997ea99d492218a98; no voters:
I20240503 08:13:06.337822 355 raft_consensus.cc:2781] T f463dfe26ee94b5f9447c6dbdf8759c3 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:06.337932 355 raft_consensus.cc:482] T f463dfe26ee94b5f9447c6dbdf8759c3 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:06.338014 355 raft_consensus.cc:3037] T f463dfe26ee94b5f9447c6dbdf8759c3 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:06.339866 355 raft_consensus.cc:504] T f463dfe26ee94b5f9447c6dbdf8759c3 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:06.340349 355 leader_election.cc:288] T f463dfe26ee94b5f9447c6dbdf8759c3 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 election: Requested vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:13:06.340708 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f463dfe26ee94b5f9447c6dbdf8759c3" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:13:06.340866 28132 raft_consensus.cc:3037] T f463dfe26ee94b5f9447c6dbdf8759c3 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:06.340848 28389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f463dfe26ee94b5f9447c6dbdf8759c3" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:13:06.341012 28389 raft_consensus.cc:3037] T f463dfe26ee94b5f9447c6dbdf8759c3 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:06.342820 28389 raft_consensus.cc:2454] T f463dfe26ee94b5f9447c6dbdf8759c3 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 1.
I20240503 08:13:06.343163 31097 leader_election.cc:302] T f463dfe26ee94b5f9447c6dbdf8759c3 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:13:06.343310 28132 raft_consensus.cc:2454] T f463dfe26ee94b5f9447c6dbdf8759c3 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 1.
I20240503 08:13:06.343432 355 raft_consensus.cc:2781] T f463dfe26ee94b5f9447c6dbdf8759c3 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:06.343657 355 raft_consensus.cc:686] T f463dfe26ee94b5f9447c6dbdf8759c3 P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Becoming Leader. State: Replica: b00f244c69df401997ea99d492218a98, State: Running, Role: LEADER
I20240503 08:13:06.343916 355 consensus_queue.cc:237] T f463dfe26ee94b5f9447c6dbdf8759c3 P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:06.347070 27859 catalog_manager.cc:5484] T f463dfe26ee94b5f9447c6dbdf8759c3 P b00f244c69df401997ea99d492218a98 reported cstate change: term changed from 0 to 1, leader changed from <none> to b00f244c69df401997ea99d492218a98 (127.20.230.72). New cstate: current_term: 1 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:06.412555 324 raft_consensus.cc:482] T d0a06e419c6c4f3db6705cb2377e9336 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:06.412712 324 raft_consensus.cc:504] T d0a06e419c6c4f3db6705cb2377e9336 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:06.413301 324 leader_election.cc:288] T d0a06e419c6c4f3db6705cb2377e9336 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:13:06.413681 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0a06e419c6c4f3db6705cb2377e9336" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:13:06.413720 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0a06e419c6c4f3db6705cb2377e9336" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:13:06.413882 28002 raft_consensus.cc:2454] T d0a06e419c6c4f3db6705cb2377e9336 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:13:06.413969 30895 raft_consensus.cc:2454] T d0a06e419c6c4f3db6705cb2377e9336 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 0.
I20240503 08:13:06.414300 28066 leader_election.cc:302] T d0a06e419c6c4f3db6705cb2377e9336 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:13:06.414624 324 raft_consensus.cc:2781] T d0a06e419c6c4f3db6705cb2377e9336 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:06.414746 324 raft_consensus.cc:482] T d0a06e419c6c4f3db6705cb2377e9336 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:06.414844 324 raft_consensus.cc:3037] T d0a06e419c6c4f3db6705cb2377e9336 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:06.416854 324 raft_consensus.cc:504] T d0a06e419c6c4f3db6705cb2377e9336 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:06.417400 324 leader_election.cc:288] T d0a06e419c6c4f3db6705cb2377e9336 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Requested vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:13:06.417966 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0a06e419c6c4f3db6705cb2377e9336" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:13:06.418058 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0a06e419c6c4f3db6705cb2377e9336" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:13:06.418133 28002 raft_consensus.cc:3037] T d0a06e419c6c4f3db6705cb2377e9336 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:06.418256 30895 raft_consensus.cc:3037] T d0a06e419c6c4f3db6705cb2377e9336 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:06.420794 28002 raft_consensus.cc:2454] T d0a06e419c6c4f3db6705cb2377e9336 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:13:06.420810 30895 raft_consensus.cc:2454] T d0a06e419c6c4f3db6705cb2377e9336 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:13:06.421262 28069 leader_election.cc:302] T d0a06e419c6c4f3db6705cb2377e9336 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 833582ff68c94269be364ebd187a75db, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:13:06.421587 331 raft_consensus.cc:2781] T d0a06e419c6c4f3db6705cb2377e9336 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:06.421721 331 raft_consensus.cc:686] T d0a06e419c6c4f3db6705cb2377e9336 P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Becoming Leader. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Running, Role: LEADER
I20240503 08:13:06.421996 331 consensus_queue.cc:237] T d0a06e419c6c4f3db6705cb2377e9336 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:06.425640 27859 catalog_manager.cc:5484] T d0a06e419c6c4f3db6705cb2377e9336 P 1b411c6bb25549368041773d7b710d48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b411c6bb25549368041773d7b710d48 (127.20.230.66). New cstate: current_term: 1 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:06.429912 32754 raft_consensus.cc:482] T 7c8c7659181040cabf20ae116fae78c8 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:06.430022 32754 raft_consensus.cc:504] T 7c8c7659181040cabf20ae116fae78c8 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:06.430531 32754 leader_election.cc:288] T 7c8c7659181040cabf20ae116fae78c8 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:13:06.430951 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8c7659181040cabf20ae116fae78c8" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:13:06.430943 28261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8c7659181040cabf20ae116fae78c8" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:13:06.431165 30761 raft_consensus.cc:2454] T 7c8c7659181040cabf20ae116fae78c8 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:13:06.431165 28261 raft_consensus.cc:2454] T 7c8c7659181040cabf20ae116fae78c8 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 0.
I20240503 08:13:06.431586 27937 leader_election.cc:302] T 7c8c7659181040cabf20ae116fae78c8 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:13:06.431805 32754 raft_consensus.cc:2781] T 7c8c7659181040cabf20ae116fae78c8 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:06.431903 32754 raft_consensus.cc:482] T 7c8c7659181040cabf20ae116fae78c8 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:06.431968 32754 raft_consensus.cc:3037] T 7c8c7659181040cabf20ae116fae78c8 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:06.432166 27859 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:13:06.433725 32754 raft_consensus.cc:504] T 7c8c7659181040cabf20ae116fae78c8 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:06.434242 32754 leader_election.cc:288] T 7c8c7659181040cabf20ae116fae78c8 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:13:06.434688 28261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8c7659181040cabf20ae116fae78c8" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:13:06.434868 28261 raft_consensus.cc:3037] T 7c8c7659181040cabf20ae116fae78c8 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:06.434932 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8c7659181040cabf20ae116fae78c8" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:13:06.435094 30761 raft_consensus.cc:3037] T 7c8c7659181040cabf20ae116fae78c8 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:06.437474 28261 raft_consensus.cc:2454] T 7c8c7659181040cabf20ae116fae78c8 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:13:06.437660 30761 raft_consensus.cc:2454] T 7c8c7659181040cabf20ae116fae78c8 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:13:06.437883 27937 leader_election.cc:302] T 7c8c7659181040cabf20ae116fae78c8 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:13:06.438086 32754 raft_consensus.cc:2781] T 7c8c7659181040cabf20ae116fae78c8 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:06.438195 32754 raft_consensus.cc:686] T 7c8c7659181040cabf20ae116fae78c8 P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Becoming Leader. State: Replica: 833582ff68c94269be364ebd187a75db, State: Running, Role: LEADER
I20240503 08:13:06.438625 32754 consensus_queue.cc:237] T 7c8c7659181040cabf20ae116fae78c8 P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:06.445326 377 tablet.cc:1722] T 4f8fb9f623124e2a956a56a38754b0f3 P 833582ff68c94269be364ebd187a75db: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:13:06.451750 381 tablet.cc:1722] T 4f8fb9f623124e2a956a56a38754b0f3 P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:13:06.460940 27859 catalog_manager.cc:5484] T 7c8c7659181040cabf20ae116fae78c8 P 833582ff68c94269be364ebd187a75db reported cstate change: term changed from 0 to 1, leader changed from <none> to 833582ff68c94269be364ebd187a75db (127.20.230.65). New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:06.466822 27858 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table_new_name [id=76e29da269bb4a6ca00083d58e880fd2] alter complete (version 3)
I20240503 08:13:06.470423 329 tablet.cc:1722] T 4f8fb9f623124e2a956a56a38754b0f3 P d09a11a03989421b9f28019dc1c14cc2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:13:06.568810 27858 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:13:06.569118 27858 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:13:06.572674 27858 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet 4f8fb9f623124e2a956a56a38754b0f3
I20240503 08:13:06.573370 27983 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f8fb9f623124e2a956a56a38754b0f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:06 UTC) from {username='slave'} at 127.0.0.1:55610
I20240503 08:13:06.573575 28373 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f8fb9f623124e2a956a56a38754b0f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:06 UTC) from {username='slave'} at 127.0.0.1:34786
I20240503 08:13:06.573657 31006 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f8fb9f623124e2a956a56a38754b0f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:06 UTC) from {username='slave'} at 127.0.0.1:47020
I20240503 08:13:06.574257 392 tablet_replica.cc:330] T 4f8fb9f623124e2a956a56a38754b0f3 P d09a11a03989421b9f28019dc1c14cc2: stopping tablet replica
I20240503 08:13:06.574575 392 raft_consensus.cc:2229] T 4f8fb9f623124e2a956a56a38754b0f3 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:06.574863 392 raft_consensus.cc:2258] T 4f8fb9f623124e2a956a56a38754b0f3 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:06.575771 392 ts_tablet_manager.cc:1875] T 4f8fb9f623124e2a956a56a38754b0f3 P d09a11a03989421b9f28019dc1c14cc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:06.583429 391 tablet_replica.cc:330] T 4f8fb9f623124e2a956a56a38754b0f3 P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:13:06.583606 392 ts_tablet_manager.cc:1888] T 4f8fb9f623124e2a956a56a38754b0f3 P d09a11a03989421b9f28019dc1c14cc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:13:06.583756 393 tablet_replica.cc:330] T 4f8fb9f623124e2a956a56a38754b0f3 P ced21ca983b749d0b4d4f6a0e2426bf8: stopping tablet replica
I20240503 08:13:06.583858 392 log.cc:1196] T 4f8fb9f623124e2a956a56a38754b0f3 P d09a11a03989421b9f28019dc1c14cc2: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/wals/4f8fb9f623124e2a956a56a38754b0f3
I20240503 08:13:06.583840 391 raft_consensus.cc:2229] T 4f8fb9f623124e2a956a56a38754b0f3 P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:13:06.584021 393 raft_consensus.cc:2229] T 4f8fb9f623124e2a956a56a38754b0f3 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:06.584229 391 raft_consensus.cc:2258] T 4f8fb9f623124e2a956a56a38754b0f3 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:06.584277 393 raft_consensus.cc:2258] T 4f8fb9f623124e2a956a56a38754b0f3 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:06.584518 392 ts_tablet_manager.cc:1909] T 4f8fb9f623124e2a956a56a38754b0f3 P d09a11a03989421b9f28019dc1c14cc2: Deleting consensus metadata
I20240503 08:13:06.585160 393 ts_tablet_manager.cc:1875] T 4f8fb9f623124e2a956a56a38754b0f3 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:06.585278 391 ts_tablet_manager.cc:1875] T 4f8fb9f623124e2a956a56a38754b0f3 P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:06.587623 27849 catalog_manager.cc:4830] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): tablet 4f8fb9f623124e2a956a56a38754b0f3 (table rebalancer_dynamic_table_new_name [id=76e29da269bb4a6ca00083d58e880fd2]) successfully deleted
I20240503 08:13:06.590607 391 ts_tablet_manager.cc:1888] T 4f8fb9f623124e2a956a56a38754b0f3 P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:13:06.590723 391 log.cc:1196] T 4f8fb9f623124e2a956a56a38754b0f3 P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/4f8fb9f623124e2a956a56a38754b0f3
I20240503 08:13:06.591037 391 ts_tablet_manager.cc:1909] T 4f8fb9f623124e2a956a56a38754b0f3 P 833582ff68c94269be364ebd187a75db: Deleting consensus metadata
I20240503 08:13:06.591105 393 ts_tablet_manager.cc:1888] T 4f8fb9f623124e2a956a56a38754b0f3 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:13:06.591223 393 log.cc:1196] T 4f8fb9f623124e2a956a56a38754b0f3 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal/wals/4f8fb9f623124e2a956a56a38754b0f3
I20240503 08:13:06.591555 393 ts_tablet_manager.cc:1909] T 4f8fb9f623124e2a956a56a38754b0f3 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting consensus metadata
I20240503 08:13:06.592021 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet 4f8fb9f623124e2a956a56a38754b0f3 (table rebalancer_dynamic_table_new_name [id=76e29da269bb4a6ca00083d58e880fd2]) successfully deleted
I20240503 08:13:06.592540 27847 catalog_manager.cc:4830] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): tablet 4f8fb9f623124e2a956a56a38754b0f3 (table rebalancer_dynamic_table_new_name [id=76e29da269bb4a6ca00083d58e880fd2]) successfully deleted
I20240503 08:13:06.675427 27858 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:13:06.684154 31006 tablet_service.cc:1467] Processing CreateTablet for tablet 5ae8cfd5411d489e96cbf21da84401ac (DEFAULT_TABLE table=rebalancer_dynamic_table [id=912224c75f7d47e1aa74e2c6024759b9]), partition=
I20240503 08:13:06.684218 28373 tablet_service.cc:1467] Processing CreateTablet for tablet 5ae8cfd5411d489e96cbf21da84401ac (DEFAULT_TABLE table=rebalancer_dynamic_table [id=912224c75f7d47e1aa74e2c6024759b9]), partition=
I20240503 08:13:06.684234 30874 tablet_service.cc:1467] Processing CreateTablet for tablet 5ae8cfd5411d489e96cbf21da84401ac (DEFAULT_TABLE table=rebalancer_dynamic_table [id=912224c75f7d47e1aa74e2c6024759b9]), partition=
I20240503 08:13:06.684579 30874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ae8cfd5411d489e96cbf21da84401ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:06.684582 31006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ae8cfd5411d489e96cbf21da84401ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:06.684582 28373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ae8cfd5411d489e96cbf21da84401ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:06.689855 364 tablet_bootstrap.cc:492] T 5ae8cfd5411d489e96cbf21da84401ac P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:13:06.691421 364 tablet_bootstrap.cc:654] T 5ae8cfd5411d489e96cbf21da84401ac P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:06.694794 364 tablet_bootstrap.cc:492] T 5ae8cfd5411d489e96cbf21da84401ac P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:13:06.694972 364 ts_tablet_manager.cc:1367] T 5ae8cfd5411d489e96cbf21da84401ac P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 08:13:06.696305 364 raft_consensus.cc:348] T 5ae8cfd5411d489e96cbf21da84401ac P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:06.696492 364 raft_consensus.cc:374] T 5ae8cfd5411d489e96cbf21da84401ac P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:06.696535 364 raft_consensus.cc:729] T 5ae8cfd5411d489e96cbf21da84401ac P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:13:06.696732 364 consensus_queue.cc:260] T 5ae8cfd5411d489e96cbf21da84401ac P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:06.697430 396 tablet_bootstrap.cc:492] T 5ae8cfd5411d489e96cbf21da84401ac P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:13:06.700286 364 ts_tablet_manager.cc:1398] T 5ae8cfd5411d489e96cbf21da84401ac P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 08:13:06.700366 396 tablet_bootstrap.cc:654] T 5ae8cfd5411d489e96cbf21da84401ac P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:06.703392 396 tablet_bootstrap.cc:492] T 5ae8cfd5411d489e96cbf21da84401ac P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:13:06.703680 394 tablet_bootstrap.cc:492] T 5ae8cfd5411d489e96cbf21da84401ac P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:13:06.705006 396 ts_tablet_manager.cc:1367] T 5ae8cfd5411d489e96cbf21da84401ac P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20240503 08:13:06.706948 394 tablet_bootstrap.cc:654] T 5ae8cfd5411d489e96cbf21da84401ac P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:06.705866 396 raft_consensus.cc:348] T 5ae8cfd5411d489e96cbf21da84401ac P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:06.707291 396 raft_consensus.cc:374] T 5ae8cfd5411d489e96cbf21da84401ac P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:06.707393 396 raft_consensus.cc:729] T 5ae8cfd5411d489e96cbf21da84401ac P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:13:06.709245 396 consensus_queue.cc:260] T 5ae8cfd5411d489e96cbf21da84401ac P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:06.710078 396 ts_tablet_manager.cc:1398] T 5ae8cfd5411d489e96cbf21da84401ac P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:13:06.711781 394 tablet_bootstrap.cc:492] T 5ae8cfd5411d489e96cbf21da84401ac P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:13:06.711936 394 ts_tablet_manager.cc:1367] T 5ae8cfd5411d489e96cbf21da84401ac P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20240503 08:13:06.712841 394 raft_consensus.cc:348] T 5ae8cfd5411d489e96cbf21da84401ac P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:06.713140 394 raft_consensus.cc:374] T 5ae8cfd5411d489e96cbf21da84401ac P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:06.713271 394 raft_consensus.cc:729] T 5ae8cfd5411d489e96cbf21da84401ac P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:13:06.713665 394 consensus_queue.cc:260] T 5ae8cfd5411d489e96cbf21da84401ac P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:06.714350 394 ts_tablet_manager.cc:1398] T 5ae8cfd5411d489e96cbf21da84401ac P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:06.737664 32493 consensus_queue.cc:1035] T f463dfe26ee94b5f9447c6dbdf8759c3 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:06.741992 32493 consensus_queue.cc:1035] T f463dfe26ee94b5f9447c6dbdf8759c3 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:06.823037 324 consensus_queue.cc:1035] T d0a06e419c6c4f3db6705cb2377e9336 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:06.828694 331 consensus_queue.cc:1035] T d0a06e419c6c4f3db6705cb2377e9336 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:06.914453 311 consensus_queue.cc:1035] T 7c8c7659181040cabf20ae116fae78c8 P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:06.918542 386 consensus_queue.cc:1035] T 7c8c7659181040cabf20ae116fae78c8 P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:06.925441 305 raft_consensus.cc:482] T 5ae8cfd5411d489e96cbf21da84401ac P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:06.925590 305 raft_consensus.cc:504] T 5ae8cfd5411d489e96cbf21da84401ac P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:06.926321 305 leader_election.cc:288] T 5ae8cfd5411d489e96cbf21da84401ac P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:13:06.926635 28389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae8cfd5411d489e96cbf21da84401ac" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:13:06.926874 28389 raft_consensus.cc:2454] T 5ae8cfd5411d489e96cbf21da84401ac P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 0.
I20240503 08:13:06.927223 30962 leader_election.cc:302] T 5ae8cfd5411d489e96cbf21da84401ac P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:13:06.927534 305 raft_consensus.cc:2781] T 5ae8cfd5411d489e96cbf21da84401ac P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:06.927634 305 raft_consensus.cc:482] T 5ae8cfd5411d489e96cbf21da84401ac P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:06.927724 305 raft_consensus.cc:3037] T 5ae8cfd5411d489e96cbf21da84401ac P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:06.929610 305 raft_consensus.cc:504] T 5ae8cfd5411d489e96cbf21da84401ac P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:06.930192 305 leader_election.cc:288] T 5ae8cfd5411d489e96cbf21da84401ac P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 election: Requested vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:13:06.930792 28389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae8cfd5411d489e96cbf21da84401ac" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:13:06.930902 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae8cfd5411d489e96cbf21da84401ac" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:13:06.930990 28389 raft_consensus.cc:3037] T 5ae8cfd5411d489e96cbf21da84401ac P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:06.930931 30893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae8cfd5411d489e96cbf21da84401ac" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:13:06.931115 30895 raft_consensus.cc:2454] T 5ae8cfd5411d489e96cbf21da84401ac P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 0.
I20240503 08:13:06.933018 28389 raft_consensus.cc:2454] T 5ae8cfd5411d489e96cbf21da84401ac P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 1.
I20240503 08:13:06.933372 30962 leader_election.cc:302] T 5ae8cfd5411d489e96cbf21da84401ac P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8, d09a11a03989421b9f28019dc1c14cc2; no voters: ce1496a47d084365ad786cd09e472a2c
I20240503 08:13:06.933940 305 raft_consensus.cc:2781] T 5ae8cfd5411d489e96cbf21da84401ac P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:06.934373 305 raft_consensus.cc:686] T 5ae8cfd5411d489e96cbf21da84401ac P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 LEADER]: Becoming Leader. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Running, Role: LEADER
I20240503 08:13:06.934649 305 consensus_queue.cc:237] T 5ae8cfd5411d489e96cbf21da84401ac P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:06.938297 27858 catalog_manager.cc:5484] T 5ae8cfd5411d489e96cbf21da84401ac P ced21ca983b749d0b4d4f6a0e2426bf8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71). New cstate: current_term: 1 leader_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:07.036710 27858 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_extra_table_60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:13:07.046029 31006 tablet_service.cc:1467] Processing CreateTablet for tablet 74f43feb3bc342a18723bc5a143bdd99 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=2979b3f65625430db5d8039280a09b13]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:07.046451 31006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74f43feb3bc342a18723bc5a143bdd99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:07.047580 30741 tablet_service.cc:1467] Processing CreateTablet for tablet e3a6b72060804beb81a177988d287f63 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=2979b3f65625430db5d8039280a09b13]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:07.048007 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3a6b72060804beb81a177988d287f63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:07.048331 31140 tablet_service.cc:1467] Processing CreateTablet for tablet 74f43feb3bc342a18723bc5a143bdd99 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=2979b3f65625430db5d8039280a09b13]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:07.048735 31140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74f43feb3bc342a18723bc5a143bdd99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:07.049175 28372 tablet_service.cc:1467] Processing CreateTablet for tablet 03b02a4425784a30b1007b3a14139d8d (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=2979b3f65625430db5d8039280a09b13]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:07.049538 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03b02a4425784a30b1007b3a14139d8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:07.050388 27859 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:13:07.052299 28373 tablet_service.cc:1467] Processing CreateTablet for tablet e3a6b72060804beb81a177988d287f63 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=2979b3f65625430db5d8039280a09b13]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:07.052660 28373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3a6b72060804beb81a177988d287f63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:07.053671 28113 tablet_service.cc:1467] Processing CreateTablet for tablet 74f43feb3bc342a18723bc5a143bdd99 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=2979b3f65625430db5d8039280a09b13]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:07.054049 28113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74f43feb3bc342a18723bc5a143bdd99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:07.054132 31007 tablet_service.cc:1467] Processing CreateTablet for tablet 03b02a4425784a30b1007b3a14139d8d (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=2979b3f65625430db5d8039280a09b13]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:07.054490 31007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03b02a4425784a30b1007b3a14139d8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:07.059572 394 tablet_bootstrap.cc:492] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:13:07.061064 394 tablet_bootstrap.cc:654] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:07.063011 396 tablet_bootstrap.cc:492] T 03b02a4425784a30b1007b3a14139d8d P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:13:07.064561 396 tablet_bootstrap.cc:654] T 03b02a4425784a30b1007b3a14139d8d P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:07.066356 27983 tablet_service.cc:1467] Processing CreateTablet for tablet e3a6b72060804beb81a177988d287f63 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=2979b3f65625430db5d8039280a09b13]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:07.066761 27983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3a6b72060804beb81a177988d287f63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:07.082245 28111 tablet_service.cc:1467] Processing CreateTablet for tablet 03b02a4425784a30b1007b3a14139d8d (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=2979b3f65625430db5d8039280a09b13]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:07.082717 28111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03b02a4425784a30b1007b3a14139d8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:07.086472 418 tablet_bootstrap.cc:492] T e3a6b72060804beb81a177988d287f63 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:13:07.090390 416 tablet_bootstrap.cc:492] T e3a6b72060804beb81a177988d287f63 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:13:07.091979 417 tablet_bootstrap.cc:492] T 74f43feb3bc342a18723bc5a143bdd99 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:13:07.093261 416 tablet_bootstrap.cc:654] T e3a6b72060804beb81a177988d287f63 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:07.094029 396 tablet_bootstrap.cc:492] T 03b02a4425784a30b1007b3a14139d8d P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:13:07.094228 396 ts_tablet_manager.cc:1367] T 03b02a4425784a30b1007b3a14139d8d P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.000s
I20240503 08:13:07.095028 419 tablet_bootstrap.cc:492] T 03b02a4425784a30b1007b3a14139d8d P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:13:07.096351 419 tablet_bootstrap.cc:654] T 03b02a4425784a30b1007b3a14139d8d P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:07.096555 417 tablet_bootstrap.cc:654] T 74f43feb3bc342a18723bc5a143bdd99 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:07.097724 418 tablet_bootstrap.cc:654] T e3a6b72060804beb81a177988d287f63 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:07.100188 28389 raft_consensus.cc:1261] T 5ae8cfd5411d489e96cbf21da84401ac P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer ced21ca983b749d0b4d4f6a0e2426bf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:07.101320 30893 raft_consensus.cc:3037] T 5ae8cfd5411d489e96cbf21da84401ac P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:07.101549 412 consensus_queue.cc:1035] T 5ae8cfd5411d489e96cbf21da84401ac P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:07.102401 396 raft_consensus.cc:348] T 03b02a4425784a30b1007b3a14139d8d P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:07.102968 396 raft_consensus.cc:374] T 03b02a4425784a30b1007b3a14139d8d P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:07.103099 396 raft_consensus.cc:729] T 03b02a4425784a30b1007b3a14139d8d P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:13:07.103384 396 consensus_queue.cc:260] T 03b02a4425784a30b1007b3a14139d8d P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:07.104487 30893 raft_consensus.cc:1261] T 5ae8cfd5411d489e96cbf21da84401ac P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer ced21ca983b749d0b4d4f6a0e2426bf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:07.105787 396 ts_tablet_manager.cc:1398] T 03b02a4425784a30b1007b3a14139d8d P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240503 08:13:07.106217 396 tablet_bootstrap.cc:492] T 74f43feb3bc342a18723bc5a143bdd99 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:13:07.109826 412 consensus_queue.cc:1035] T 5ae8cfd5411d489e96cbf21da84401ac P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:07.111884 396 tablet_bootstrap.cc:654] T 74f43feb3bc342a18723bc5a143bdd99 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:07.121301 419 tablet_bootstrap.cc:492] T 03b02a4425784a30b1007b3a14139d8d P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:13:07.121275 418 tablet_bootstrap.cc:492] T e3a6b72060804beb81a177988d287f63 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:13:07.121472 419 ts_tablet_manager.cc:1367] T 03b02a4425784a30b1007b3a14139d8d P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.000s
I20240503 08:13:07.121546 418 ts_tablet_manager.cc:1367] T e3a6b72060804beb81a177988d287f63 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.004s
I20240503 08:13:07.122292 27859 catalog_manager.cc:5576] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747) does not have the latest schema for tablet 5ae8cfd5411d489e96cbf21da84401ac (table rebalancer_dynamic_table [id=912224c75f7d47e1aa74e2c6024759b9]). Expected version 1 got 0
I20240503 08:13:07.122464 419 raft_consensus.cc:348] T 03b02a4425784a30b1007b3a14139d8d P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:07.122818 419 raft_consensus.cc:374] T 03b02a4425784a30b1007b3a14139d8d P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:07.122932 419 raft_consensus.cc:729] T 03b02a4425784a30b1007b3a14139d8d P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:13:07.123190 419 consensus_queue.cc:260] T 03b02a4425784a30b1007b3a14139d8d P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:07.123517 418 raft_consensus.cc:348] T e3a6b72060804beb81a177988d287f63 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:07.123795 418 raft_consensus.cc:374] T e3a6b72060804beb81a177988d287f63 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:07.123914 418 raft_consensus.cc:729] T e3a6b72060804beb81a177988d287f63 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:13:07.124171 394 tablet_bootstrap.cc:492] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:13:07.124214 418 consensus_queue.cc:260] T e3a6b72060804beb81a177988d287f63 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:07.124399 394 ts_tablet_manager.cc:1367] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.065s user 0.010s sys 0.008s
I20240503 08:13:07.125993 394 raft_consensus.cc:348] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:07.127233 405 mvcc.cc:205] Tried to move back new op lower bound from 7023509451118084096 to 7023509450486484992. Current Snapshot: MvccSnapshot[applied={T|T < 7023509451118084096}]
I20240503 08:13:07.127800 396 tablet_bootstrap.cc:492] T 74f43feb3bc342a18723bc5a143bdd99 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:13:07.127933 396 ts_tablet_manager.cc:1367] T 74f43feb3bc342a18723bc5a143bdd99 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.002s
I20240503 08:13:07.128625 396 raft_consensus.cc:348] T 74f43feb3bc342a18723bc5a143bdd99 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:07.128820 396 raft_consensus.cc:374] T 74f43feb3bc342a18723bc5a143bdd99 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:07.128914 396 raft_consensus.cc:729] T 74f43feb3bc342a18723bc5a143bdd99 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:13:07.129892 396 consensus_queue.cc:260] T 74f43feb3bc342a18723bc5a143bdd99 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:07.130410 396 ts_tablet_manager.cc:1398] T 74f43feb3bc342a18723bc5a143bdd99 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:07.130561 27858 catalog_manager.cc:5576] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211) does not have the latest schema for tablet 5ae8cfd5411d489e96cbf21da84401ac (table rebalancer_dynamic_table [id=912224c75f7d47e1aa74e2c6024759b9]). Expected version 1 got 0
I20240503 08:13:07.131250 417 tablet_bootstrap.cc:492] T 74f43feb3bc342a18723bc5a143bdd99 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:13:07.131492 417 ts_tablet_manager.cc:1367] T 74f43feb3bc342a18723bc5a143bdd99 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.005s
I20240503 08:13:07.132490 417 raft_consensus.cc:348] T 74f43feb3bc342a18723bc5a143bdd99 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:07.132617 418 ts_tablet_manager.cc:1398] T e3a6b72060804beb81a177988d287f63 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240503 08:13:07.132687 417 raft_consensus.cc:374] T 74f43feb3bc342a18723bc5a143bdd99 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:07.132769 417 raft_consensus.cc:729] T 74f43feb3bc342a18723bc5a143bdd99 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:13:07.133042 417 consensus_queue.cc:260] T 74f43feb3bc342a18723bc5a143bdd99 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:07.135424 417 ts_tablet_manager.cc:1398] T 74f43feb3bc342a18723bc5a143bdd99 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 08:13:07.145218 394 raft_consensus.cc:374] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:07.145618 394 raft_consensus.cc:729] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:13:07.149895 394 consensus_queue.cc:260] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:07.150888 32478 raft_consensus.cc:482] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:07.151038 32478 raft_consensus.cc:504] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:07.152026 32478 leader_election.cc:288] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:13:07.152591 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b02a4425784a30b1007b3a14139d8d" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:13:07.152768 28132 raft_consensus.cc:2454] T 03b02a4425784a30b1007b3a14139d8d P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:13:07.153249 28328 leader_election.cc:302] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:13:07.154037 32478 raft_consensus.cc:2781] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:07.154122 32478 raft_consensus.cc:482] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:07.154225 32478 raft_consensus.cc:3037] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:07.155084 419 ts_tablet_manager.cc:1398] T 03b02a4425784a30b1007b3a14139d8d P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.033s user 0.001s sys 0.000s
I20240503 08:13:07.156862 32478 raft_consensus.cc:504] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:07.158035 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b02a4425784a30b1007b3a14139d8d" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:13:07.158179 28132 raft_consensus.cc:3037] T 03b02a4425784a30b1007b3a14139d8d P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:07.160699 28132 raft_consensus.cc:2454] T 03b02a4425784a30b1007b3a14139d8d P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:13:07.160840 419 tablet_bootstrap.cc:492] T 74f43feb3bc342a18723bc5a143bdd99 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:13:07.161274 28328 leader_election.cc:302] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:13:07.161545 378 raft_consensus.cc:2781] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:07.164467 31026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b02a4425784a30b1007b3a14139d8d" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:13:07.164559 419 tablet_bootstrap.cc:654] T 74f43feb3bc342a18723bc5a143bdd99 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:07.164687 31026 raft_consensus.cc:2454] T 03b02a4425784a30b1007b3a14139d8d P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:13:07.167551 428 tablet.cc:1722] T 5ae8cfd5411d489e96cbf21da84401ac P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:13:07.168162 428 tablet.cc:1620] T 5ae8cfd5411d489e96cbf21da84401ac P ce1496a47d084365ad786cd09e472a2c: MemRowSet was empty: no flush needed.
I20240503 08:13:07.168568 31026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b02a4425784a30b1007b3a14139d8d" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:13:07.168764 31026 raft_consensus.cc:3037] T 03b02a4425784a30b1007b3a14139d8d P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:07.170199 427 tablet.cc:1722] T 5ae8cfd5411d489e96cbf21da84401ac P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:13:07.170712 427 tablet.cc:1620] T 5ae8cfd5411d489e96cbf21da84401ac P ced21ca983b749d0b4d4f6a0e2426bf8: MemRowSet was empty: no flush needed.
I20240503 08:13:07.171427 31026 raft_consensus.cc:2454] T 03b02a4425784a30b1007b3a14139d8d P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:13:07.175707 32478 leader_election.cc:288] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Requested vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:13:07.177379 27859 catalog_manager.cc:5576] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747) does not have the latest schema for tablet 5ae8cfd5411d489e96cbf21da84401ac (table rebalancer_dynamic_table [id=912224c75f7d47e1aa74e2c6024759b9]). Expected version 1 got 0
I20240503 08:13:07.179514 419 tablet_bootstrap.cc:492] T 74f43feb3bc342a18723bc5a143bdd99 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:13:07.179687 419 ts_tablet_manager.cc:1367] T 74f43feb3bc342a18723bc5a143bdd99 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20240503 08:13:07.182857 419 raft_consensus.cc:348] T 74f43feb3bc342a18723bc5a143bdd99 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:07.183102 419 raft_consensus.cc:374] T 74f43feb3bc342a18723bc5a143bdd99 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:07.183190 419 raft_consensus.cc:729] T 74f43feb3bc342a18723bc5a143bdd99 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:13:07.183463 419 consensus_queue.cc:260] T 74f43feb3bc342a18723bc5a143bdd99 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:07.185966 27859 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=912224c75f7d47e1aa74e2c6024759b9] alter complete (version 1)
I20240503 08:13:07.187927 429 tablet.cc:1722] T 5ae8cfd5411d489e96cbf21da84401ac P d09a11a03989421b9f28019dc1c14cc2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:13:07.188092 419 ts_tablet_manager.cc:1398] T 74f43feb3bc342a18723bc5a143bdd99 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240503 08:13:07.188484 429 tablet.cc:1620] T 5ae8cfd5411d489e96cbf21da84401ac P d09a11a03989421b9f28019dc1c14cc2: MemRowSet was empty: no flush needed.
I20240503 08:13:07.189214 378 raft_consensus.cc:686] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Becoming Leader. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Running, Role: LEADER
I20240503 08:13:07.189890 378 consensus_queue.cc:237] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:07.190290 416 tablet_bootstrap.cc:492] T e3a6b72060804beb81a177988d287f63 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:13:07.190531 416 ts_tablet_manager.cc:1367] T e3a6b72060804beb81a177988d287f63 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.102s user 0.002s sys 0.004s
I20240503 08:13:07.192284 394 ts_tablet_manager.cc:1398] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.068s user 0.002s sys 0.000s
I20240503 08:13:07.192586 394 tablet_bootstrap.cc:492] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:13:07.192219 416 raft_consensus.cc:348] T e3a6b72060804beb81a177988d287f63 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:07.193063 416 raft_consensus.cc:374] T e3a6b72060804beb81a177988d287f63 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:07.193349 416 raft_consensus.cc:729] T e3a6b72060804beb81a177988d287f63 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:13:07.193593 416 consensus_queue.cc:260] T e3a6b72060804beb81a177988d287f63 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:07.194185 27858 catalog_manager.cc:5576] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211) does not have the latest schema for tablet 5ae8cfd5411d489e96cbf21da84401ac (table rebalancer_dynamic_table [id=912224c75f7d47e1aa74e2c6024759b9]). Expected version 1 got 0
I20240503 08:13:07.195314 416 ts_tablet_manager.cc:1398] T e3a6b72060804beb81a177988d287f63 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 08:13:07.196748 394 tablet_bootstrap.cc:654] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:07.198513 27858 catalog_manager.cc:5484] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68). New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:13:07.207875 394 tablet_bootstrap.cc:492] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:13:07.208063 394 ts_tablet_manager.cc:1367] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.002s
I20240503 08:13:07.210016 394 raft_consensus.cc:348] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:07.210222 394 raft_consensus.cc:374] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:07.210322 394 raft_consensus.cc:729] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:13:07.210527 394 consensus_queue.cc:260] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:07.219414 394 ts_tablet_manager.cc:1398] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20240503 08:13:07.255718 412 raft_consensus.cc:482] T 74f43feb3bc342a18723bc5a143bdd99 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:07.255920 412 raft_consensus.cc:504] T 74f43feb3bc342a18723bc5a143bdd99 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:07.256561 412 leader_election.cc:288] T 74f43feb3bc342a18723bc5a143bdd99 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:13:07.256933 28133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f43feb3bc342a18723bc5a143bdd99" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:13:07.257160 28133 raft_consensus.cc:2454] T 74f43feb3bc342a18723bc5a143bdd99 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 0.
I20240503 08:13:07.257112 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f43feb3bc342a18723bc5a143bdd99" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:13:07.257352 31159 raft_consensus.cc:2454] T 74f43feb3bc342a18723bc5a143bdd99 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 0.
I20240503 08:13:07.257625 30961 leader_election.cc:302] T 74f43feb3bc342a18723bc5a143bdd99 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:13:07.257977 412 raft_consensus.cc:2781] T 74f43feb3bc342a18723bc5a143bdd99 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:07.258088 412 raft_consensus.cc:482] T 74f43feb3bc342a18723bc5a143bdd99 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:07.258152 412 raft_consensus.cc:3037] T 74f43feb3bc342a18723bc5a143bdd99 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:07.260277 412 raft_consensus.cc:504] T 74f43feb3bc342a18723bc5a143bdd99 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:07.260830 412 leader_election.cc:288] T 74f43feb3bc342a18723bc5a143bdd99 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 election: Requested vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:13:07.261101 28133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f43feb3bc342a18723bc5a143bdd99" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:13:07.261225 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f43feb3bc342a18723bc5a143bdd99" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:13:07.261269 28133 raft_consensus.cc:3037] T 74f43feb3bc342a18723bc5a143bdd99 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:07.261423 31159 raft_consensus.cc:3037] T 74f43feb3bc342a18723bc5a143bdd99 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:07.263896 28133 raft_consensus.cc:2454] T 74f43feb3bc342a18723bc5a143bdd99 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 1.
I20240503 08:13:07.264113 31159 raft_consensus.cc:2454] T 74f43feb3bc342a18723bc5a143bdd99 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 1.
I20240503 08:13:07.264241 30961 leader_election.cc:302] T 74f43feb3bc342a18723bc5a143bdd99 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:13:07.264634 412 raft_consensus.cc:2781] T 74f43feb3bc342a18723bc5a143bdd99 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:07.264761 412 raft_consensus.cc:686] T 74f43feb3bc342a18723bc5a143bdd99 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 LEADER]: Becoming Leader. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Running, Role: LEADER
I20240503 08:13:07.265192 412 consensus_queue.cc:237] T 74f43feb3bc342a18723bc5a143bdd99 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:07.268576 27854 catalog_manager.cc:5484] T 74f43feb3bc342a18723bc5a143bdd99 P ced21ca983b749d0b4d4f6a0e2426bf8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71). New cstate: current_term: 1 leader_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:13:07.280599 378 raft_consensus.cc:482] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:07.280746 378 raft_consensus.cc:504] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:07.281349 378 leader_election.cc:288] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:13:07.281819 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3a6b72060804beb81a177988d287f63" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:13:07.282007 30762 raft_consensus.cc:2454] T e3a6b72060804beb81a177988d287f63 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:13:07.281962 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3a6b72060804beb81a177988d287f63" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:13:07.282171 28002 raft_consensus.cc:2454] T e3a6b72060804beb81a177988d287f63 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:13:07.282294 28328 leader_election.cc:302] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:13:07.282522 378 raft_consensus.cc:2781] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:07.282598 378 raft_consensus.cc:482] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:07.282662 378 raft_consensus.cc:3037] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:07.284302 378 raft_consensus.cc:504] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:07.284812 378 leader_election.cc:288] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Requested vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:13:07.285240 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3a6b72060804beb81a177988d287f63" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:13:07.285295 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3a6b72060804beb81a177988d287f63" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:13:07.285444 28002 raft_consensus.cc:3037] T e3a6b72060804beb81a177988d287f63 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:07.285444 30762 raft_consensus.cc:3037] T e3a6b72060804beb81a177988d287f63 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:07.288297 30762 raft_consensus.cc:2454] T e3a6b72060804beb81a177988d287f63 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:13:07.288735 28328 leader_election.cc:302] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:13:07.288833 28002 raft_consensus.cc:2454] T e3a6b72060804beb81a177988d287f63 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:13:07.288995 378 raft_consensus.cc:2781] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:07.289095 378 raft_consensus.cc:686] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Becoming Leader. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Running, Role: LEADER
I20240503 08:13:07.289248 378 consensus_queue.cc:237] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:07.292348 27858 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:13:07.294178 27854 catalog_manager.cc:5484] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68). New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:07.300374 427 tablet.cc:1722] T 5ae8cfd5411d489e96cbf21da84401ac P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:13:07.300845 427 tablet.cc:1620] T 5ae8cfd5411d489e96cbf21da84401ac P ced21ca983b749d0b4d4f6a0e2426bf8: MemRowSet was empty: no flush needed.
I20240503 08:13:07.300952 429 tablet.cc:1722] T 5ae8cfd5411d489e96cbf21da84401ac P d09a11a03989421b9f28019dc1c14cc2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:13:07.301313 429 tablet.cc:1620] T 5ae8cfd5411d489e96cbf21da84401ac P d09a11a03989421b9f28019dc1c14cc2: MemRowSet was empty: no flush needed.
I20240503 08:13:07.302608 428 tablet.cc:1722] T 5ae8cfd5411d489e96cbf21da84401ac P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:13:07.303006 428 tablet.cc:1620] T 5ae8cfd5411d489e96cbf21da84401ac P ce1496a47d084365ad786cd09e472a2c: MemRowSet was empty: no flush needed.
I20240503 08:13:07.315392 27854 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=912224c75f7d47e1aa74e2c6024759b9] alter complete (version 2)
I20240503 08:13:07.418175 27862 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:13:07.425277 427 tablet.cc:1722] T 5ae8cfd5411d489e96cbf21da84401ac P ced21ca983b749d0b4d4f6a0e2426bf8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:13:07.426218 429 tablet.cc:1722] T 5ae8cfd5411d489e96cbf21da84401ac P d09a11a03989421b9f28019dc1c14cc2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:13:07.430999 428 tablet.cc:1722] T 5ae8cfd5411d489e96cbf21da84401ac P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:13:07.435082 27862 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table_new_name [id=912224c75f7d47e1aa74e2c6024759b9] alter complete (version 3)
I20240503 08:13:07.539381 27854 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:13:07.539605 27854 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:13:07.543051 27854 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet 5ae8cfd5411d489e96cbf21da84401ac
I20240503 08:13:07.543681 31006 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae8cfd5411d489e96cbf21da84401ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:07 UTC) from {username='slave'} at 127.0.0.1:47020
I20240503 08:13:07.544303 436 tablet_replica.cc:330] T 5ae8cfd5411d489e96cbf21da84401ac P ced21ca983b749d0b4d4f6a0e2426bf8: stopping tablet replica
I20240503 08:13:07.544250 30874 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae8cfd5411d489e96cbf21da84401ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:07 UTC) from {username='slave'} at 127.0.0.1:34426
I20240503 08:13:07.544605 28372 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae8cfd5411d489e96cbf21da84401ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:07 UTC) from {username='slave'} at 127.0.0.1:34786
I20240503 08:13:07.545928 436 raft_consensus.cc:2229] T 5ae8cfd5411d489e96cbf21da84401ac P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:13:07.546269 436 raft_consensus.cc:2258] T 5ae8cfd5411d489e96cbf21da84401ac P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:07.546888 438 tablet_replica.cc:330] T 5ae8cfd5411d489e96cbf21da84401ac P d09a11a03989421b9f28019dc1c14cc2: stopping tablet replica
I20240503 08:13:07.547072 438 raft_consensus.cc:2229] T 5ae8cfd5411d489e96cbf21da84401ac P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:07.547236 438 raft_consensus.cc:2258] T 5ae8cfd5411d489e96cbf21da84401ac P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:07.547971 438 ts_tablet_manager.cc:1875] T 5ae8cfd5411d489e96cbf21da84401ac P d09a11a03989421b9f28019dc1c14cc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:07.548043 436 ts_tablet_manager.cc:1875] T 5ae8cfd5411d489e96cbf21da84401ac P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:07.554183 436 ts_tablet_manager.cc:1888] T 5ae8cfd5411d489e96cbf21da84401ac P ced21ca983b749d0b4d4f6a0e2426bf8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:13:07.554407 436 log.cc:1196] T 5ae8cfd5411d489e96cbf21da84401ac P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal/wals/5ae8cfd5411d489e96cbf21da84401ac
I20240503 08:13:07.554929 436 ts_tablet_manager.cc:1909] T 5ae8cfd5411d489e96cbf21da84401ac P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting consensus metadata
I20240503 08:13:07.555058 438 ts_tablet_manager.cc:1888] T 5ae8cfd5411d489e96cbf21da84401ac P d09a11a03989421b9f28019dc1c14cc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:13:07.555233 438 log.cc:1196] T 5ae8cfd5411d489e96cbf21da84401ac P d09a11a03989421b9f28019dc1c14cc2: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/wals/5ae8cfd5411d489e96cbf21da84401ac
I20240503 08:13:07.555593 438 ts_tablet_manager.cc:1909] T 5ae8cfd5411d489e96cbf21da84401ac P d09a11a03989421b9f28019dc1c14cc2: Deleting consensus metadata
I20240503 08:13:07.555975 27847 catalog_manager.cc:4830] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): tablet 5ae8cfd5411d489e96cbf21da84401ac (table rebalancer_dynamic_table_new_name [id=912224c75f7d47e1aa74e2c6024759b9]) successfully deleted
I20240503 08:13:07.557097 27849 catalog_manager.cc:4830] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): tablet 5ae8cfd5411d489e96cbf21da84401ac (table rebalancer_dynamic_table_new_name [id=912224c75f7d47e1aa74e2c6024759b9]) successfully deleted
I20240503 08:13:07.559608 437 tablet_replica.cc:330] T 5ae8cfd5411d489e96cbf21da84401ac P ce1496a47d084365ad786cd09e472a2c: stopping tablet replica
I20240503 08:13:07.559799 437 raft_consensus.cc:2229] T 5ae8cfd5411d489e96cbf21da84401ac P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:07.559998 437 raft_consensus.cc:2258] T 5ae8cfd5411d489e96cbf21da84401ac P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:07.560647 437 ts_tablet_manager.cc:1875] T 5ae8cfd5411d489e96cbf21da84401ac P ce1496a47d084365ad786cd09e472a2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:07.564913 437 ts_tablet_manager.cc:1888] T 5ae8cfd5411d489e96cbf21da84401ac P ce1496a47d084365ad786cd09e472a2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:13:07.565012 437 log.cc:1196] T 5ae8cfd5411d489e96cbf21da84401ac P ce1496a47d084365ad786cd09e472a2c: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal/wals/5ae8cfd5411d489e96cbf21da84401ac
I20240503 08:13:07.565373 437 ts_tablet_manager.cc:1909] T 5ae8cfd5411d489e96cbf21da84401ac P ce1496a47d084365ad786cd09e472a2c: Deleting consensus metadata
I20240503 08:13:07.566215 27846 catalog_manager.cc:4830] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): tablet 5ae8cfd5411d489e96cbf21da84401ac (table rebalancer_dynamic_table_new_name [id=912224c75f7d47e1aa74e2c6024759b9]) successfully deleted
I20240503 08:13:07.646067 27854 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:13:07.652521 430 consensus_queue.cc:1035] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:07.654039 28243 tablet_service.cc:1467] Processing CreateTablet for tablet e352522f8a584822891f68c758971757 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1851668b571346c59948f2019df38c35]), partition=
I20240503 08:13:07.654489 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e352522f8a584822891f68c758971757. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:07.655026 30874 tablet_service.cc:1467] Processing CreateTablet for tablet e352522f8a584822891f68c758971757 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1851668b571346c59948f2019df38c35]), partition=
I20240503 08:13:07.655328 30874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e352522f8a584822891f68c758971757. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:07.655946 31140 tablet_service.cc:1467] Processing CreateTablet for tablet e352522f8a584822891f68c758971757 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1851668b571346c59948f2019df38c35]), partition=
I20240503 08:13:07.656301 31140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e352522f8a584822891f68c758971757. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:07.661634 430 consensus_queue.cc:1035] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:07.661805 441 tablet_bootstrap.cc:492] T e352522f8a584822891f68c758971757 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:13:07.663370 441 tablet_bootstrap.cc:654] T e352522f8a584822891f68c758971757 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:07.672207 441 tablet_bootstrap.cc:492] T e352522f8a584822891f68c758971757 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:13:07.672428 441 ts_tablet_manager.cc:1367] T e352522f8a584822891f68c758971757 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.010s
I20240503 08:13:07.673345 441 raft_consensus.cc:348] T e352522f8a584822891f68c758971757 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:07.673611 441 raft_consensus.cc:374] T e352522f8a584822891f68c758971757 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:07.673781 441 raft_consensus.cc:729] T e352522f8a584822891f68c758971757 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:13:07.674078 441 consensus_queue.cc:260] T e352522f8a584822891f68c758971757 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:07.678563 441 ts_tablet_manager.cc:1398] T e352522f8a584822891f68c758971757 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:13:07.696990 442 tablet_bootstrap.cc:492] T e352522f8a584822891f68c758971757 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:13:07.700541 446 tablet_bootstrap.cc:492] T e352522f8a584822891f68c758971757 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:13:07.701929 446 tablet_bootstrap.cc:654] T e352522f8a584822891f68c758971757 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:07.704340 446 tablet_bootstrap.cc:492] T e352522f8a584822891f68c758971757 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:13:07.704455 446 ts_tablet_manager.cc:1367] T e352522f8a584822891f68c758971757 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 08:13:07.705040 446 raft_consensus.cc:348] T e352522f8a584822891f68c758971757 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:07.705191 446 raft_consensus.cc:374] T e352522f8a584822891f68c758971757 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:07.705256 446 raft_consensus.cc:729] T e352522f8a584822891f68c758971757 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:13:07.705518 446 consensus_queue.cc:260] T e352522f8a584822891f68c758971757 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:07.706226 446 ts_tablet_manager.cc:1398] T e352522f8a584822891f68c758971757 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:07.714208 442 tablet_bootstrap.cc:654] T e352522f8a584822891f68c758971757 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:07.715773 442 tablet_bootstrap.cc:492] T e352522f8a584822891f68c758971757 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:13:07.715907 442 ts_tablet_manager.cc:1367] T e352522f8a584822891f68c758971757 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.003s
I20240503 08:13:07.716428 442 raft_consensus.cc:348] T e352522f8a584822891f68c758971757 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:07.716581 442 raft_consensus.cc:374] T e352522f8a584822891f68c758971757 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:07.716674 442 raft_consensus.cc:729] T e352522f8a584822891f68c758971757 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:13:07.716903 442 consensus_queue.cc:260] T e352522f8a584822891f68c758971757 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:07.717571 442 ts_tablet_manager.cc:1398] T e352522f8a584822891f68c758971757 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:13:07.769948 32493 raft_consensus.cc:482] T e352522f8a584822891f68c758971757 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:07.770110 32493 raft_consensus.cc:504] T e352522f8a584822891f68c758971757 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:07.770732 32493 leader_election.cc:288] T e352522f8a584822891f68c758971757 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:13:07.771059 30893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e352522f8a584822891f68c758971757" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:13:07.771086 28263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e352522f8a584822891f68c758971757" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:13:07.771251 30893 raft_consensus.cc:2454] T e352522f8a584822891f68c758971757 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 0.
I20240503 08:13:07.771291 28263 raft_consensus.cc:2454] T e352522f8a584822891f68c758971757 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 0.
I20240503 08:13:07.771572 31094 leader_election.cc:302] T e352522f8a584822891f68c758971757 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:13:07.771831 32493 raft_consensus.cc:2781] T e352522f8a584822891f68c758971757 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:07.771947 32493 raft_consensus.cc:482] T e352522f8a584822891f68c758971757 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:07.772053 32493 raft_consensus.cc:3037] T e352522f8a584822891f68c758971757 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:07.774243 32493 raft_consensus.cc:504] T e352522f8a584822891f68c758971757 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:07.774729 32493 leader_election.cc:288] T e352522f8a584822891f68c758971757 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 election: Requested vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:13:07.775030 30893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e352522f8a584822891f68c758971757" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:13:07.775104 28263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e352522f8a584822891f68c758971757" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:13:07.775206 30893 raft_consensus.cc:3037] T e352522f8a584822891f68c758971757 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:07.775288 28263 raft_consensus.cc:3037] T e352522f8a584822891f68c758971757 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:07.776989 30893 raft_consensus.cc:2454] T e352522f8a584822891f68c758971757 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 1.
I20240503 08:13:07.777171 28263 raft_consensus.cc:2454] T e352522f8a584822891f68c758971757 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 1.
I20240503 08:13:07.777348 31094 leader_election.cc:302] T e352522f8a584822891f68c758971757 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:13:07.777717 32493 raft_consensus.cc:2781] T e352522f8a584822891f68c758971757 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:07.777858 32493 raft_consensus.cc:686] T e352522f8a584822891f68c758971757 P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Becoming Leader. State: Replica: b00f244c69df401997ea99d492218a98, State: Running, Role: LEADER
I20240503 08:13:07.778106 32493 consensus_queue.cc:237] T e352522f8a584822891f68c758971757 P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:07.781749 27854 catalog_manager.cc:5484] T e352522f8a584822891f68c758971757 P b00f244c69df401997ea99d492218a98 reported cstate change: term changed from 0 to 1, leader changed from <none> to b00f244c69df401997ea99d492218a98 (127.20.230.72). New cstate: current_term: 1 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:13:07.793880 378 consensus_queue.cc:1035] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:07.799225 32478 consensus_queue.cc:1035] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:07.799288 412 consensus_queue.cc:1035] T 74f43feb3bc342a18723bc5a143bdd99 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:07.804281 412 consensus_queue.cc:1035] T 74f43feb3bc342a18723bc5a143bdd99 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:07.838716 27854 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_extra_table_62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:13:07.847735 30741 tablet_service.cc:1467] Processing CreateTablet for tablet e30557ffd4594f7d9cda13025e4c5e60 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=7adc395c24f24d63aa9f9ceed2c44c01]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:07.848199 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e30557ffd4594f7d9cda13025e4c5e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:07.848299 30874 tablet_service.cc:1467] Processing CreateTablet for tablet f7a8c95ac81942dabef98a07caa84082 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=7adc395c24f24d63aa9f9ceed2c44c01]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:07.848358 28111 tablet_service.cc:1467] Processing CreateTablet for tablet f7a8c95ac81942dabef98a07caa84082 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=7adc395c24f24d63aa9f9ceed2c44c01]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:07.848680 30874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7a8c95ac81942dabef98a07caa84082. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:07.848731 28111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7a8c95ac81942dabef98a07caa84082. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:07.850574 31006 tablet_service.cc:1467] Processing CreateTablet for tablet e30557ffd4594f7d9cda13025e4c5e60 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=7adc395c24f24d63aa9f9ceed2c44c01]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:07.850960 31006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e30557ffd4594f7d9cda13025e4c5e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:07.858361 27983 tablet_service.cc:1467] Processing CreateTablet for tablet e30557ffd4594f7d9cda13025e4c5e60 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=7adc395c24f24d63aa9f9ceed2c44c01]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:07.858770 27983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e30557ffd4594f7d9cda13025e4c5e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:07.859171 30742 tablet_service.cc:1467] Processing CreateTablet for tablet 7f2743ab5fa24c9899b67ea52f48f9d8 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=7adc395c24f24d63aa9f9ceed2c44c01]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:07.859553 30742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f2743ab5fa24c9899b67ea52f48f9d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:07.862144 28243 tablet_service.cc:1467] Processing CreateTablet for tablet f7a8c95ac81942dabef98a07caa84082 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=7adc395c24f24d63aa9f9ceed2c44c01]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:07.862166 28242 tablet_service.cc:1467] Processing CreateTablet for tablet 7f2743ab5fa24c9899b67ea52f48f9d8 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=7adc395c24f24d63aa9f9ceed2c44c01]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:07.862617 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7a8c95ac81942dabef98a07caa84082. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:07.862890 28242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f2743ab5fa24c9899b67ea52f48f9d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:07.866123 27981 tablet_service.cc:1467] Processing CreateTablet for tablet 7f2743ab5fa24c9899b67ea52f48f9d8 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=7adc395c24f24d63aa9f9ceed2c44c01]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:07.866485 27981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f2743ab5fa24c9899b67ea52f48f9d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:07.869122 442 tablet_bootstrap.cc:492] T f7a8c95ac81942dabef98a07caa84082 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:13:07.870689 442 tablet_bootstrap.cc:654] T f7a8c95ac81942dabef98a07caa84082 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:07.874281 461 tablet_bootstrap.cc:492] T f7a8c95ac81942dabef98a07caa84082 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:13:07.875680 461 tablet_bootstrap.cc:654] T f7a8c95ac81942dabef98a07caa84082 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:07.878275 458 tablet_bootstrap.cc:492] T e30557ffd4594f7d9cda13025e4c5e60 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:13:07.879037 442 tablet_bootstrap.cc:492] T f7a8c95ac81942dabef98a07caa84082 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:13:07.879793 442 ts_tablet_manager.cc:1367] T f7a8c95ac81942dabef98a07caa84082 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.002s
I20240503 08:13:07.880590 442 raft_consensus.cc:348] T f7a8c95ac81942dabef98a07caa84082 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:07.880815 442 raft_consensus.cc:374] T f7a8c95ac81942dabef98a07caa84082 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:07.880884 442 raft_consensus.cc:729] T f7a8c95ac81942dabef98a07caa84082 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:13:07.881094 442 consensus_queue.cc:260] T f7a8c95ac81942dabef98a07caa84082 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:07.881647 442 ts_tablet_manager.cc:1398] T f7a8c95ac81942dabef98a07caa84082 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:13:07.883670 458 tablet_bootstrap.cc:654] T e30557ffd4594f7d9cda13025e4c5e60 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:07.886335 27858 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:13:07.887420 442 tablet_bootstrap.cc:492] T 7f2743ab5fa24c9899b67ea52f48f9d8 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:13:07.890902 442 tablet_bootstrap.cc:654] T 7f2743ab5fa24c9899b67ea52f48f9d8 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:07.892158 460 tablet_bootstrap.cc:492] T e30557ffd4594f7d9cda13025e4c5e60 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:13:07.895159 460 tablet_bootstrap.cc:654] T e30557ffd4594f7d9cda13025e4c5e60 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:07.897471 28263 raft_consensus.cc:1261] T e352522f8a584822891f68c758971757 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:07.897955 30895 raft_consensus.cc:1261] T e352522f8a584822891f68c758971757 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:07.898138 458 tablet_bootstrap.cc:492] T e30557ffd4594f7d9cda13025e4c5e60 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:13:07.898038 32493 consensus_queue.cc:1035] T e352522f8a584822891f68c758971757 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:07.898274 458 ts_tablet_manager.cc:1367] T e30557ffd4594f7d9cda13025e4c5e60 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.003s
I20240503 08:13:07.900502 458 raft_consensus.cc:348] T e30557ffd4594f7d9cda13025e4c5e60 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:07.900796 355 consensus_queue.cc:1035] T e352522f8a584822891f68c758971757 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:07.900789 458 raft_consensus.cc:374] T e30557ffd4594f7d9cda13025e4c5e60 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:07.903002 458 raft_consensus.cc:729] T e30557ffd4594f7d9cda13025e4c5e60 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:13:07.903508 458 consensus_queue.cc:260] T e30557ffd4594f7d9cda13025e4c5e60 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:07.905414 441 tablet_bootstrap.cc:492] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:13:07.905784 458 ts_tablet_manager.cc:1398] T e30557ffd4594f7d9cda13025e4c5e60 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20240503 08:13:07.906044 458 tablet_bootstrap.cc:492] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:13:07.906953 441 tablet_bootstrap.cc:654] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:07.913902 467 tablet_bootstrap.cc:492] T e30557ffd4594f7d9cda13025e4c5e60 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:13:07.914330 27858 catalog_manager.cc:5576] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747) does not have the latest schema for tablet e352522f8a584822891f68c758971757 (table rebalancer_dynamic_table [id=1851668b571346c59948f2019df38c35]). Expected version 1 got 0
I20240503 08:13:07.915700 458 tablet_bootstrap.cc:654] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:07.918591 428 tablet.cc:1722] T e352522f8a584822891f68c758971757 P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:13:07.919040 428 tablet.cc:1620] T e352522f8a584822891f68c758971757 P ce1496a47d084365ad786cd09e472a2c: MemRowSet was empty: no flush needed.
I20240503 08:13:07.930048 27854 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=1851668b571346c59948f2019df38c35] alter complete (version 1)
I20240503 08:13:07.936069 27854 catalog_manager.cc:5576] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919) does not have the latest schema for tablet e352522f8a584822891f68c758971757 (table rebalancer_dynamic_table [id=1851668b571346c59948f2019df38c35]). Expected version 1 got 0
I20240503 08:13:07.941197 441 tablet_bootstrap.cc:492] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:13:07.941354 441 ts_tablet_manager.cc:1367] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.036s user 0.005s sys 0.014s
I20240503 08:13:07.941910 467 tablet_bootstrap.cc:654] T e30557ffd4594f7d9cda13025e4c5e60 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:07.942504 441 raft_consensus.cc:348] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:07.942791 441 raft_consensus.cc:374] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:07.942770 442 tablet_bootstrap.cc:492] T 7f2743ab5fa24c9899b67ea52f48f9d8 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:13:07.942917 441 raft_consensus.cc:729] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:13:07.942937 442 ts_tablet_manager.cc:1367] T 7f2743ab5fa24c9899b67ea52f48f9d8 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.056s user 0.004s sys 0.003s
I20240503 08:13:07.943230 441 consensus_queue.cc:260] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:07.943950 441 ts_tablet_manager.cc:1398] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:13:07.944371 442 raft_consensus.cc:348] T 7f2743ab5fa24c9899b67ea52f48f9d8 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:07.944589 442 raft_consensus.cc:374] T 7f2743ab5fa24c9899b67ea52f48f9d8 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:07.944777 442 raft_consensus.cc:729] T 7f2743ab5fa24c9899b67ea52f48f9d8 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:13:07.947522 472 tablet.cc:1722] T e352522f8a584822891f68c758971757 P b00f244c69df401997ea99d492218a98: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:13:07.945004 442 consensus_queue.cc:260] T 7f2743ab5fa24c9899b67ea52f48f9d8 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:07.949079 472 tablet.cc:1620] T e352522f8a584822891f68c758971757 P b00f244c69df401997ea99d492218a98: MemRowSet was empty: no flush needed.
I20240503 08:13:07.950399 460 tablet_bootstrap.cc:492] T e30557ffd4594f7d9cda13025e4c5e60 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:13:07.950619 460 ts_tablet_manager.cc:1367] T e30557ffd4594f7d9cda13025e4c5e60 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.059s user 0.004s sys 0.000s
I20240503 08:13:07.952795 442 ts_tablet_manager.cc:1398] T 7f2743ab5fa24c9899b67ea52f48f9d8 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240503 08:13:07.953543 460 raft_consensus.cc:348] T e30557ffd4594f7d9cda13025e4c5e60 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:07.954237 460 raft_consensus.cc:374] T e30557ffd4594f7d9cda13025e4c5e60 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:07.954456 460 raft_consensus.cc:729] T e30557ffd4594f7d9cda13025e4c5e60 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:13:07.954736 460 consensus_queue.cc:260] T e30557ffd4594f7d9cda13025e4c5e60 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:07.955657 458 tablet_bootstrap.cc:492] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:13:07.955868 458 ts_tablet_manager.cc:1367] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.050s user 0.000s sys 0.006s
I20240503 08:13:07.956277 374 raft_consensus.cc:482] T e30557ffd4594f7d9cda13025e4c5e60 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:07.956421 374 raft_consensus.cc:504] T e30557ffd4594f7d9cda13025e4c5e60 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:07.957147 374 leader_election.cc:288] T e30557ffd4594f7d9cda13025e4c5e60 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:13:07.957098 476 tablet.cc:1722] T e352522f8a584822891f68c758971757 P aae860d7ae2b4002bf7794c278f9cc56: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:13:07.958616 31026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30557ffd4594f7d9cda13025e4c5e60" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
W20240503 08:13:07.959300 30695 leader_election.cc:341] T e30557ffd4594f7d9cda13025e4c5e60 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:13:07.960114 458 raft_consensus.cc:348] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:07.960449 458 raft_consensus.cc:374] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:07.960572 458 raft_consensus.cc:729] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:13:07.960889 458 consensus_queue.cc:260] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:07.961416 476 tablet.cc:1620] T e352522f8a584822891f68c758971757 P aae860d7ae2b4002bf7794c278f9cc56: MemRowSet was empty: no flush needed.
I20240503 08:13:07.963109 27858 catalog_manager.cc:5576] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919) does not have the latest schema for tablet e352522f8a584822891f68c758971757 (table rebalancer_dynamic_table [id=1851668b571346c59948f2019df38c35]). Expected version 1 got 0
I20240503 08:13:07.963302 458 ts_tablet_manager.cc:1398] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240503 08:13:07.959820 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30557ffd4594f7d9cda13025e4c5e60" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:13:07.963891 28003 raft_consensus.cc:2454] T e30557ffd4594f7d9cda13025e4c5e60 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:13:07.964315 30697 leader_election.cc:302] T e30557ffd4594f7d9cda13025e4c5e60 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, 833582ff68c94269be364ebd187a75db; no voters: ced21ca983b749d0b4d4f6a0e2426bf8
I20240503 08:13:07.965801 374 raft_consensus.cc:2781] T e30557ffd4594f7d9cda13025e4c5e60 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:07.965921 374 raft_consensus.cc:482] T e30557ffd4594f7d9cda13025e4c5e60 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:07.965991 374 raft_consensus.cc:3037] T e30557ffd4594f7d9cda13025e4c5e60 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:07.969295 374 raft_consensus.cc:504] T e30557ffd4594f7d9cda13025e4c5e60 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:07.971889 31026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30557ffd4594f7d9cda13025e4c5e60" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
W20240503 08:13:07.972481 30695 leader_election.cc:341] T e30557ffd4594f7d9cda13025e4c5e60 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:13:07.973254 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30557ffd4594f7d9cda13025e4c5e60" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:13:07.973461 28003 raft_consensus.cc:3037] T e30557ffd4594f7d9cda13025e4c5e60 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:07.976300 28003 raft_consensus.cc:2454] T e30557ffd4594f7d9cda13025e4c5e60 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:13:07.976855 30697 leader_election.cc:302] T e30557ffd4594f7d9cda13025e4c5e60 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, 833582ff68c94269be364ebd187a75db; no voters: ced21ca983b749d0b4d4f6a0e2426bf8
I20240503 08:13:07.978814 374 leader_election.cc:288] T e30557ffd4594f7d9cda13025e4c5e60 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Requested vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:13:07.979156 374 raft_consensus.cc:2781] T e30557ffd4594f7d9cda13025e4c5e60 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:07.979591 374 raft_consensus.cc:686] T e30557ffd4594f7d9cda13025e4c5e60 P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Becoming Leader. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Running, Role: LEADER
I20240503 08:13:07.980041 374 consensus_queue.cc:237] T e30557ffd4594f7d9cda13025e4c5e60 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
W20240503 08:13:07.983487 31094 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.70:33747, user_credentials={real_user=slave}} blocked reactor thread for 64500.2us
I20240503 08:13:07.998306 460 ts_tablet_manager.cc:1398] T e30557ffd4594f7d9cda13025e4c5e60 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.045s user 0.002s sys 0.000s
I20240503 08:13:07.998680 460 tablet_bootstrap.cc:492] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:13:08.001268 27854 catalog_manager.cc:5484] T e30557ffd4594f7d9cda13025e4c5e60 P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69). New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:08.003881 461 tablet_bootstrap.cc:492] T f7a8c95ac81942dabef98a07caa84082 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:13:08.004050 461 ts_tablet_manager.cc:1367] T f7a8c95ac81942dabef98a07caa84082 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.130s user 0.043s sys 0.039s
I20240503 08:13:08.004803 461 raft_consensus.cc:348] T f7a8c95ac81942dabef98a07caa84082 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:08.005017 461 raft_consensus.cc:374] T f7a8c95ac81942dabef98a07caa84082 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:08.005242 461 raft_consensus.cc:729] T f7a8c95ac81942dabef98a07caa84082 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:13:08.005555 461 consensus_queue.cc:260] T f7a8c95ac81942dabef98a07caa84082 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:08.006078 467 tablet_bootstrap.cc:492] T e30557ffd4594f7d9cda13025e4c5e60 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:13:08.006265 467 ts_tablet_manager.cc:1367] T e30557ffd4594f7d9cda13025e4c5e60 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.092s user 0.018s sys 0.012s
I20240503 08:13:08.006215 461 ts_tablet_manager.cc:1398] T f7a8c95ac81942dabef98a07caa84082 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:13:08.005745 460 tablet_bootstrap.cc:654] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:08.007495 467 raft_consensus.cc:348] T e30557ffd4594f7d9cda13025e4c5e60 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:08.007766 467 raft_consensus.cc:374] T e30557ffd4594f7d9cda13025e4c5e60 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:08.007896 467 raft_consensus.cc:729] T e30557ffd4594f7d9cda13025e4c5e60 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:13:08.008178 467 consensus_queue.cc:260] T e30557ffd4594f7d9cda13025e4c5e60 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:08.010013 467 ts_tablet_manager.cc:1398] T e30557ffd4594f7d9cda13025e4c5e60 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 08:13:08.018705 460 tablet_bootstrap.cc:492] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:13:08.018831 460 ts_tablet_manager.cc:1367] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.000s
I20240503 08:13:08.019322 460 raft_consensus.cc:348] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:08.019939 460 raft_consensus.cc:374] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:08.020088 460 raft_consensus.cc:729] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:13:08.020282 460 consensus_queue.cc:260] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:08.020880 460 ts_tablet_manager.cc:1398] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:08.032399 27858 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:13:08.039837 472 tablet.cc:1722] T e352522f8a584822891f68c758971757 P b00f244c69df401997ea99d492218a98: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:13:08.040334 472 tablet.cc:1620] T e352522f8a584822891f68c758971757 P b00f244c69df401997ea99d492218a98: MemRowSet was empty: no flush needed.
I20240503 08:13:08.041249 476 tablet.cc:1722] T e352522f8a584822891f68c758971757 P aae860d7ae2b4002bf7794c278f9cc56: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:13:08.041669 476 tablet.cc:1620] T e352522f8a584822891f68c758971757 P aae860d7ae2b4002bf7794c278f9cc56: MemRowSet was empty: no flush needed.
I20240503 08:13:08.046293 428 tablet.cc:1722] T e352522f8a584822891f68c758971757 P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:13:08.046734 428 tablet.cc:1620] T e352522f8a584822891f68c758971757 P ce1496a47d084365ad786cd09e472a2c: MemRowSet was empty: no flush needed.
I20240503 08:13:08.051028 27854 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=1851668b571346c59948f2019df38c35] alter complete (version 2)
I20240503 08:13:08.119858 374 raft_consensus.cc:482] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:08.120047 374 raft_consensus.cc:504] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:08.120620 374 leader_election.cc:288] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:13:08.120959 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f2743ab5fa24c9899b67ea52f48f9d8" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:13:08.121140 28002 raft_consensus.cc:2454] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:13:08.121071 28263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f2743ab5fa24c9899b67ea52f48f9d8" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:13:08.121274 28263 raft_consensus.cc:2454] T 7f2743ab5fa24c9899b67ea52f48f9d8 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 0.
I20240503 08:13:08.121448 30697 leader_election.cc:302] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, 833582ff68c94269be364ebd187a75db; no voters:
I20240503 08:13:08.121716 374 raft_consensus.cc:2781] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:08.121814 374 raft_consensus.cc:482] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:08.121910 374 raft_consensus.cc:3037] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:08.124047 374 raft_consensus.cc:504] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:08.124689 374 leader_election.cc:288] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Requested vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:13:08.124965 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f2743ab5fa24c9899b67ea52f48f9d8" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:13:08.125046 28263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f2743ab5fa24c9899b67ea52f48f9d8" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:13:08.125131 28002 raft_consensus.cc:3037] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:08.125254 28263 raft_consensus.cc:3037] T 7f2743ab5fa24c9899b67ea52f48f9d8 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:08.127102 28002 raft_consensus.cc:2454] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:13:08.127118 28263 raft_consensus.cc:2454] T 7f2743ab5fa24c9899b67ea52f48f9d8 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:13:08.127434 30695 leader_election.cc:302] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:13:08.127713 374 raft_consensus.cc:2781] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:08.127859 374 raft_consensus.cc:686] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Becoming Leader. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Running, Role: LEADER
I20240503 08:13:08.128082 374 consensus_queue.cc:237] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:08.133306 27858 catalog_manager.cc:5484] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69). New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:08.157873 27858 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:13:08.164199 472 tablet.cc:1722] T e352522f8a584822891f68c758971757 P b00f244c69df401997ea99d492218a98: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:13:08.165756 476 tablet.cc:1722] T e352522f8a584822891f68c758971757 P aae860d7ae2b4002bf7794c278f9cc56: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:13:08.166376 428 tablet.cc:1722] T e352522f8a584822891f68c758971757 P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:13:08.174803 27854 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table_new_name [id=1851668b571346c59948f2019df38c35] alter complete (version 3)
I20240503 08:13:08.246369 339 raft_consensus.cc:482] T f7a8c95ac81942dabef98a07caa84082 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:08.246531 339 raft_consensus.cc:504] T f7a8c95ac81942dabef98a07caa84082 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:08.247342 339 leader_election.cc:288] T f7a8c95ac81942dabef98a07caa84082 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:13:08.247692 30893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7a8c95ac81942dabef98a07caa84082" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:13:08.247901 30893 raft_consensus.cc:2454] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:13:08.247887 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7a8c95ac81942dabef98a07caa84082" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:13:08.248072 28132 raft_consensus.cc:2454] T f7a8c95ac81942dabef98a07caa84082 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 0.
I20240503 08:13:08.248298 28196 leader_election.cc:302] T f7a8c95ac81942dabef98a07caa84082 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:13:08.248600 339 raft_consensus.cc:2781] T f7a8c95ac81942dabef98a07caa84082 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:08.248703 339 raft_consensus.cc:482] T f7a8c95ac81942dabef98a07caa84082 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:08.248790 339 raft_consensus.cc:3037] T f7a8c95ac81942dabef98a07caa84082 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:08.250770 339 raft_consensus.cc:504] T f7a8c95ac81942dabef98a07caa84082 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:08.251727 30893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7a8c95ac81942dabef98a07caa84082" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:13:08.251888 28132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7a8c95ac81942dabef98a07caa84082" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:13:08.251916 30893 raft_consensus.cc:3037] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:08.252025 28132 raft_consensus.cc:3037] T f7a8c95ac81942dabef98a07caa84082 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:08.251361 339 leader_election.cc:288] T f7a8c95ac81942dabef98a07caa84082 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Requested vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:13:08.254130 28132 raft_consensus.cc:2454] T f7a8c95ac81942dabef98a07caa84082 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:13:08.254510 28198 leader_election.cc:302] T f7a8c95ac81942dabef98a07caa84082 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:13:08.254863 339 raft_consensus.cc:2781] T f7a8c95ac81942dabef98a07caa84082 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:08.254976 339 raft_consensus.cc:686] T f7a8c95ac81942dabef98a07caa84082 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Becoming Leader. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Running, Role: LEADER
I20240503 08:13:08.254979 30893 raft_consensus.cc:2454] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 1.
I20240503 08:13:08.255221 339 consensus_queue.cc:237] T f7a8c95ac81942dabef98a07caa84082 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:08.259081 27862 catalog_manager.cc:5484] T f7a8c95ac81942dabef98a07caa84082 P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: term changed from 0 to 1, leader changed from <none> to aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67). New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:08.277007 27862 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:13:08.277227 27862 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:13:08.280243 27862 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet e352522f8a584822891f68c758971757
I20240503 08:13:08.281029 30874 tablet_service.cc:1514] Processing DeleteTablet for tablet e352522f8a584822891f68c758971757 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:08 UTC) from {username='slave'} at 127.0.0.1:34426
I20240503 08:13:08.281473 28242 tablet_service.cc:1514] Processing DeleteTablet for tablet e352522f8a584822891f68c758971757 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:08 UTC) from {username='slave'} at 127.0.0.1:41960
I20240503 08:13:08.282123 484 tablet_replica.cc:330] T e352522f8a584822891f68c758971757 P aae860d7ae2b4002bf7794c278f9cc56: stopping tablet replica
I20240503 08:13:08.282341 484 raft_consensus.cc:2229] T e352522f8a584822891f68c758971757 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:08.282618 484 raft_consensus.cc:2258] T e352522f8a584822891f68c758971757 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:08.283506 484 ts_tablet_manager.cc:1875] T e352522f8a584822891f68c758971757 P aae860d7ae2b4002bf7794c278f9cc56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:08.284186 31140 tablet_service.cc:1514] Processing DeleteTablet for tablet e352522f8a584822891f68c758971757 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:08 UTC) from {username='slave'} at 127.0.0.1:45814
I20240503 08:13:08.289671 484 ts_tablet_manager.cc:1888] T e352522f8a584822891f68c758971757 P aae860d7ae2b4002bf7794c278f9cc56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:13:08.289870 484 log.cc:1196] T e352522f8a584822891f68c758971757 P aae860d7ae2b4002bf7794c278f9cc56: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal/wals/e352522f8a584822891f68c758971757
I20240503 08:13:08.290432 484 ts_tablet_manager.cc:1909] T e352522f8a584822891f68c758971757 P aae860d7ae2b4002bf7794c278f9cc56: Deleting consensus metadata
I20240503 08:13:08.291401 27847 catalog_manager.cc:4830] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): tablet e352522f8a584822891f68c758971757 (table rebalancer_dynamic_table_new_name [id=1851668b571346c59948f2019df38c35]) successfully deleted
I20240503 08:13:08.292889 483 tablet_replica.cc:330] T e352522f8a584822891f68c758971757 P ce1496a47d084365ad786cd09e472a2c: stopping tablet replica
I20240503 08:13:08.293852 483 raft_consensus.cc:2229] T e352522f8a584822891f68c758971757 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:08.294085 485 tablet_replica.cc:330] T e352522f8a584822891f68c758971757 P b00f244c69df401997ea99d492218a98: stopping tablet replica
I20240503 08:13:08.294101 483 raft_consensus.cc:2258] T e352522f8a584822891f68c758971757 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:08.294268 485 raft_consensus.cc:2229] T e352522f8a584822891f68c758971757 P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:13:08.294605 485 raft_consensus.cc:2258] T e352522f8a584822891f68c758971757 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:08.295186 483 ts_tablet_manager.cc:1875] T e352522f8a584822891f68c758971757 P ce1496a47d084365ad786cd09e472a2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:08.295297 485 ts_tablet_manager.cc:1875] T e352522f8a584822891f68c758971757 P b00f244c69df401997ea99d492218a98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:08.301399 483 ts_tablet_manager.cc:1888] T e352522f8a584822891f68c758971757 P ce1496a47d084365ad786cd09e472a2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:13:08.301425 485 ts_tablet_manager.cc:1888] T e352522f8a584822891f68c758971757 P b00f244c69df401997ea99d492218a98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:13:08.301522 483 log.cc:1196] T e352522f8a584822891f68c758971757 P ce1496a47d084365ad786cd09e472a2c: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal/wals/e352522f8a584822891f68c758971757
I20240503 08:13:08.301522 485 log.cc:1196] T e352522f8a584822891f68c758971757 P b00f244c69df401997ea99d492218a98: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal/wals/e352522f8a584822891f68c758971757
I20240503 08:13:08.301970 485 ts_tablet_manager.cc:1909] T e352522f8a584822891f68c758971757 P b00f244c69df401997ea99d492218a98: Deleting consensus metadata
I20240503 08:13:08.301970 483 ts_tablet_manager.cc:1909] T e352522f8a584822891f68c758971757 P ce1496a47d084365ad786cd09e472a2c: Deleting consensus metadata
I20240503 08:13:08.302845 27846 catalog_manager.cc:4830] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): tablet e352522f8a584822891f68c758971757 (table rebalancer_dynamic_table_new_name [id=1851668b571346c59948f2019df38c35]) successfully deleted
I20240503 08:13:08.303076 27846 catalog_manager.cc:4830] TS b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): tablet e352522f8a584822891f68c758971757 (table rebalancer_dynamic_table_new_name [id=1851668b571346c59948f2019df38c35]) successfully deleted
I20240503 08:13:08.385510 27862 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:13:08.393569 30874 tablet_service.cc:1467] Processing CreateTablet for tablet 3eb86788550f4c94ae4f98d92d4bbf10 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=686a739e37944ae997f3ce67e9413129]), partition=
I20240503 08:13:08.393602 30741 tablet_service.cc:1467] Processing CreateTablet for tablet 3eb86788550f4c94ae4f98d92d4bbf10 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=686a739e37944ae997f3ce67e9413129]), partition=
I20240503 08:13:08.393945 30874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3eb86788550f4c94ae4f98d92d4bbf10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:08.394016 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3eb86788550f4c94ae4f98d92d4bbf10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:08.394922 27983 tablet_service.cc:1467] Processing CreateTablet for tablet 3eb86788550f4c94ae4f98d92d4bbf10 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=686a739e37944ae997f3ce67e9413129]), partition=
I20240503 08:13:08.395216 27983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3eb86788550f4c94ae4f98d92d4bbf10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:08.398783 458 tablet_bootstrap.cc:492] T 3eb86788550f4c94ae4f98d92d4bbf10 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:13:08.399601 441 tablet_bootstrap.cc:492] T 3eb86788550f4c94ae4f98d92d4bbf10 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:13:08.400180 458 tablet_bootstrap.cc:654] T 3eb86788550f4c94ae4f98d92d4bbf10 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:08.400943 441 tablet_bootstrap.cc:654] T 3eb86788550f4c94ae4f98d92d4bbf10 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:08.403040 441 tablet_bootstrap.cc:492] T 3eb86788550f4c94ae4f98d92d4bbf10 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:13:08.403169 441 ts_tablet_manager.cc:1367] T 3eb86788550f4c94ae4f98d92d4bbf10 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 08:13:08.403244 460 tablet_bootstrap.cc:492] T 3eb86788550f4c94ae4f98d92d4bbf10 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:13:08.403816 441 raft_consensus.cc:348] T 3eb86788550f4c94ae4f98d92d4bbf10 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:08.403954 441 raft_consensus.cc:374] T 3eb86788550f4c94ae4f98d92d4bbf10 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:08.404028 441 raft_consensus.cc:729] T 3eb86788550f4c94ae4f98d92d4bbf10 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:13:08.404223 441 consensus_queue.cc:260] T 3eb86788550f4c94ae4f98d92d4bbf10 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:08.404453 460 tablet_bootstrap.cc:654] T 3eb86788550f4c94ae4f98d92d4bbf10 P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:08.404717 441 ts_tablet_manager.cc:1398] T 3eb86788550f4c94ae4f98d92d4bbf10 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 08:13:08.406912 458 tablet_bootstrap.cc:492] T 3eb86788550f4c94ae4f98d92d4bbf10 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:13:08.407047 458 ts_tablet_manager.cc:1367] T 3eb86788550f4c94ae4f98d92d4bbf10 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20240503 08:13:08.407157 460 tablet_bootstrap.cc:492] T 3eb86788550f4c94ae4f98d92d4bbf10 P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:13:08.407286 460 ts_tablet_manager.cc:1367] T 3eb86788550f4c94ae4f98d92d4bbf10 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20240503 08:13:08.407622 458 raft_consensus.cc:348] T 3eb86788550f4c94ae4f98d92d4bbf10 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:08.407749 458 raft_consensus.cc:374] T 3eb86788550f4c94ae4f98d92d4bbf10 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:08.407802 458 raft_consensus.cc:729] T 3eb86788550f4c94ae4f98d92d4bbf10 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:13:08.407873 460 raft_consensus.cc:348] T 3eb86788550f4c94ae4f98d92d4bbf10 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:08.407961 458 consensus_queue.cc:260] T 3eb86788550f4c94ae4f98d92d4bbf10 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:08.408082 460 raft_consensus.cc:374] T 3eb86788550f4c94ae4f98d92d4bbf10 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:08.408159 460 raft_consensus.cc:729] T 3eb86788550f4c94ae4f98d92d4bbf10 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:13:08.408447 458 ts_tablet_manager.cc:1398] T 3eb86788550f4c94ae4f98d92d4bbf10 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:13:08.408380 460 consensus_queue.cc:260] T 3eb86788550f4c94ae4f98d92d4bbf10 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:08.408861 460 ts_tablet_manager.cc:1398] T 3eb86788550f4c94ae4f98d92d4bbf10 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 08:13:08.410545 412 raft_consensus.cc:482] T e30557ffd4594f7d9cda13025e4c5e60 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:08.410717 412 raft_consensus.cc:504] T e30557ffd4594f7d9cda13025e4c5e60 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:08.411460 412 leader_election.cc:288] T e30557ffd4594f7d9cda13025e4c5e60 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:13:08.411789 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30557ffd4594f7d9cda13025e4c5e60" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:13:08.413645 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30557ffd4594f7d9cda13025e4c5e60" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:13:08.413919 28002 raft_consensus.cc:2379] T e30557ffd4594f7d9cda13025e4c5e60 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ced21ca983b749d0b4d4f6a0e2426bf8 in current term 1: Already voted for candidate 6d1d7de19459435fbde1999e3dfd226b in this term.
I20240503 08:13:08.414551 30962 leader_election.cc:302] T e30557ffd4594f7d9cda13025e4c5e60 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8; no voters: 6d1d7de19459435fbde1999e3dfd226b, 833582ff68c94269be364ebd187a75db
I20240503 08:13:08.414881 412 raft_consensus.cc:3037] T e30557ffd4594f7d9cda13025e4c5e60 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:08.416915 412 raft_consensus.cc:2726] T e30557ffd4594f7d9cda13025e4c5e60 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:13:08.486258 374 consensus_queue.cc:1035] T e30557ffd4594f7d9cda13025e4c5e60 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:08.495893 374 consensus_queue.cc:1035] T e30557ffd4594f7d9cda13025e4c5e60 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:08.548383 439 raft_consensus.cc:482] T 3eb86788550f4c94ae4f98d92d4bbf10 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:08.548568 439 raft_consensus.cc:504] T 3eb86788550f4c94ae4f98d92d4bbf10 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:08.549283 439 leader_election.cc:288] T 3eb86788550f4c94ae4f98d92d4bbf10 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:13:08.549640 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eb86788550f4c94ae4f98d92d4bbf10" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:13:08.549804 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eb86788550f4c94ae4f98d92d4bbf10" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:13:08.549906 30761 raft_consensus.cc:2454] T 3eb86788550f4c94ae4f98d92d4bbf10 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:13:08.550029 28002 raft_consensus.cc:2454] T 3eb86788550f4c94ae4f98d92d4bbf10 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:13:08.550305 30829 leader_election.cc:302] T 3eb86788550f4c94ae4f98d92d4bbf10 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:13:08.550554 439 raft_consensus.cc:2781] T 3eb86788550f4c94ae4f98d92d4bbf10 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:08.550724 439 raft_consensus.cc:482] T 3eb86788550f4c94ae4f98d92d4bbf10 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:08.550827 439 raft_consensus.cc:3037] T 3eb86788550f4c94ae4f98d92d4bbf10 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:08.553721 439 raft_consensus.cc:504] T 3eb86788550f4c94ae4f98d92d4bbf10 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:08.554508 439 leader_election.cc:288] T 3eb86788550f4c94ae4f98d92d4bbf10 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Requested vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:13:08.555156 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eb86788550f4c94ae4f98d92d4bbf10" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:13:08.555399 30761 raft_consensus.cc:3037] T 3eb86788550f4c94ae4f98d92d4bbf10 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:08.556957 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eb86788550f4c94ae4f98d92d4bbf10" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:13:08.557148 28002 raft_consensus.cc:3037] T 3eb86788550f4c94ae4f98d92d4bbf10 P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:08.558557 30761 raft_consensus.cc:2454] T 3eb86788550f4c94ae4f98d92d4bbf10 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:13:08.558964 30829 leader_election.cc:302] T 3eb86788550f4c94ae4f98d92d4bbf10 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:13:08.559273 439 raft_consensus.cc:2781] T 3eb86788550f4c94ae4f98d92d4bbf10 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:08.559531 439 raft_consensus.cc:686] T 3eb86788550f4c94ae4f98d92d4bbf10 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Becoming Leader. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Running, Role: LEADER
I20240503 08:13:08.559868 439 consensus_queue.cc:237] T 3eb86788550f4c94ae4f98d92d4bbf10 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:08.560443 28002 raft_consensus.cc:2454] T 3eb86788550f4c94ae4f98d92d4bbf10 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:13:08.564414 27858 catalog_manager.cc:5484] T 3eb86788550f4c94ae4f98d92d4bbf10 P ce1496a47d084365ad786cd09e472a2c reported cstate change: term changed from 0 to 1, leader changed from <none> to ce1496a47d084365ad786cd09e472a2c (127.20.230.70). New cstate: current_term: 1 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:08.596293 477 consensus_queue.cc:1035] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:08.600600 477 consensus_queue.cc:1035] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:08.665148 338 consensus_queue.cc:1035] T f7a8c95ac81942dabef98a07caa84082 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:08.669827 338 consensus_queue.cc:1035] T f7a8c95ac81942dabef98a07caa84082 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:08.694701 27862 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:13:08.702081 28002 raft_consensus.cc:1261] T 3eb86788550f4c94ae4f98d92d4bbf10 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:08.702251 30761 raft_consensus.cc:1261] T 3eb86788550f4c94ae4f98d92d4bbf10 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:08.702708 439 consensus_queue.cc:1035] T 3eb86788550f4c94ae4f98d92d4bbf10 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:08.702993 502 consensus_queue.cc:1035] T 3eb86788550f4c94ae4f98d92d4bbf10 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:08.712805 491 mvcc.cc:205] Tried to move back new op lower bound from 7023509457716944896 to 7023509457144602624. Current Snapshot: MvccSnapshot[applied={T|T < 7023509457716944896}]
I20240503 08:13:08.712275 27862 catalog_manager.cc:5576] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081) does not have the latest schema for tablet 3eb86788550f4c94ae4f98d92d4bbf10 (table rebalancer_dynamic_table [id=686a739e37944ae997f3ce67e9413129]). Expected version 1 got 0
I20240503 08:13:08.713277 506 tablet.cc:1722] T 3eb86788550f4c94ae4f98d92d4bbf10 P 833582ff68c94269be364ebd187a75db: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:13:08.720459 506 tablet.cc:1620] T 3eb86788550f4c94ae4f98d92d4bbf10 P 833582ff68c94269be364ebd187a75db: MemRowSet was empty: no flush needed.
I20240503 08:13:08.722743 27862 catalog_manager.cc:5576] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081) does not have the latest schema for tablet 3eb86788550f4c94ae4f98d92d4bbf10 (table rebalancer_dynamic_table [id=686a739e37944ae997f3ce67e9413129]). Expected version 1 got 0
I20240503 08:13:08.725363 27858 catalog_manager.cc:5576] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725) does not have the latest schema for tablet 3eb86788550f4c94ae4f98d92d4bbf10 (table rebalancer_dynamic_table [id=686a739e37944ae997f3ce67e9413129]). Expected version 1 got 0
I20240503 08:13:08.728072 505 tablet.cc:1722] T 3eb86788550f4c94ae4f98d92d4bbf10 P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:13:08.729271 505 tablet.cc:1620] T 3eb86788550f4c94ae4f98d92d4bbf10 P ce1496a47d084365ad786cd09e472a2c: MemRowSet was empty: no flush needed.
I20240503 08:13:08.735524 27858 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=686a739e37944ae997f3ce67e9413129] alter complete (version 1)
I20240503 08:13:08.754381 507 tablet.cc:1722] T 3eb86788550f4c94ae4f98d92d4bbf10 P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:13:08.755211 507 tablet.cc:1620] T 3eb86788550f4c94ae4f98d92d4bbf10 P 6d1d7de19459435fbde1999e3dfd226b: MemRowSet was empty: no flush needed.
I20240503 08:13:08.757356 27862 catalog_manager.cc:5576] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725) does not have the latest schema for tablet 3eb86788550f4c94ae4f98d92d4bbf10 (table rebalancer_dynamic_table [id=686a739e37944ae997f3ce67e9413129]). Expected version 1 got 0
I20240503 08:13:08.837991 27862 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:13:08.844250 27858 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_extra_table_64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:13:08.845945 505 tablet.cc:1722] T 3eb86788550f4c94ae4f98d92d4bbf10 P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:13:08.846518 505 tablet.cc:1620] T 3eb86788550f4c94ae4f98d92d4bbf10 P ce1496a47d084365ad786cd09e472a2c: MemRowSet was empty: no flush needed.
I20240503 08:13:08.848380 506 tablet.cc:1722] T 3eb86788550f4c94ae4f98d92d4bbf10 P 833582ff68c94269be364ebd187a75db: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:13:08.848904 506 tablet.cc:1620] T 3eb86788550f4c94ae4f98d92d4bbf10 P 833582ff68c94269be364ebd187a75db: MemRowSet was empty: no flush needed.
I20240503 08:13:08.855854 507 tablet.cc:1722] T 3eb86788550f4c94ae4f98d92d4bbf10 P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:13:08.856405 507 tablet.cc:1620] T 3eb86788550f4c94ae4f98d92d4bbf10 P 6d1d7de19459435fbde1999e3dfd226b: MemRowSet was empty: no flush needed.
I20240503 08:13:08.859442 31140 tablet_service.cc:1467] Processing CreateTablet for tablet 45e7a03f838d4beaae2080d523319d58 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=00d43ce433a44235afa9ed3a6510e586]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:08.859943 31140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45e7a03f838d4beaae2080d523319d58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:08.861316 30742 tablet_service.cc:1467] Processing CreateTablet for tablet 48075c4a45d24e84bb2b9760710de3a0 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=00d43ce433a44235afa9ed3a6510e586]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:08.861802 30742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48075c4a45d24e84bb2b9760710de3a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:08.863302 30741 tablet_service.cc:1467] Processing CreateTablet for tablet 607a503b64d8440a81cf66ab158ae273 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=00d43ce433a44235afa9ed3a6510e586]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:08.863734 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 607a503b64d8440a81cf66ab158ae273. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:08.866200 31141 tablet_service.cc:1467] Processing CreateTablet for tablet 48075c4a45d24e84bb2b9760710de3a0 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=00d43ce433a44235afa9ed3a6510e586]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:08.866631 31141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48075c4a45d24e84bb2b9760710de3a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:08.869042 28372 tablet_service.cc:1467] Processing CreateTablet for tablet 607a503b64d8440a81cf66ab158ae273 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=00d43ce433a44235afa9ed3a6510e586]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:08.869457 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 607a503b64d8440a81cf66ab158ae273. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:08.870803 28111 tablet_service.cc:1467] Processing CreateTablet for tablet 607a503b64d8440a81cf66ab158ae273 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=00d43ce433a44235afa9ed3a6510e586]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:08.870988 28373 tablet_service.cc:1467] Processing CreateTablet for tablet 45e7a03f838d4beaae2080d523319d58 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=00d43ce433a44235afa9ed3a6510e586]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:08.871229 28111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 607a503b64d8440a81cf66ab158ae273. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:08.871313 28373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45e7a03f838d4beaae2080d523319d58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:08.872776 31006 tablet_service.cc:1467] Processing CreateTablet for tablet 45e7a03f838d4beaae2080d523319d58 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=00d43ce433a44235afa9ed3a6510e586]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:08.873190 31006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45e7a03f838d4beaae2080d523319d58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:08.879137 31008 tablet_service.cc:1467] Processing CreateTablet for tablet 48075c4a45d24e84bb2b9760710de3a0 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=00d43ce433a44235afa9ed3a6510e586]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:08.879593 31008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48075c4a45d24e84bb2b9760710de3a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:08.883378 27862 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=686a739e37944ae997f3ce67e9413129] alter complete (version 2)
I20240503 08:13:08.891086 458 tablet_bootstrap.cc:492] T 607a503b64d8440a81cf66ab158ae273 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:13:08.892930 458 tablet_bootstrap.cc:654] T 607a503b64d8440a81cf66ab158ae273 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:08.900113 513 tablet_bootstrap.cc:492] T 48075c4a45d24e84bb2b9760710de3a0 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:13:08.900496 509 tablet_bootstrap.cc:492] T 607a503b64d8440a81cf66ab158ae273 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:13:08.901371 512 tablet_bootstrap.cc:492] T 45e7a03f838d4beaae2080d523319d58 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:13:08.902307 513 tablet_bootstrap.cc:654] T 48075c4a45d24e84bb2b9760710de3a0 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:08.903142 509 tablet_bootstrap.cc:654] T 607a503b64d8440a81cf66ab158ae273 P 1b411c6bb25549368041773d7b710d48: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:08.903527 512 tablet_bootstrap.cc:654] T 45e7a03f838d4beaae2080d523319d58 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:08.907302 509 tablet_bootstrap.cc:492] T 607a503b64d8440a81cf66ab158ae273 P 1b411c6bb25549368041773d7b710d48: No bootstrap required, opened a new log
I20240503 08:13:08.907523 509 ts_tablet_manager.cc:1367] T 607a503b64d8440a81cf66ab158ae273 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 08:13:08.907778 511 tablet_bootstrap.cc:492] T 45e7a03f838d4beaae2080d523319d58 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:13:08.908383 513 tablet_bootstrap.cc:492] T 48075c4a45d24e84bb2b9760710de3a0 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:13:08.909142 458 tablet_bootstrap.cc:492] T 607a503b64d8440a81cf66ab158ae273 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:13:08.909464 458 ts_tablet_manager.cc:1367] T 607a503b64d8440a81cf66ab158ae273 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.006s
I20240503 08:13:08.909610 511 tablet_bootstrap.cc:654] T 45e7a03f838d4beaae2080d523319d58 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:08.910552 509 raft_consensus.cc:348] T 607a503b64d8440a81cf66ab158ae273 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:08.910853 509 raft_consensus.cc:374] T 607a503b64d8440a81cf66ab158ae273 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:08.911020 509 raft_consensus.cc:729] T 607a503b64d8440a81cf66ab158ae273 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:13:08.911592 458 raft_consensus.cc:348] T 607a503b64d8440a81cf66ab158ae273 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:08.911495 509 consensus_queue.cc:260] T 607a503b64d8440a81cf66ab158ae273 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:08.911862 458 raft_consensus.cc:374] T 607a503b64d8440a81cf66ab158ae273 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:08.911950 458 raft_consensus.cc:729] T 607a503b64d8440a81cf66ab158ae273 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:13:08.912247 458 consensus_queue.cc:260] T 607a503b64d8440a81cf66ab158ae273 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:08.912779 509 ts_tablet_manager.cc:1398] T 607a503b64d8440a81cf66ab158ae273 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:13:08.912820 458 ts_tablet_manager.cc:1398] T 607a503b64d8440a81cf66ab158ae273 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:13:08.921080 458 tablet_bootstrap.cc:492] T 48075c4a45d24e84bb2b9760710de3a0 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:13:08.925168 458 tablet_bootstrap.cc:654] T 48075c4a45d24e84bb2b9760710de3a0 P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:08.926023 513 ts_tablet_manager.cc:1367] T 48075c4a45d24e84bb2b9760710de3a0 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20240503 08:13:08.926890 513 raft_consensus.cc:348] T 48075c4a45d24e84bb2b9760710de3a0 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:08.927120 513 raft_consensus.cc:374] T 48075c4a45d24e84bb2b9760710de3a0 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:08.927214 513 raft_consensus.cc:729] T 48075c4a45d24e84bb2b9760710de3a0 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:13:08.927567 513 consensus_queue.cc:260] T 48075c4a45d24e84bb2b9760710de3a0 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:08.927809 474 raft_consensus.cc:482] T 48075c4a45d24e84bb2b9760710de3a0 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:08.927929 474 raft_consensus.cc:504] T 48075c4a45d24e84bb2b9760710de3a0 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:08.929275 474 leader_election.cc:288] T 48075c4a45d24e84bb2b9760710de3a0 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:13:08.929817 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48075c4a45d24e84bb2b9760710de3a0" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:13:08.930550 513 ts_tablet_manager.cc:1398] T 48075c4a45d24e84bb2b9760710de3a0 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
W20240503 08:13:08.931360 31096 leader_election.cc:341] T 48075c4a45d24e84bb2b9760710de3a0 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:13:08.931612 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48075c4a45d24e84bb2b9760710de3a0" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
W20240503 08:13:08.932205 31095 leader_election.cc:341] T 48075c4a45d24e84bb2b9760710de3a0 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:13:08.932350 31095 leader_election.cc:302] T 48075c4a45d24e84bb2b9760710de3a0 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b00f244c69df401997ea99d492218a98; no voters: 6d1d7de19459435fbde1999e3dfd226b, ced21ca983b749d0b4d4f6a0e2426bf8
I20240503 08:13:08.932495 513 tablet_bootstrap.cc:492] T 45e7a03f838d4beaae2080d523319d58 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:13:08.932829 474 raft_consensus.cc:2726] T 48075c4a45d24e84bb2b9760710de3a0 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:13:08.933518 512 tablet_bootstrap.cc:492] T 45e7a03f838d4beaae2080d523319d58 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:13:08.933689 512 ts_tablet_manager.cc:1367] T 45e7a03f838d4beaae2080d523319d58 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.005s
I20240503 08:13:08.934204 513 tablet_bootstrap.cc:654] T 45e7a03f838d4beaae2080d523319d58 P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:08.934389 512 raft_consensus.cc:348] T 45e7a03f838d4beaae2080d523319d58 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:08.934578 512 raft_consensus.cc:374] T 45e7a03f838d4beaae2080d523319d58 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:08.934625 512 raft_consensus.cc:729] T 45e7a03f838d4beaae2080d523319d58 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:13:08.934929 512 consensus_queue.cc:260] T 45e7a03f838d4beaae2080d523319d58 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:08.936363 513 tablet_bootstrap.cc:492] T 45e7a03f838d4beaae2080d523319d58 P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:13:08.936661 513 ts_tablet_manager.cc:1367] T 45e7a03f838d4beaae2080d523319d58 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 08:13:08.937606 513 raft_consensus.cc:348] T 45e7a03f838d4beaae2080d523319d58 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:08.937927 513 raft_consensus.cc:374] T 45e7a03f838d4beaae2080d523319d58 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:08.938097 513 raft_consensus.cc:729] T 45e7a03f838d4beaae2080d523319d58 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:13:08.938405 513 consensus_queue.cc:260] T 45e7a03f838d4beaae2080d523319d58 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:08.939009 513 ts_tablet_manager.cc:1398] T 45e7a03f838d4beaae2080d523319d58 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:08.939697 512 ts_tablet_manager.cc:1398] T 45e7a03f838d4beaae2080d523319d58 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:13:08.942322 458 tablet_bootstrap.cc:492] T 48075c4a45d24e84bb2b9760710de3a0 P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:13:08.940954 512 tablet_bootstrap.cc:492] T 607a503b64d8440a81cf66ab158ae273 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:13:08.942498 458 ts_tablet_manager.cc:1367] T 48075c4a45d24e84bb2b9760710de3a0 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.003s
I20240503 08:13:08.943367 458 raft_consensus.cc:348] T 48075c4a45d24e84bb2b9760710de3a0 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:08.944028 458 raft_consensus.cc:374] T 48075c4a45d24e84bb2b9760710de3a0 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:08.944119 458 raft_consensus.cc:729] T 48075c4a45d24e84bb2b9760710de3a0 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:13:08.944346 458 consensus_queue.cc:260] T 48075c4a45d24e84bb2b9760710de3a0 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:08.944540 512 tablet_bootstrap.cc:654] T 607a503b64d8440a81cf66ab158ae273 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:08.944954 458 ts_tablet_manager.cc:1398] T 48075c4a45d24e84bb2b9760710de3a0 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:08.946273 511 tablet_bootstrap.cc:492] T 45e7a03f838d4beaae2080d523319d58 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:13:08.946463 511 ts_tablet_manager.cc:1367] T 45e7a03f838d4beaae2080d523319d58 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.039s user 0.012s sys 0.011s
I20240503 08:13:08.947295 511 raft_consensus.cc:348] T 45e7a03f838d4beaae2080d523319d58 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:08.947548 511 raft_consensus.cc:374] T 45e7a03f838d4beaae2080d523319d58 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:08.947647 511 raft_consensus.cc:729] T 45e7a03f838d4beaae2080d523319d58 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:13:08.947893 511 consensus_queue.cc:260] T 45e7a03f838d4beaae2080d523319d58 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:08.948463 511 ts_tablet_manager.cc:1398] T 45e7a03f838d4beaae2080d523319d58 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:08.948748 512 tablet_bootstrap.cc:492] T 607a503b64d8440a81cf66ab158ae273 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:13:08.948801 511 tablet_bootstrap.cc:492] T 48075c4a45d24e84bb2b9760710de3a0 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:13:08.948951 512 ts_tablet_manager.cc:1367] T 607a503b64d8440a81cf66ab158ae273 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20240503 08:13:08.949644 512 raft_consensus.cc:348] T 607a503b64d8440a81cf66ab158ae273 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:08.949949 512 raft_consensus.cc:374] T 607a503b64d8440a81cf66ab158ae273 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:08.950095 512 raft_consensus.cc:729] T 607a503b64d8440a81cf66ab158ae273 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:13:08.950413 512 consensus_queue.cc:260] T 607a503b64d8440a81cf66ab158ae273 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:08.950760 511 tablet_bootstrap.cc:654] T 48075c4a45d24e84bb2b9760710de3a0 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:08.951059 512 ts_tablet_manager.cc:1398] T 607a503b64d8440a81cf66ab158ae273 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:08.952994 511 tablet_bootstrap.cc:492] T 48075c4a45d24e84bb2b9760710de3a0 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:13:08.953217 511 ts_tablet_manager.cc:1367] T 48075c4a45d24e84bb2b9760710de3a0 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 08:13:08.954088 511 raft_consensus.cc:348] T 48075c4a45d24e84bb2b9760710de3a0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:08.954361 511 raft_consensus.cc:374] T 48075c4a45d24e84bb2b9760710de3a0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:08.954510 511 raft_consensus.cc:729] T 48075c4a45d24e84bb2b9760710de3a0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:13:08.954816 511 consensus_queue.cc:260] T 48075c4a45d24e84bb2b9760710de3a0 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:08.955420 511 ts_tablet_manager.cc:1398] T 48075c4a45d24e84bb2b9760710de3a0 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:08.995463 378 raft_consensus.cc:482] T 607a503b64d8440a81cf66ab158ae273 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:08.995409 27862 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:13:08.995615 378 raft_consensus.cc:504] T 607a503b64d8440a81cf66ab158ae273 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:08.996170 378 leader_election.cc:288] T 607a503b64d8440a81cf66ab158ae273 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:13:08.997048 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "607a503b64d8440a81cf66ab158ae273" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:13:08.997359 30761 raft_consensus.cc:2454] T 607a503b64d8440a81cf66ab158ae273 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:13:08.997293 28133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "607a503b64d8440a81cf66ab158ae273" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:13:08.997515 28133 raft_consensus.cc:2454] T 607a503b64d8440a81cf66ab158ae273 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:13:08.997822 28328 leader_election.cc:302] T 607a503b64d8440a81cf66ab158ae273 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:13:08.998301 378 raft_consensus.cc:2781] T 607a503b64d8440a81cf66ab158ae273 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:08.998379 378 raft_consensus.cc:482] T 607a503b64d8440a81cf66ab158ae273 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:08.998450 378 raft_consensus.cc:3037] T 607a503b64d8440a81cf66ab158ae273 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:09.001034 378 raft_consensus.cc:504] T 607a503b64d8440a81cf66ab158ae273 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:09.002305 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "607a503b64d8440a81cf66ab158ae273" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:13:09.002566 30761 raft_consensus.cc:3037] T 607a503b64d8440a81cf66ab158ae273 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:09.003959 28133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "607a503b64d8440a81cf66ab158ae273" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:13:09.004166 28133 raft_consensus.cc:3037] T 607a503b64d8440a81cf66ab158ae273 P 1b411c6bb25549368041773d7b710d48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:09.005448 30761 raft_consensus.cc:2454] T 607a503b64d8440a81cf66ab158ae273 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:13:09.005817 505 tablet.cc:1722] T 3eb86788550f4c94ae4f98d92d4bbf10 P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:13:09.006119 28328 leader_election.cc:302] T 607a503b64d8440a81cf66ab158ae273 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:13:09.006611 510 raft_consensus.cc:2781] T 607a503b64d8440a81cf66ab158ae273 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:09.006736 510 raft_consensus.cc:686] T 607a503b64d8440a81cf66ab158ae273 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Becoming Leader. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Running, Role: LEADER
I20240503 08:13:09.006975 510 consensus_queue.cc:237] T 607a503b64d8440a81cf66ab158ae273 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:09.008901 28133 raft_consensus.cc:2454] T 607a503b64d8440a81cf66ab158ae273 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:13:09.009407 378 leader_election.cc:288] T 607a503b64d8440a81cf66ab158ae273 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Requested vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:13:09.010567 507 tablet.cc:1722] T 3eb86788550f4c94ae4f98d92d4bbf10 P 6d1d7de19459435fbde1999e3dfd226b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:13:09.013404 27862 catalog_manager.cc:5484] T 607a503b64d8440a81cf66ab158ae273 P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68). New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:09.014971 506 tablet.cc:1722] T 3eb86788550f4c94ae4f98d92d4bbf10 P 833582ff68c94269be364ebd187a75db: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:13:09.020432 27854 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table_new_name [id=686a739e37944ae997f3ce67e9413129] alter complete (version 3)
I20240503 08:13:09.100991 508 raft_consensus.cc:482] T 48075c4a45d24e84bb2b9760710de3a0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:09.101161 508 raft_consensus.cc:504] T 48075c4a45d24e84bb2b9760710de3a0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:09.101914 508 leader_election.cc:288] T 48075c4a45d24e84bb2b9760710de3a0 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:13:09.102519 30760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48075c4a45d24e84bb2b9760710de3a0" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:13:09.102823 30760 raft_consensus.cc:2454] T 48075c4a45d24e84bb2b9760710de3a0 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 0.
I20240503 08:13:09.102751 31160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48075c4a45d24e84bb2b9760710de3a0" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:13:09.102994 31160 raft_consensus.cc:2454] T 48075c4a45d24e84bb2b9760710de3a0 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 0.
I20240503 08:13:09.103283 30961 leader_election.cc:302] T 48075c4a45d24e84bb2b9760710de3a0 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:13:09.103556 508 raft_consensus.cc:2781] T 48075c4a45d24e84bb2b9760710de3a0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:09.103674 508 raft_consensus.cc:482] T 48075c4a45d24e84bb2b9760710de3a0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:09.103765 508 raft_consensus.cc:3037] T 48075c4a45d24e84bb2b9760710de3a0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:09.106096 508 raft_consensus.cc:504] T 48075c4a45d24e84bb2b9760710de3a0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:09.106741 508 leader_election.cc:288] T 48075c4a45d24e84bb2b9760710de3a0 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 election: Requested vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:13:09.107043 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48075c4a45d24e84bb2b9760710de3a0" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:13:09.107230 31159 raft_consensus.cc:3037] T 48075c4a45d24e84bb2b9760710de3a0 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:09.107400 30760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48075c4a45d24e84bb2b9760710de3a0" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:13:09.107553 30760 raft_consensus.cc:3037] T 48075c4a45d24e84bb2b9760710de3a0 P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:09.109704 30760 raft_consensus.cc:2454] T 48075c4a45d24e84bb2b9760710de3a0 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 1.
I20240503 08:13:09.110054 31159 raft_consensus.cc:2454] T 48075c4a45d24e84bb2b9760710de3a0 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 1.
I20240503 08:13:09.110158 30961 leader_election.cc:302] T 48075c4a45d24e84bb2b9760710de3a0 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:13:09.111150 508 raft_consensus.cc:2781] T 48075c4a45d24e84bb2b9760710de3a0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:09.111421 508 raft_consensus.cc:686] T 48075c4a45d24e84bb2b9760710de3a0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 LEADER]: Becoming Leader. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Running, Role: LEADER
I20240503 08:13:09.111730 508 consensus_queue.cc:237] T 48075c4a45d24e84bb2b9760710de3a0 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:09.116065 27862 catalog_manager.cc:5484] T 48075c4a45d24e84bb2b9760710de3a0 P ced21ca983b749d0b4d4f6a0e2426bf8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71). New cstate: current_term: 1 leader_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:09.125427 27862 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:13:09.125720 27862 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:13:09.128736 27862 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet 3eb86788550f4c94ae4f98d92d4bbf10
I20240503 08:13:09.129614 30742 tablet_service.cc:1514] Processing DeleteTablet for tablet 3eb86788550f4c94ae4f98d92d4bbf10 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:09 UTC) from {username='slave'} at 127.0.0.1:46794
I20240503 08:13:09.129681 30874 tablet_service.cc:1514] Processing DeleteTablet for tablet 3eb86788550f4c94ae4f98d92d4bbf10 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:09 UTC) from {username='slave'} at 127.0.0.1:34426
I20240503 08:13:09.130554 27983 tablet_service.cc:1514] Processing DeleteTablet for tablet 3eb86788550f4c94ae4f98d92d4bbf10 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:09 UTC) from {username='slave'} at 127.0.0.1:55610
I20240503 08:13:09.143644 527 tablet_replica.cc:330] T 3eb86788550f4c94ae4f98d92d4bbf10 P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:13:09.144153 527 raft_consensus.cc:2229] T 3eb86788550f4c94ae4f98d92d4bbf10 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:09.144241 526 tablet_replica.cc:330] T 3eb86788550f4c94ae4f98d92d4bbf10 P 6d1d7de19459435fbde1999e3dfd226b: stopping tablet replica
I20240503 08:13:09.144513 526 raft_consensus.cc:2229] T 3eb86788550f4c94ae4f98d92d4bbf10 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:09.145061 527 raft_consensus.cc:2258] T 3eb86788550f4c94ae4f98d92d4bbf10 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:09.145254 526 raft_consensus.cc:2258] T 3eb86788550f4c94ae4f98d92d4bbf10 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:09.146446 526 ts_tablet_manager.cc:1875] T 3eb86788550f4c94ae4f98d92d4bbf10 P 6d1d7de19459435fbde1999e3dfd226b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:09.146862 527 ts_tablet_manager.cc:1875] T 3eb86788550f4c94ae4f98d92d4bbf10 P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:09.153287 525 tablet_replica.cc:330] T 3eb86788550f4c94ae4f98d92d4bbf10 P ce1496a47d084365ad786cd09e472a2c: stopping tablet replica
I20240503 08:13:09.153555 525 raft_consensus.cc:2229] T 3eb86788550f4c94ae4f98d92d4bbf10 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:13:09.153949 525 raft_consensus.cc:2258] T 3eb86788550f4c94ae4f98d92d4bbf10 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:09.154157 526 ts_tablet_manager.cc:1888] T 3eb86788550f4c94ae4f98d92d4bbf10 P 6d1d7de19459435fbde1999e3dfd226b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:13:09.154314 526 log.cc:1196] T 3eb86788550f4c94ae4f98d92d4bbf10 P 6d1d7de19459435fbde1999e3dfd226b: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal/wals/3eb86788550f4c94ae4f98d92d4bbf10
I20240503 08:13:09.154793 526 ts_tablet_manager.cc:1909] T 3eb86788550f4c94ae4f98d92d4bbf10 P 6d1d7de19459435fbde1999e3dfd226b: Deleting consensus metadata
I20240503 08:13:09.154794 527 ts_tablet_manager.cc:1888] T 3eb86788550f4c94ae4f98d92d4bbf10 P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:13:09.154902 527 log.cc:1196] T 3eb86788550f4c94ae4f98d92d4bbf10 P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/3eb86788550f4c94ae4f98d92d4bbf10
I20240503 08:13:09.155303 527 ts_tablet_manager.cc:1909] T 3eb86788550f4c94ae4f98d92d4bbf10 P 833582ff68c94269be364ebd187a75db: Deleting consensus metadata
I20240503 08:13:09.155905 27848 catalog_manager.cc:4830] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): tablet 3eb86788550f4c94ae4f98d92d4bbf10 (table rebalancer_dynamic_table_new_name [id=686a739e37944ae997f3ce67e9413129]) successfully deleted
I20240503 08:13:09.156018 525 ts_tablet_manager.cc:1875] T 3eb86788550f4c94ae4f98d92d4bbf10 P ce1496a47d084365ad786cd09e472a2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:09.156432 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet 3eb86788550f4c94ae4f98d92d4bbf10 (table rebalancer_dynamic_table_new_name [id=686a739e37944ae997f3ce67e9413129]) successfully deleted
I20240503 08:13:09.161311 525 ts_tablet_manager.cc:1888] T 3eb86788550f4c94ae4f98d92d4bbf10 P ce1496a47d084365ad786cd09e472a2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:13:09.161442 525 log.cc:1196] T 3eb86788550f4c94ae4f98d92d4bbf10 P ce1496a47d084365ad786cd09e472a2c: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal/wals/3eb86788550f4c94ae4f98d92d4bbf10
I20240503 08:13:09.161868 525 ts_tablet_manager.cc:1909] T 3eb86788550f4c94ae4f98d92d4bbf10 P ce1496a47d084365ad786cd09e472a2c: Deleting consensus metadata
I20240503 08:13:09.162763 27846 catalog_manager.cc:4830] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): tablet 3eb86788550f4c94ae4f98d92d4bbf10 (table rebalancer_dynamic_table_new_name [id=686a739e37944ae997f3ce67e9413129]) successfully deleted
I20240503 08:13:09.231523 27862 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:13:09.240337 31141 tablet_service.cc:1467] Processing CreateTablet for tablet 2036b24fbe1647e6a781be6b1534bc9b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=16330782a2e54f7082e781905b55f1c7]), partition=
I20240503 08:13:09.240751 31141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2036b24fbe1647e6a781be6b1534bc9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:09.240979 27983 tablet_service.cc:1467] Processing CreateTablet for tablet 2036b24fbe1647e6a781be6b1534bc9b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=16330782a2e54f7082e781905b55f1c7]), partition=
I20240503 08:13:09.241323 27983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2036b24fbe1647e6a781be6b1534bc9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:09.248056 513 tablet_bootstrap.cc:492] T 2036b24fbe1647e6a781be6b1534bc9b P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:13:09.248366 30874 tablet_service.cc:1467] Processing CreateTablet for tablet 2036b24fbe1647e6a781be6b1534bc9b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=16330782a2e54f7082e781905b55f1c7]), partition=
I20240503 08:13:09.248709 30874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2036b24fbe1647e6a781be6b1534bc9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:09.249594 513 tablet_bootstrap.cc:654] T 2036b24fbe1647e6a781be6b1534bc9b P b00f244c69df401997ea99d492218a98: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:09.254040 513 tablet_bootstrap.cc:492] T 2036b24fbe1647e6a781be6b1534bc9b P b00f244c69df401997ea99d492218a98: No bootstrap required, opened a new log
I20240503 08:13:09.254246 513 ts_tablet_manager.cc:1367] T 2036b24fbe1647e6a781be6b1534bc9b P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20240503 08:13:09.255338 513 raft_consensus.cc:348] T 2036b24fbe1647e6a781be6b1534bc9b P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:09.255517 513 raft_consensus.cc:374] T 2036b24fbe1647e6a781be6b1534bc9b P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:09.255617 513 raft_consensus.cc:729] T 2036b24fbe1647e6a781be6b1534bc9b P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:13:09.255841 513 consensus_queue.cc:260] T 2036b24fbe1647e6a781be6b1534bc9b P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:09.257782 513 ts_tablet_manager.cc:1398] T 2036b24fbe1647e6a781be6b1534bc9b P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:13:09.255287 531 tablet_bootstrap.cc:492] T 2036b24fbe1647e6a781be6b1534bc9b P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:13:09.259299 529 tablet_bootstrap.cc:492] T 2036b24fbe1647e6a781be6b1534bc9b P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:13:09.260707 531 tablet_bootstrap.cc:654] T 2036b24fbe1647e6a781be6b1534bc9b P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:09.261055 529 tablet_bootstrap.cc:654] T 2036b24fbe1647e6a781be6b1534bc9b P 833582ff68c94269be364ebd187a75db: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:09.264559 531 tablet_bootstrap.cc:492] T 2036b24fbe1647e6a781be6b1534bc9b P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:13:09.264606 529 tablet_bootstrap.cc:492] T 2036b24fbe1647e6a781be6b1534bc9b P 833582ff68c94269be364ebd187a75db: No bootstrap required, opened a new log
I20240503 08:13:09.264734 531 ts_tablet_manager.cc:1367] T 2036b24fbe1647e6a781be6b1534bc9b P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240503 08:13:09.264747 529 ts_tablet_manager.cc:1367] T 2036b24fbe1647e6a781be6b1534bc9b P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 08:13:09.265370 531 raft_consensus.cc:348] T 2036b24fbe1647e6a781be6b1534bc9b P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:09.265377 529 raft_consensus.cc:348] T 2036b24fbe1647e6a781be6b1534bc9b P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:09.265573 531 raft_consensus.cc:374] T 2036b24fbe1647e6a781be6b1534bc9b P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:09.265669 531 raft_consensus.cc:729] T 2036b24fbe1647e6a781be6b1534bc9b P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:13:09.265897 531 consensus_queue.cc:260] T 2036b24fbe1647e6a781be6b1534bc9b P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:09.266031 529 raft_consensus.cc:374] T 2036b24fbe1647e6a781be6b1534bc9b P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:09.266132 529 raft_consensus.cc:729] T 2036b24fbe1647e6a781be6b1534bc9b P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:13:09.266459 531 ts_tablet_manager.cc:1398] T 2036b24fbe1647e6a781be6b1534bc9b P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:13:09.266363 529 consensus_queue.cc:260] T 2036b24fbe1647e6a781be6b1534bc9b P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:09.268227 529 ts_tablet_manager.cc:1398] T 2036b24fbe1647e6a781be6b1534bc9b P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:13:09.268360 510 raft_consensus.cc:482] T 45e7a03f838d4beaae2080d523319d58 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:09.268481 510 raft_consensus.cc:504] T 45e7a03f838d4beaae2080d523319d58 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:09.269037 510 leader_election.cc:288] T 45e7a03f838d4beaae2080d523319d58 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:13:09.269567 31160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e7a03f838d4beaae2080d523319d58" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:13:09.269834 31160 raft_consensus.cc:2454] T 45e7a03f838d4beaae2080d523319d58 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:13:09.270066 31028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e7a03f838d4beaae2080d523319d58" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:13:09.270258 31028 raft_consensus.cc:2454] T 45e7a03f838d4beaae2080d523319d58 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:13:09.270665 28327 leader_election.cc:302] T 45e7a03f838d4beaae2080d523319d58 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:13:09.270931 510 raft_consensus.cc:2781] T 45e7a03f838d4beaae2080d523319d58 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:09.271015 510 raft_consensus.cc:482] T 45e7a03f838d4beaae2080d523319d58 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:09.271086 510 raft_consensus.cc:3037] T 45e7a03f838d4beaae2080d523319d58 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:09.273558 510 raft_consensus.cc:504] T 45e7a03f838d4beaae2080d523319d58 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:09.274183 510 leader_election.cc:288] T 45e7a03f838d4beaae2080d523319d58 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Requested vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:13:09.274617 31160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e7a03f838d4beaae2080d523319d58" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:13:09.274806 31160 raft_consensus.cc:3037] T 45e7a03f838d4beaae2080d523319d58 P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:09.276786 31160 raft_consensus.cc:2454] T 45e7a03f838d4beaae2080d523319d58 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:13:09.277118 28326 leader_election.cc:302] T 45e7a03f838d4beaae2080d523319d58 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:13:09.277352 510 raft_consensus.cc:2781] T 45e7a03f838d4beaae2080d523319d58 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:09.277431 31028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e7a03f838d4beaae2080d523319d58" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:13:09.277617 31028 raft_consensus.cc:3037] T 45e7a03f838d4beaae2080d523319d58 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:09.277649 510 raft_consensus.cc:686] T 45e7a03f838d4beaae2080d523319d58 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Becoming Leader. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Running, Role: LEADER
I20240503 08:13:09.278422 510 consensus_queue.cc:237] T 45e7a03f838d4beaae2080d523319d58 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:09.279671 31028 raft_consensus.cc:2454] T 45e7a03f838d4beaae2080d523319d58 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:13:09.283136 27858 catalog_manager.cc:5484] T 45e7a03f838d4beaae2080d523319d58 P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68). New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:09.404310 327 raft_consensus.cc:482] T 2036b24fbe1647e6a781be6b1534bc9b P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:09.404476 327 raft_consensus.cc:504] T 2036b24fbe1647e6a781be6b1534bc9b P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:09.405043 327 leader_election.cc:288] T 2036b24fbe1647e6a781be6b1534bc9b P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:13:09.405375 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2036b24fbe1647e6a781be6b1534bc9b" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:13:09.405526 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2036b24fbe1647e6a781be6b1534bc9b" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:13:09.405608 31159 raft_consensus.cc:2454] T 2036b24fbe1647e6a781be6b1534bc9b P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:13:09.405741 28003 raft_consensus.cc:2454] T 2036b24fbe1647e6a781be6b1534bc9b P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 0.
I20240503 08:13:09.406054 30827 leader_election.cc:302] T 2036b24fbe1647e6a781be6b1534bc9b P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:13:09.406366 327 raft_consensus.cc:2781] T 2036b24fbe1647e6a781be6b1534bc9b P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:09.406446 327 raft_consensus.cc:482] T 2036b24fbe1647e6a781be6b1534bc9b P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:09.406510 327 raft_consensus.cc:3037] T 2036b24fbe1647e6a781be6b1534bc9b P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:09.408303 327 raft_consensus.cc:504] T 2036b24fbe1647e6a781be6b1534bc9b P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:09.408823 327 leader_election.cc:288] T 2036b24fbe1647e6a781be6b1534bc9b P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Requested vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:13:09.409197 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2036b24fbe1647e6a781be6b1534bc9b" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:13:09.409232 28003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2036b24fbe1647e6a781be6b1534bc9b" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:13:09.409364 28003 raft_consensus.cc:3037] T 2036b24fbe1647e6a781be6b1534bc9b P 833582ff68c94269be364ebd187a75db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:09.409375 31159 raft_consensus.cc:3037] T 2036b24fbe1647e6a781be6b1534bc9b P b00f244c69df401997ea99d492218a98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:09.411278 28003 raft_consensus.cc:2454] T 2036b24fbe1647e6a781be6b1534bc9b P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:13:09.411522 31159 raft_consensus.cc:2454] T 2036b24fbe1647e6a781be6b1534bc9b P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:13:09.411664 30830 leader_election.cc:302] T 2036b24fbe1647e6a781be6b1534bc9b P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:13:09.411979 327 raft_consensus.cc:2781] T 2036b24fbe1647e6a781be6b1534bc9b P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:09.412242 327 raft_consensus.cc:686] T 2036b24fbe1647e6a781be6b1534bc9b P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Becoming Leader. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Running, Role: LEADER
I20240503 08:13:09.412508 327 consensus_queue.cc:237] T 2036b24fbe1647e6a781be6b1534bc9b P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:09.416806 27858 catalog_manager.cc:5484] T 2036b24fbe1647e6a781be6b1534bc9b P ce1496a47d084365ad786cd09e472a2c reported cstate change: term changed from 0 to 1, leader changed from <none> to ce1496a47d084365ad786cd09e472a2c (127.20.230.70). New cstate: current_term: 1 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:09.518914 510 consensus_queue.cc:1035] T 607a503b64d8440a81cf66ab158ae273 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:09.520380 32478 consensus_queue.cc:1035] T 607a503b64d8440a81cf66ab158ae273 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:09.541018 27858 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:13:09.555424 31160 raft_consensus.cc:1261] T 2036b24fbe1647e6a781be6b1534bc9b P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:09.555944 502 consensus_queue.cc:1035] T 2036b24fbe1647e6a781be6b1534bc9b P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:09.556032 28003 raft_consensus.cc:1261] T 2036b24fbe1647e6a781be6b1534bc9b P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:09.557843 327 consensus_queue.cc:1035] T 2036b24fbe1647e6a781be6b1534bc9b P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:09.565950 27858 catalog_manager.cc:5576] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081) does not have the latest schema for tablet 2036b24fbe1647e6a781be6b1534bc9b (table rebalancer_dynamic_table [id=16330782a2e54f7082e781905b55f1c7]). Expected version 1 got 0
I20240503 08:13:09.566991 547 tablet.cc:1722] T 2036b24fbe1647e6a781be6b1534bc9b P b00f244c69df401997ea99d492218a98: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:13:09.567751 547 tablet.cc:1620] T 2036b24fbe1647e6a781be6b1534bc9b P b00f244c69df401997ea99d492218a98: MemRowSet was empty: no flush needed.
I20240503 08:13:09.569442 27858 catalog_manager.cc:5576] TS b00f244c69df401997ea99d492218a98 (127.20.230.72:40893) does not have the latest schema for tablet 2036b24fbe1647e6a781be6b1534bc9b (table rebalancer_dynamic_table [id=16330782a2e54f7082e781905b55f1c7]). Expected version 1 got 0
I20240503 08:13:09.570164 546 tablet.cc:1722] T 2036b24fbe1647e6a781be6b1534bc9b P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:13:09.570668 546 tablet.cc:1620] T 2036b24fbe1647e6a781be6b1534bc9b P ce1496a47d084365ad786cd09e472a2c: MemRowSet was empty: no flush needed.
I20240503 08:13:09.580724 549 tablet.cc:1722] T 2036b24fbe1647e6a781be6b1534bc9b P 833582ff68c94269be364ebd187a75db: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:13:09.581223 549 tablet.cc:1620] T 2036b24fbe1647e6a781be6b1534bc9b P 833582ff68c94269be364ebd187a75db: MemRowSet was empty: no flush needed.
I20240503 08:13:09.581990 27858 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=16330782a2e54f7082e781905b55f1c7] alter complete (version 1)
I20240503 08:13:09.584491 27858 catalog_manager.cc:5576] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081) does not have the latest schema for tablet 2036b24fbe1647e6a781be6b1534bc9b (table rebalancer_dynamic_table [id=16330782a2e54f7082e781905b55f1c7]). Expected version 1 got 0
I20240503 08:13:09.642261 508 consensus_queue.cc:1035] T 48075c4a45d24e84bb2b9760710de3a0 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:09.649363 508 consensus_queue.cc:1035] T 48075c4a45d24e84bb2b9760710de3a0 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:09.689410 27858 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:13:09.696375 546 tablet.cc:1722] T 2036b24fbe1647e6a781be6b1534bc9b P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:13:09.696746 549 tablet.cc:1722] T 2036b24fbe1647e6a781be6b1534bc9b P 833582ff68c94269be364ebd187a75db: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:13:09.696875 546 tablet.cc:1620] T 2036b24fbe1647e6a781be6b1534bc9b P ce1496a47d084365ad786cd09e472a2c: MemRowSet was empty: no flush needed.
I20240503 08:13:09.697151 549 tablet.cc:1620] T 2036b24fbe1647e6a781be6b1534bc9b P 833582ff68c94269be364ebd187a75db: MemRowSet was empty: no flush needed.
I20240503 08:13:09.704041 547 tablet.cc:1722] T 2036b24fbe1647e6a781be6b1534bc9b P b00f244c69df401997ea99d492218a98: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:13:09.704506 547 tablet.cc:1620] T 2036b24fbe1647e6a781be6b1534bc9b P b00f244c69df401997ea99d492218a98: MemRowSet was empty: no flush needed.
I20240503 08:13:09.709645 27858 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table [id=16330782a2e54f7082e781905b55f1c7] alter complete (version 2)
I20240503 08:13:09.719897 510 consensus_queue.cc:1035] T 45e7a03f838d4beaae2080d523319d58 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:09.726716 510 consensus_queue.cc:1035] T 45e7a03f838d4beaae2080d523319d58 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:09.814013 27859 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:13:09.820996 546 tablet.cc:1722] T 2036b24fbe1647e6a781be6b1534bc9b P ce1496a47d084365ad786cd09e472a2c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:13:09.824575 547 tablet.cc:1722] T 2036b24fbe1647e6a781be6b1534bc9b P b00f244c69df401997ea99d492218a98: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:13:09.825313 549 tablet.cc:1722] T 2036b24fbe1647e6a781be6b1534bc9b P 833582ff68c94269be364ebd187a75db: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:13:09.829977 27862 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: rebalancer_dynamic_table_new_name [id=16330782a2e54f7082e781905b55f1c7] alter complete (version 3)
I20240503 08:13:09.855429 27862 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47966:
name: "rebalancer_extra_table_66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:13:09.864017 28242 tablet_service.cc:1467] Processing CreateTablet for tablet 1fca3d7f4b734eceb210e1186478eb72 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=80792788b9164bd4af02769ad074bd3a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:09.864188 30874 tablet_service.cc:1467] Processing CreateTablet for tablet 1fca3d7f4b734eceb210e1186478eb72 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=80792788b9164bd4af02769ad074bd3a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:09.864463 28242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fca3d7f4b734eceb210e1186478eb72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:09.864526 30874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fca3d7f4b734eceb210e1186478eb72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:09.864756 30742 tablet_service.cc:1467] Processing CreateTablet for tablet eeee6a265fc94e91bf9daa8301662e5f (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=80792788b9164bd4af02769ad074bd3a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:09.865113 30742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeee6a265fc94e91bf9daa8301662e5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:09.870098 28373 tablet_service.cc:1467] Processing CreateTablet for tablet 1fca3d7f4b734eceb210e1186478eb72 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=80792788b9164bd4af02769ad074bd3a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:09.870479 28373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fca3d7f4b734eceb210e1186478eb72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:09.871704 31006 tablet_service.cc:1467] Processing CreateTablet for tablet 49b145f6946647248c967cf4badf8b40 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=80792788b9164bd4af02769ad074bd3a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:09.871807 30873 tablet_service.cc:1467] Processing CreateTablet for tablet 49b145f6946647248c967cf4badf8b40 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=80792788b9164bd4af02769ad074bd3a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:09.872097 31006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49b145f6946647248c967cf4badf8b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:09.872228 30873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49b145f6946647248c967cf4badf8b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:09.874099 28372 tablet_service.cc:1467] Processing CreateTablet for tablet eeee6a265fc94e91bf9daa8301662e5f (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=80792788b9164bd4af02769ad074bd3a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:09.874143 28371 tablet_service.cc:1467] Processing CreateTablet for tablet 49b145f6946647248c967cf4badf8b40 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=80792788b9164bd4af02769ad074bd3a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:09.874442 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeee6a265fc94e91bf9daa8301662e5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:09.874715 28371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49b145f6946647248c967cf4badf8b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:09.875705 28243 tablet_service.cc:1467] Processing CreateTablet for tablet eeee6a265fc94e91bf9daa8301662e5f (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=80792788b9164bd4af02769ad074bd3a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:09.876086 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeee6a265fc94e91bf9daa8301662e5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:09.885710 560 tablet_bootstrap.cc:492] T 1fca3d7f4b734eceb210e1186478eb72 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:13:09.887104 560 tablet_bootstrap.cc:654] T 1fca3d7f4b734eceb210e1186478eb72 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:09.893168 556 tablet_bootstrap.cc:492] T eeee6a265fc94e91bf9daa8301662e5f P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:13:09.896813 556 tablet_bootstrap.cc:654] T eeee6a265fc94e91bf9daa8301662e5f P 6d1d7de19459435fbde1999e3dfd226b: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:09.897346 558 tablet_bootstrap.cc:492] T 49b145f6946647248c967cf4badf8b40 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:13:09.899437 558 tablet_bootstrap.cc:654] T 49b145f6946647248c967cf4badf8b40 P ced21ca983b749d0b4d4f6a0e2426bf8: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:09.900687 559 tablet_bootstrap.cc:492] T eeee6a265fc94e91bf9daa8301662e5f P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:13:09.901285 557 tablet_bootstrap.cc:492] T 1fca3d7f4b734eceb210e1186478eb72 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:13:09.902361 559 tablet_bootstrap.cc:654] T eeee6a265fc94e91bf9daa8301662e5f P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:09.903319 557 tablet_bootstrap.cc:654] T 1fca3d7f4b734eceb210e1186478eb72 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:09.905040 558 tablet_bootstrap.cc:492] T 49b145f6946647248c967cf4badf8b40 P ced21ca983b749d0b4d4f6a0e2426bf8: No bootstrap required, opened a new log
I20240503 08:13:09.905227 558 ts_tablet_manager.cc:1367] T 49b145f6946647248c967cf4badf8b40 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240503 08:13:09.906054 558 raft_consensus.cc:348] T 49b145f6946647248c967cf4badf8b40 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:09.906213 558 raft_consensus.cc:374] T 49b145f6946647248c967cf4badf8b40 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:09.906312 558 raft_consensus.cc:729] T 49b145f6946647248c967cf4badf8b40 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:13:09.906510 558 consensus_queue.cc:260] T 49b145f6946647248c967cf4badf8b40 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:09.906987 558 ts_tablet_manager.cc:1398] T 49b145f6946647248c967cf4badf8b40 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:09.907796 560 tablet_bootstrap.cc:492] T 1fca3d7f4b734eceb210e1186478eb72 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:13:09.907918 556 tablet_bootstrap.cc:492] T eeee6a265fc94e91bf9daa8301662e5f P 6d1d7de19459435fbde1999e3dfd226b: No bootstrap required, opened a new log
I20240503 08:13:09.908067 560 ts_tablet_manager.cc:1367] T 1fca3d7f4b734eceb210e1186478eb72 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.000s
I20240503 08:13:09.908176 556 ts_tablet_manager.cc:1367] T eeee6a265fc94e91bf9daa8301662e5f P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.005s
I20240503 08:13:09.908748 560 raft_consensus.cc:348] T 1fca3d7f4b734eceb210e1186478eb72 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:09.908913 560 raft_consensus.cc:374] T 1fca3d7f4b734eceb210e1186478eb72 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:09.909190 560 raft_consensus.cc:729] T 1fca3d7f4b734eceb210e1186478eb72 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:13:09.909372 560 consensus_queue.cc:260] T 1fca3d7f4b734eceb210e1186478eb72 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:09.910097 560 ts_tablet_manager.cc:1398] T 1fca3d7f4b734eceb210e1186478eb72 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:09.910941 559 tablet_bootstrap.cc:492] T eeee6a265fc94e91bf9daa8301662e5f P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:13:09.911064 559 ts_tablet_manager.cc:1367] T eeee6a265fc94e91bf9daa8301662e5f P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240503 08:13:09.911845 559 raft_consensus.cc:348] T eeee6a265fc94e91bf9daa8301662e5f P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:09.912153 559 raft_consensus.cc:374] T eeee6a265fc94e91bf9daa8301662e5f P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:09.912701 559 raft_consensus.cc:729] T eeee6a265fc94e91bf9daa8301662e5f P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:13:09.913033 559 consensus_queue.cc:260] T eeee6a265fc94e91bf9daa8301662e5f P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:09.913853 556 raft_consensus.cc:348] T eeee6a265fc94e91bf9daa8301662e5f P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:09.914736 556 raft_consensus.cc:374] T eeee6a265fc94e91bf9daa8301662e5f P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:09.914875 556 raft_consensus.cc:729] T eeee6a265fc94e91bf9daa8301662e5f P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:13:09.915566 559 ts_tablet_manager.cc:1398] T eeee6a265fc94e91bf9daa8301662e5f P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:13:09.915894 559 tablet_bootstrap.cc:492] T 1fca3d7f4b734eceb210e1186478eb72 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:13:09.917833 560 tablet_bootstrap.cc:492] T eeee6a265fc94e91bf9daa8301662e5f P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:13:09.918817 557 tablet_bootstrap.cc:492] T 1fca3d7f4b734eceb210e1186478eb72 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:13:09.919049 557 ts_tablet_manager.cc:1367] T 1fca3d7f4b734eceb210e1186478eb72 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.000s
I20240503 08:13:09.919131 560 tablet_bootstrap.cc:654] T eeee6a265fc94e91bf9daa8301662e5f P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:09.919461 559 tablet_bootstrap.cc:654] T 1fca3d7f4b734eceb210e1186478eb72 P aae860d7ae2b4002bf7794c278f9cc56: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:09.919862 557 raft_consensus.cc:348] T 1fca3d7f4b734eceb210e1186478eb72 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:09.920217 557 raft_consensus.cc:374] T 1fca3d7f4b734eceb210e1186478eb72 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:09.920356 557 raft_consensus.cc:729] T 1fca3d7f4b734eceb210e1186478eb72 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:13:09.921209 557 consensus_queue.cc:260] T 1fca3d7f4b734eceb210e1186478eb72 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:09.923913 557 ts_tablet_manager.cc:1398] T 1fca3d7f4b734eceb210e1186478eb72 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:13:09.924224 557 tablet_bootstrap.cc:492] T 49b145f6946647248c967cf4badf8b40 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:13:09.925984 557 tablet_bootstrap.cc:654] T 49b145f6946647248c967cf4badf8b40 P ce1496a47d084365ad786cd09e472a2c: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:09.925896 556 consensus_queue.cc:260] T eeee6a265fc94e91bf9daa8301662e5f P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:09.927783 556 ts_tablet_manager.cc:1398] T eeee6a265fc94e91bf9daa8301662e5f P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20240503 08:13:09.934700 557 tablet_bootstrap.cc:492] T 49b145f6946647248c967cf4badf8b40 P ce1496a47d084365ad786cd09e472a2c: No bootstrap required, opened a new log
I20240503 08:13:09.934895 557 ts_tablet_manager.cc:1367] T 49b145f6946647248c967cf4badf8b40 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.005s
I20240503 08:13:09.935277 27862 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:13:09.935503 27862 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47966:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:13:09.935745 557 raft_consensus.cc:348] T 49b145f6946647248c967cf4badf8b40 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:09.935992 557 raft_consensus.cc:374] T 49b145f6946647248c967cf4badf8b40 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:09.936105 557 raft_consensus.cc:729] T 49b145f6946647248c967cf4badf8b40 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:13:09.936379 557 consensus_queue.cc:260] T 49b145f6946647248c967cf4badf8b40 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:09.936992 557 ts_tablet_manager.cc:1398] T 49b145f6946647248c967cf4badf8b40 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:13:09.938855 559 tablet_bootstrap.cc:492] T 1fca3d7f4b734eceb210e1186478eb72 P aae860d7ae2b4002bf7794c278f9cc56: No bootstrap required, opened a new log
I20240503 08:13:09.939011 559 ts_tablet_manager.cc:1367] T 1fca3d7f4b734eceb210e1186478eb72 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.008s
I20240503 08:13:09.939783 559 raft_consensus.cc:348] T 1fca3d7f4b734eceb210e1186478eb72 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:09.939997 559 raft_consensus.cc:374] T 1fca3d7f4b734eceb210e1186478eb72 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:09.940100 559 raft_consensus.cc:729] T 1fca3d7f4b734eceb210e1186478eb72 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:13:09.940302 27862 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2b7d1c3c8a84f53b716ab764eecfa71: Sending DeleteTablet for 3 replicas of tablet 2036b24fbe1647e6a781be6b1534bc9b
I20240503 08:13:09.940320 559 consensus_queue.cc:260] T 1fca3d7f4b734eceb210e1186478eb72 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:09.941152 30874 tablet_service.cc:1514] Processing DeleteTablet for tablet 2036b24fbe1647e6a781be6b1534bc9b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:09 UTC) from {username='slave'} at 127.0.0.1:34426
I20240503 08:13:09.941322 27983 tablet_service.cc:1514] Processing DeleteTablet for tablet 2036b24fbe1647e6a781be6b1534bc9b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:09 UTC) from {username='slave'} at 127.0.0.1:55610
I20240503 08:13:09.941509 31141 tablet_service.cc:1514] Processing DeleteTablet for tablet 2036b24fbe1647e6a781be6b1534bc9b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:09 UTC) from {username='slave'} at 127.0.0.1:45814
I20240503 08:13:09.941840 569 tablet_replica.cc:330] T 2036b24fbe1647e6a781be6b1534bc9b P ce1496a47d084365ad786cd09e472a2c: stopping tablet replica
I20240503 08:13:09.942266 559 ts_tablet_manager.cc:1398] T 1fca3d7f4b734eceb210e1186478eb72 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:13:09.944167 569 raft_consensus.cc:2229] T 2036b24fbe1647e6a781be6b1534bc9b P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:13:09.944571 569 raft_consensus.cc:2258] T 2036b24fbe1647e6a781be6b1534bc9b P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:09.945430 569 ts_tablet_manager.cc:1875] T 2036b24fbe1647e6a781be6b1534bc9b P ce1496a47d084365ad786cd09e472a2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:09.950889 560 tablet_bootstrap.cc:492] T eeee6a265fc94e91bf9daa8301662e5f P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:13:09.951594 560 ts_tablet_manager.cc:1367] T eeee6a265fc94e91bf9daa8301662e5f P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.034s user 0.001s sys 0.004s
I20240503 08:13:09.954394 569 ts_tablet_manager.cc:1888] T 2036b24fbe1647e6a781be6b1534bc9b P ce1496a47d084365ad786cd09e472a2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:13:09.954313 560 raft_consensus.cc:348] T eeee6a265fc94e91bf9daa8301662e5f P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:09.954511 560 raft_consensus.cc:374] T eeee6a265fc94e91bf9daa8301662e5f P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:09.954524 569 log.cc:1196] T 2036b24fbe1647e6a781be6b1534bc9b P ce1496a47d084365ad786cd09e472a2c: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal/wals/2036b24fbe1647e6a781be6b1534bc9b
I20240503 08:13:09.954934 569 ts_tablet_manager.cc:1909] T 2036b24fbe1647e6a781be6b1534bc9b P ce1496a47d084365ad786cd09e472a2c: Deleting consensus metadata
I20240503 08:13:09.954950 560 raft_consensus.cc:729] T eeee6a265fc94e91bf9daa8301662e5f P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:13:09.955610 560 consensus_queue.cc:260] T eeee6a265fc94e91bf9daa8301662e5f P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:09.957151 27846 catalog_manager.cc:4830] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): tablet 2036b24fbe1647e6a781be6b1534bc9b (table rebalancer_dynamic_table_new_name [id=16330782a2e54f7082e781905b55f1c7]) successfully deleted
I20240503 08:13:09.956828 560 ts_tablet_manager.cc:1398] T eeee6a265fc94e91bf9daa8301662e5f P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240503 08:13:09.957518 560 tablet_bootstrap.cc:492] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:13:09.959862 571 tablet_replica.cc:330] T 2036b24fbe1647e6a781be6b1534bc9b P b00f244c69df401997ea99d492218a98: stopping tablet replica
I20240503 08:13:09.960163 571 raft_consensus.cc:2229] T 2036b24fbe1647e6a781be6b1534bc9b P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:09.961269 560 tablet_bootstrap.cc:654] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:09.961515 571 raft_consensus.cc:2258] T 2036b24fbe1647e6a781be6b1534bc9b P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:09.962383 571 ts_tablet_manager.cc:1875] T 2036b24fbe1647e6a781be6b1534bc9b P b00f244c69df401997ea99d492218a98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:09.963927 570 tablet_replica.cc:330] T 2036b24fbe1647e6a781be6b1534bc9b P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:13:09.964113 570 raft_consensus.cc:2229] T 2036b24fbe1647e6a781be6b1534bc9b P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:09.964296 570 raft_consensus.cc:2258] T 2036b24fbe1647e6a781be6b1534bc9b P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:09.965252 570 ts_tablet_manager.cc:1875] T 2036b24fbe1647e6a781be6b1534bc9b P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:09.967186 571 ts_tablet_manager.cc:1888] T 2036b24fbe1647e6a781be6b1534bc9b P b00f244c69df401997ea99d492218a98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:13:09.969009 571 log.cc:1196] T 2036b24fbe1647e6a781be6b1534bc9b P b00f244c69df401997ea99d492218a98: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal/wals/2036b24fbe1647e6a781be6b1534bc9b
I20240503 08:13:09.969658 571 ts_tablet_manager.cc:1909] T 2036b24fbe1647e6a781be6b1534bc9b P b00f244c69df401997ea99d492218a98: Deleting consensus metadata
I20240503 08:13:09.970494 27846 catalog_manager.cc:4830] TS b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): tablet 2036b24fbe1647e6a781be6b1534bc9b (table rebalancer_dynamic_table_new_name [id=16330782a2e54f7082e781905b55f1c7]) successfully deleted
I20240503 08:13:09.972393 570 ts_tablet_manager.cc:1888] T 2036b24fbe1647e6a781be6b1534bc9b P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:13:09.972573 570 log.cc:1196] T 2036b24fbe1647e6a781be6b1534bc9b P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/2036b24fbe1647e6a781be6b1534bc9b
I20240503 08:13:09.972995 570 ts_tablet_manager.cc:1909] T 2036b24fbe1647e6a781be6b1534bc9b P 833582ff68c94269be364ebd187a75db: Deleting consensus metadata
I20240503 08:13:09.973407 560 tablet_bootstrap.cc:492] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2: No bootstrap required, opened a new log
I20240503 08:13:09.973559 560 ts_tablet_manager.cc:1367] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.001s
I20240503 08:13:09.974324 560 raft_consensus.cc:348] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:09.974539 560 raft_consensus.cc:374] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:09.974624 560 raft_consensus.cc:729] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:13:09.974891 560 consensus_queue.cc:260] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:09.975391 560 ts_tablet_manager.cc:1398] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:09.976263 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet 2036b24fbe1647e6a781be6b1534bc9b (table rebalancer_dynamic_table_new_name [id=16330782a2e54f7082e781905b55f1c7]) successfully deleted
I20240503 08:13:09.988104 510 raft_consensus.cc:482] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:09.988229 510 raft_consensus.cc:504] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:09.988806 510 leader_election.cc:288] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:13:09.989364 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b145f6946647248c967cf4badf8b40" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:13:09.989537 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b145f6946647248c967cf4badf8b40" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:13:09.989595 30895 raft_consensus.cc:2454] T 49b145f6946647248c967cf4badf8b40 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:13:09.989733 31027 raft_consensus.cc:2454] T 49b145f6946647248c967cf4badf8b40 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:13:09.990175 28326 leader_election.cc:302] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:13:09.990440 510 raft_consensus.cc:2781] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:09.990507 510 raft_consensus.cc:482] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:09.990562 510 raft_consensus.cc:3037] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:09.993079 510 raft_consensus.cc:504] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:09.993975 510 leader_election.cc:288] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Requested vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:13:09.994103 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b145f6946647248c967cf4badf8b40" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:13:09.994275 31027 raft_consensus.cc:3037] T 49b145f6946647248c967cf4badf8b40 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:09.996657 31027 raft_consensus.cc:2454] T 49b145f6946647248c967cf4badf8b40 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:13:09.996968 28327 leader_election.cc:302] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:13:09.996973 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b145f6946647248c967cf4badf8b40" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:13:09.997148 30895 raft_consensus.cc:3037] T 49b145f6946647248c967cf4badf8b40 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:09.997200 510 raft_consensus.cc:2781] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:09.997545 510 raft_consensus.cc:686] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Becoming Leader. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Running, Role: LEADER
I20240503 08:13:09.999168 510 consensus_queue.cc:237] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:09.999578 30895 raft_consensus.cc:2454] T 49b145f6946647248c967cf4badf8b40 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:13:10.005170 27862 catalog_manager.cc:5484] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68). New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:13:10.063156 510 raft_consensus.cc:482] T 1fca3d7f4b734eceb210e1186478eb72 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:10.063372 510 raft_consensus.cc:504] T 1fca3d7f4b734eceb210e1186478eb72 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:10.064038 510 leader_election.cc:288] T 1fca3d7f4b734eceb210e1186478eb72 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:13:10.064388 28261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fca3d7f4b734eceb210e1186478eb72" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:13:10.064545 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fca3d7f4b734eceb210e1186478eb72" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:13:10.064626 28261 raft_consensus.cc:2454] T 1fca3d7f4b734eceb210e1186478eb72 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:13:10.064752 30895 raft_consensus.cc:2454] T 1fca3d7f4b734eceb210e1186478eb72 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:13:10.065100 28327 leader_election.cc:302] T 1fca3d7f4b734eceb210e1186478eb72 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:13:10.065410 510 raft_consensus.cc:2781] T 1fca3d7f4b734eceb210e1186478eb72 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:10.065532 510 raft_consensus.cc:482] T 1fca3d7f4b734eceb210e1186478eb72 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:10.065665 510 raft_consensus.cc:3037] T 1fca3d7f4b734eceb210e1186478eb72 P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:10.067948 510 raft_consensus.cc:504] T 1fca3d7f4b734eceb210e1186478eb72 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:10.068836 28261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fca3d7f4b734eceb210e1186478eb72" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:13:10.069044 28261 raft_consensus.cc:3037] T 1fca3d7f4b734eceb210e1186478eb72 P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:10.069056 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fca3d7f4b734eceb210e1186478eb72" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:13:10.069240 30895 raft_consensus.cc:3037] T 1fca3d7f4b734eceb210e1186478eb72 P ce1496a47d084365ad786cd09e472a2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:10.071835 30895 raft_consensus.cc:2454] T 1fca3d7f4b734eceb210e1186478eb72 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:13:10.068666 510 leader_election.cc:288] T 1fca3d7f4b734eceb210e1186478eb72 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:13:10.072293 28326 leader_election.cc:302] T 1fca3d7f4b734eceb210e1186478eb72 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:13:10.072544 510 raft_consensus.cc:2781] T 1fca3d7f4b734eceb210e1186478eb72 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:10.072757 510 raft_consensus.cc:686] T 1fca3d7f4b734eceb210e1186478eb72 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Becoming Leader. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Running, Role: LEADER
I20240503 08:13:10.073009 510 consensus_queue.cc:237] T 1fca3d7f4b734eceb210e1186478eb72 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:10.075173 28261 raft_consensus.cc:2454] T 1fca3d7f4b734eceb210e1186478eb72 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:13:10.078821 27862 catalog_manager.cc:5484] T 1fca3d7f4b734eceb210e1186478eb72 P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68). New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:10.182224 510 raft_consensus.cc:482] T eeee6a265fc94e91bf9daa8301662e5f P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:10.182454 510 raft_consensus.cc:504] T eeee6a265fc94e91bf9daa8301662e5f P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:10.183174 510 leader_election.cc:288] T eeee6a265fc94e91bf9daa8301662e5f P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:13:10.183511 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeee6a265fc94e91bf9daa8301662e5f" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:13:10.183717 30761 raft_consensus.cc:2454] T eeee6a265fc94e91bf9daa8301662e5f P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:13:10.183717 28263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeee6a265fc94e91bf9daa8301662e5f" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56" is_pre_election: true
I20240503 08:13:10.183912 28263 raft_consensus.cc:2454] T eeee6a265fc94e91bf9daa8301662e5f P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 0.
I20240503 08:13:10.184067 28328 leader_election.cc:302] T eeee6a265fc94e91bf9daa8301662e5f P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:13:10.184610 510 raft_consensus.cc:2781] T eeee6a265fc94e91bf9daa8301662e5f P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:10.184679 510 raft_consensus.cc:482] T eeee6a265fc94e91bf9daa8301662e5f P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:10.184718 510 raft_consensus.cc:3037] T eeee6a265fc94e91bf9daa8301662e5f P d09a11a03989421b9f28019dc1c14cc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:10.187052 510 raft_consensus.cc:504] T eeee6a265fc94e91bf9daa8301662e5f P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:10.187541 510 leader_election.cc:288] T eeee6a265fc94e91bf9daa8301662e5f P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Requested vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:13:10.187888 28263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeee6a265fc94e91bf9daa8301662e5f" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:13:10.187959 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeee6a265fc94e91bf9daa8301662e5f" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:13:10.188023 28263 raft_consensus.cc:3037] T eeee6a265fc94e91bf9daa8301662e5f P aae860d7ae2b4002bf7794c278f9cc56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:10.188112 30761 raft_consensus.cc:3037] T eeee6a265fc94e91bf9daa8301662e5f P 6d1d7de19459435fbde1999e3dfd226b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:10.189950 28263 raft_consensus.cc:2454] T eeee6a265fc94e91bf9daa8301662e5f P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:13:10.190245 28327 leader_election.cc:302] T eeee6a265fc94e91bf9daa8301662e5f P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:13:10.190245 30761 raft_consensus.cc:2454] T eeee6a265fc94e91bf9daa8301662e5f P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:13:10.190459 510 raft_consensus.cc:2781] T eeee6a265fc94e91bf9daa8301662e5f P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:10.190569 510 raft_consensus.cc:686] T eeee6a265fc94e91bf9daa8301662e5f P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Becoming Leader. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Running, Role: LEADER
I20240503 08:13:10.190811 510 consensus_queue.cc:237] T eeee6a265fc94e91bf9daa8301662e5f P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:10.194000 27862 catalog_manager.cc:5484] T eeee6a265fc94e91bf9daa8301662e5f P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68). New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:10.479085 32478 consensus_queue.cc:1035] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:10.507056 32478 consensus_queue.cc:1035] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:10.528488 580 consensus_queue.cc:1035] T 1fca3d7f4b734eceb210e1186478eb72 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:10.576301 580 consensus_queue.cc:1035] T 1fca3d7f4b734eceb210e1186478eb72 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:10.583242 510 consensus_queue.cc:1035] T eeee6a265fc94e91bf9daa8301662e5f P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:10.633574 510 consensus_queue.cc:1035] T eeee6a265fc94e91bf9daa8301662e5f P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:13:10.636670 28196 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.72:40893, user_credentials={real_user=slave}} blocked reactor thread for 48251.6us
W20240503 08:13:10.639015 28198 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.66:37143, user_credentials={real_user=slave}} blocked reactor thread for 49750.7us
I20240503 08:13:11.085340 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:11.127833 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:11.185503 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:11.189070 28243 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:11.221558 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:11.245539 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:11.278798 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:11.295071 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:11.937597 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:11.965848 28243 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:11.978379 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:11.994030 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:12.008669 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:12.093276 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:12.099965 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240503 08:13:12.141109 30695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.71:41915, user_credentials={real_user=slave}} blocked reactor thread for 97020.7us
I20240503 08:13:12.162075 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:12.628048 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:12.633247 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:12.654316 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:12.698427 28243 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:12.704277 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240503 08:13:12.739300 28199 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.65:39081, user_credentials={real_user=slave}} blocked reactor thread for 58382.2us
I20240503 08:13:12.781742 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:12.809808 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:12.811489 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:13.106505 28003 consensus_queue.cc:237] T 417cedc3fa5c4a85a7a130005291d64b P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
W20240503 08:13:13.115051 27938 consensus_peers.cc:485] T 417cedc3fa5c4a85a7a130005291d64b P 833582ff68c94269be364ebd187a75db -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 417cedc3fa5c4a85a7a130005291d64b. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:13.116540 31159 raft_consensus.cc:1261] T 417cedc3fa5c4a85a7a130005291d64b P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:13.116750 31028 raft_consensus.cc:1261] T 417cedc3fa5c4a85a7a130005291d64b P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:13.117208 497 consensus_queue.cc:1035] T 417cedc3fa5c4a85a7a130005291d64b P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:13.117460 656 consensus_queue.cc:1035] T 417cedc3fa5c4a85a7a130005291d64b P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:13.127185 656 raft_consensus.cc:2932] T 417cedc3fa5c4a85a7a130005291d64b P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:13.127558 31160 raft_consensus.cc:2932] T 417cedc3fa5c4a85a7a130005291d64b P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:13.132138 31028 raft_consensus.cc:2932] T 417cedc3fa5c4a85a7a130005291d64b P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:13.140075 27861 catalog_manager.cc:5484] T 417cedc3fa5c4a85a7a130005291d64b P ced21ca983b749d0b4d4f6a0e2426bf8 reported cstate change: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:13.181058 30895 consensus_queue.cc:237] T c9055668b86e4dc5aa321a4d29beaf74 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:13.183763 28390 raft_consensus.cc:1261] T c9055668b86e4dc5aa321a4d29beaf74 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:13.183924 31028 raft_consensus.cc:1261] T c9055668b86e4dc5aa321a4d29beaf74 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:13:13.184082 30830 consensus_peers.cc:485] T c9055668b86e4dc5aa321a4d29beaf74 P ce1496a47d084365ad786cd09e472a2c -> Peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Couldn't send request to peer 833582ff68c94269be364ebd187a75db. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c9055668b86e4dc5aa321a4d29beaf74. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:13.184387 553 consensus_queue.cc:1035] T c9055668b86e4dc5aa321a4d29beaf74 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:13.184933 579 consensus_queue.cc:1035] T c9055668b86e4dc5aa321a4d29beaf74 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:13.190948 553 raft_consensus.cc:2932] T c9055668b86e4dc5aa321a4d29beaf74 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:13.191570 31028 raft_consensus.cc:2932] T c9055668b86e4dc5aa321a4d29beaf74 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:13.195868 28390 raft_consensus.cc:2932] T c9055668b86e4dc5aa321a4d29beaf74 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:13.199857 27862 catalog_manager.cc:5484] T c9055668b86e4dc5aa321a4d29beaf74 P ced21ca983b749d0b4d4f6a0e2426bf8 reported cstate change: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New cstate: current_term: 1 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:13.255157 28003 consensus_queue.cc:237] T 3042212758bf4aef8be5c5d12d10113c P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:13.257431 28390 raft_consensus.cc:1261] T 3042212758bf4aef8be5c5d12d10113c P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:13.257441 28262 raft_consensus.cc:1261] T 3042212758bf4aef8be5c5d12d10113c P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:13.257980 656 consensus_queue.cc:1035] T 3042212758bf4aef8be5c5d12d10113c P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:13.258255 497 consensus_queue.cc:1035] T 3042212758bf4aef8be5c5d12d10113c P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:13:13.259074 27936 consensus_peers.cc:485] T 3042212758bf4aef8be5c5d12d10113c P 833582ff68c94269be364ebd187a75db -> Peer ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): Couldn't send request to peer ce1496a47d084365ad786cd09e472a2c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3042212758bf4aef8be5c5d12d10113c. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:13.263060 497 raft_consensus.cc:2932] T 3042212758bf4aef8be5c5d12d10113c P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:13.266943 28390 raft_consensus.cc:2932] T 3042212758bf4aef8be5c5d12d10113c P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:13.270571 28262 raft_consensus.cc:2932] T 3042212758bf4aef8be5c5d12d10113c P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:13.274747 27859 catalog_manager.cc:5484] T 3042212758bf4aef8be5c5d12d10113c P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: config changed from index -1 to 4, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:13.314061 30760 consensus_queue.cc:237] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:13.316643 28262 raft_consensus.cc:1261] T 7f2743ab5fa24c9899b67ea52f48f9d8 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:13:13.317384 30697 consensus_peers.cc:485] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b -> Peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): Couldn't send request to peer d09a11a03989421b9f28019dc1c14cc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7f2743ab5fa24c9899b67ea52f48f9d8. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:13.317548 28003 raft_consensus.cc:1261] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:13.317925 878 consensus_queue.cc:1035] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:13.318415 732 consensus_queue.cc:1035] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:13.323129 732 raft_consensus.cc:2932] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:13.323655 28262 raft_consensus.cc:2932] T 7f2743ab5fa24c9899b67ea52f48f9d8 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:13.329150 27862 catalog_manager.cc:5484] T 7f2743ab5fa24c9899b67ea52f48f9d8 P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:13.333461 28003 raft_consensus.cc:2932] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:13.383212 28262 consensus_queue.cc:237] T 2222ff7fb5184d6c86010e1475d6c1f7 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:13.385654 31160 raft_consensus.cc:1261] T 2222ff7fb5184d6c86010e1475d6c1f7 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:13.385653 30760 raft_consensus.cc:1261] T 2222ff7fb5184d6c86010e1475d6c1f7 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:13:13.385794 28198 consensus_peers.cc:485] T 2222ff7fb5184d6c86010e1475d6c1f7 P aae860d7ae2b4002bf7794c278f9cc56 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2222ff7fb5184d6c86010e1475d6c1f7. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:13.386217 865 consensus_queue.cc:1035] T 2222ff7fb5184d6c86010e1475d6c1f7 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:13.386476 589 consensus_queue.cc:1035] T 2222ff7fb5184d6c86010e1475d6c1f7 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:13.389308 589 raft_consensus.cc:2932] T 2222ff7fb5184d6c86010e1475d6c1f7 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:13.392858 31160 raft_consensus.cc:2932] T 2222ff7fb5184d6c86010e1475d6c1f7 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:13.395331 30760 raft_consensus.cc:2932] T 2222ff7fb5184d6c86010e1475d6c1f7 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:13.397037 27861 catalog_manager.cc:5484] T 2222ff7fb5184d6c86010e1475d6c1f7 P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:13.438190 30760 consensus_queue.cc:237] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:13.440985 31028 raft_consensus.cc:1261] T f9b4f775b9914bdc98b9b582072cb1ab P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:13.441030 28262 raft_consensus.cc:1261] T f9b4f775b9914bdc98b9b582072cb1ab P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:13:13.441478 30697 consensus_peers.cc:485] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b -> Peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Couldn't send request to peer 833582ff68c94269be364ebd187a75db. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f9b4f775b9914bdc98b9b582072cb1ab. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:13.441493 878 consensus_queue.cc:1035] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:13.441833 590 consensus_queue.cc:1035] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:13.444386 878 raft_consensus.cc:2932] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:13.444975 28262 raft_consensus.cc:2932] T f9b4f775b9914bdc98b9b582072cb1ab P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:13.448787 31028 raft_consensus.cc:2932] T f9b4f775b9914bdc98b9b582072cb1ab P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:13.450855 27862 catalog_manager.cc:5484] T f9b4f775b9914bdc98b9b582072cb1ab P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:13.484184 28002 consensus_queue.cc:237] T d3da6397e0c34c8f96933963dd4fa65e P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:13.487161 28262 raft_consensus.cc:1261] T d3da6397e0c34c8f96933963dd4fa65e P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:13:13.487493 27936 consensus_peers.cc:485] T d3da6397e0c34c8f96933963dd4fa65e P 833582ff68c94269be364ebd187a75db -> Peer ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): Couldn't send request to peer ce1496a47d084365ad786cd09e472a2c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d3da6397e0c34c8f96933963dd4fa65e. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:13.487649 28391 raft_consensus.cc:1261] T d3da6397e0c34c8f96933963dd4fa65e P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:13.488109 497 consensus_queue.cc:1035] T d3da6397e0c34c8f96933963dd4fa65e P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:13.490816 497 consensus_queue.cc:1035] T d3da6397e0c34c8f96933963dd4fa65e P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:13.502918 497 raft_consensus.cc:2932] T d3da6397e0c34c8f96933963dd4fa65e P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:13.503643 28262 raft_consensus.cc:2932] T d3da6397e0c34c8f96933963dd4fa65e P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:13.510270 28391 raft_consensus.cc:2932] T d3da6397e0c34c8f96933963dd4fa65e P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:13.510653 27858 catalog_manager.cc:5484] T d3da6397e0c34c8f96933963dd4fa65e P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: config changed from index -1 to 4, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:13.582078 31160 consensus_queue.cc:237] T 013e00c67a1c4ae9b3c1a050b966e9a5 P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:13.584576 30760 raft_consensus.cc:1261] T 013e00c67a1c4ae9b3c1a050b966e9a5 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:13.584803 31027 raft_consensus.cc:1261] T 013e00c67a1c4ae9b3c1a050b966e9a5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:13:13.585057 31097 consensus_peers.cc:485] T 013e00c67a1c4ae9b3c1a050b966e9a5 P b00f244c69df401997ea99d492218a98 -> Peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): Couldn't send request to peer d09a11a03989421b9f28019dc1c14cc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 013e00c67a1c4ae9b3c1a050b966e9a5. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:13.585330 473 consensus_queue.cc:1035] T 013e00c67a1c4ae9b3c1a050b966e9a5 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:13.585744 955 consensus_queue.cc:1035] T 013e00c67a1c4ae9b3c1a050b966e9a5 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:13.588846 473 raft_consensus.cc:2932] T 013e00c67a1c4ae9b3c1a050b966e9a5 P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:13.589426 30760 raft_consensus.cc:2932] T 013e00c67a1c4ae9b3c1a050b966e9a5 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:13.594839 27861 catalog_manager.cc:5484] T 013e00c67a1c4ae9b3c1a050b966e9a5 P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New cstate: current_term: 1 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:13.595531 31028 raft_consensus.cc:2932] T 013e00c67a1c4ae9b3c1a050b966e9a5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:13.610790 966 ts_tablet_manager.cc:902] T 417cedc3fa5c4a85a7a130005291d64b P 1b411c6bb25549368041773d7b710d48: Initiating tablet copy from peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:13:13.612730 966 tablet_copy_client.cc:308] T 417cedc3fa5c4a85a7a130005291d64b P 1b411c6bb25549368041773d7b710d48: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.65:39081
I20240503 08:13:13.613636 28023 tablet_copy_service.cc:140] P 833582ff68c94269be364ebd187a75db: Received BeginTabletCopySession request for tablet 417cedc3fa5c4a85a7a130005291d64b from peer 1b411c6bb25549368041773d7b710d48 ({username='slave'} at 127.20.230.66:42309)
I20240503 08:13:13.613848 28023 tablet_copy_service.cc:161] P 833582ff68c94269be364ebd187a75db: Beginning new tablet copy session on tablet 417cedc3fa5c4a85a7a130005291d64b from peer 1b411c6bb25549368041773d7b710d48 at {username='slave'} at 127.20.230.66:42309: session id = 1b411c6bb25549368041773d7b710d48-417cedc3fa5c4a85a7a130005291d64b
I20240503 08:13:13.616735 28023 tablet_copy_source_session.cc:205] T 417cedc3fa5c4a85a7a130005291d64b P 833582ff68c94269be364ebd187a75db: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:13.618937 966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 417cedc3fa5c4a85a7a130005291d64b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:13.626255 966 tablet_copy_client.cc:783] T 417cedc3fa5c4a85a7a130005291d64b P 1b411c6bb25549368041773d7b710d48: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:13.626698 966 tablet_copy_client.cc:647] T 417cedc3fa5c4a85a7a130005291d64b P 1b411c6bb25549368041773d7b710d48: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:13.628536 966 tablet_copy_client.cc:523] T 417cedc3fa5c4a85a7a130005291d64b P 1b411c6bb25549368041773d7b710d48: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:13.631974 966 tablet_bootstrap.cc:492] T 417cedc3fa5c4a85a7a130005291d64b P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:13:13.661224 966 tablet_bootstrap.cc:492] T 417cedc3fa5c4a85a7a130005291d64b P 1b411c6bb25549368041773d7b710d48: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:13.661603 966 tablet_bootstrap.cc:492] T 417cedc3fa5c4a85a7a130005291d64b P 1b411c6bb25549368041773d7b710d48: Bootstrap complete.
I20240503 08:13:13.661870 966 ts_tablet_manager.cc:1367] T 417cedc3fa5c4a85a7a130005291d64b P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.030s user 0.009s sys 0.014s
I20240503 08:13:13.662840 966 raft_consensus.cc:348] T 417cedc3fa5c4a85a7a130005291d64b P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:13.663066 966 raft_consensus.cc:729] T 417cedc3fa5c4a85a7a130005291d64b P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:13:13.663311 966 consensus_queue.cc:260] T 417cedc3fa5c4a85a7a130005291d64b P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:13.663916 966 ts_tablet_manager.cc:1398] T 417cedc3fa5c4a85a7a130005291d64b P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:13:13.665172 28023 tablet_copy_service.cc:342] P 833582ff68c94269be364ebd187a75db: Request end of tablet copy session 1b411c6bb25549368041773d7b710d48-417cedc3fa5c4a85a7a130005291d64b received from {username='slave'} at 127.20.230.66:42309
I20240503 08:13:13.665336 28023 tablet_copy_service.cc:433] P 833582ff68c94269be364ebd187a75db: ending tablet copy session 1b411c6bb25549368041773d7b710d48-417cedc3fa5c4a85a7a130005291d64b on tablet 417cedc3fa5c4a85a7a130005291d64b with peer 1b411c6bb25549368041773d7b710d48
I20240503 08:13:13.680589 31028 consensus_queue.cc:237] T 7df30eb244914dcabc6965324eced236 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
W20240503 08:13:13.693632 30961 consensus_peers.cc:485] T 7df30eb244914dcabc6965324eced236 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7df30eb244914dcabc6965324eced236. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:13.694125 31160 raft_consensus.cc:1261] T 7df30eb244914dcabc6965324eced236 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer ced21ca983b749d0b4d4f6a0e2426bf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:13.694798 850 consensus_queue.cc:1035] T 7df30eb244914dcabc6965324eced236 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:13.695036 30895 raft_consensus.cc:1261] T 7df30eb244914dcabc6965324eced236 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer ced21ca983b749d0b4d4f6a0e2426bf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:13.695564 850 consensus_queue.cc:1035] T 7df30eb244914dcabc6965324eced236 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:13.698639 850 raft_consensus.cc:2932] T 7df30eb244914dcabc6965324eced236 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:13.699756 30895 raft_consensus.cc:2932] T 7df30eb244914dcabc6965324eced236 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:13.705068 27862 catalog_manager.cc:5484] T 7df30eb244914dcabc6965324eced236 P ce1496a47d084365ad786cd09e472a2c reported cstate change: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New cstate: current_term: 1 leader_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:13.705888 31160 raft_consensus.cc:2932] T 7df30eb244914dcabc6965324eced236 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:13.732308 998 ts_tablet_manager.cc:902] T 7f2743ab5fa24c9899b67ea52f48f9d8 P d09a11a03989421b9f28019dc1c14cc2: Initiating tablet copy from peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:13:13.739418 998 tablet_copy_client.cc:308] T 7f2743ab5fa24c9899b67ea52f48f9d8 P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.69:34725
I20240503 08:13:13.740326 30783 tablet_copy_service.cc:140] P 6d1d7de19459435fbde1999e3dfd226b: Received BeginTabletCopySession request for tablet 7f2743ab5fa24c9899b67ea52f48f9d8 from peer d09a11a03989421b9f28019dc1c14cc2 ({username='slave'} at 127.20.230.68:50161)
I20240503 08:13:13.741076 30783 tablet_copy_service.cc:161] P 6d1d7de19459435fbde1999e3dfd226b: Beginning new tablet copy session on tablet 7f2743ab5fa24c9899b67ea52f48f9d8 from peer d09a11a03989421b9f28019dc1c14cc2 at {username='slave'} at 127.20.230.68:50161: session id = d09a11a03989421b9f28019dc1c14cc2-7f2743ab5fa24c9899b67ea52f48f9d8
I20240503 08:13:13.744455 30783 tablet_copy_source_session.cc:205] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:13.747476 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f2743ab5fa24c9899b67ea52f48f9d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:13.752816 998 tablet_copy_client.cc:783] T 7f2743ab5fa24c9899b67ea52f48f9d8 P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:13.753049 998 tablet_copy_client.cc:647] T 7f2743ab5fa24c9899b67ea52f48f9d8 P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:13.756709 998 tablet_copy_client.cc:523] T 7f2743ab5fa24c9899b67ea52f48f9d8 P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:13.759174 1008 ts_tablet_manager.cc:902] T c9055668b86e4dc5aa321a4d29beaf74 P 833582ff68c94269be364ebd187a75db: Initiating tablet copy from peer ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:13:13.759987 28262 consensus_queue.cc:237] T de5dfccb0359462dbb7da776293fa679 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:13.761373 998 tablet_bootstrap.cc:492] T 7f2743ab5fa24c9899b67ea52f48f9d8 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
W20240503 08:13:13.763232 28197 consensus_peers.cc:485] T de5dfccb0359462dbb7da776293fa679 P aae860d7ae2b4002bf7794c278f9cc56 -> Peer ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): Couldn't send request to peer ced21ca983b749d0b4d4f6a0e2426bf8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: de5dfccb0359462dbb7da776293fa679. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:13.763442 28133 raft_consensus.cc:1261] T de5dfccb0359462dbb7da776293fa679 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:13.763489 28002 raft_consensus.cc:1261] T de5dfccb0359462dbb7da776293fa679 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:13.764158 589 consensus_queue.cc:1035] T de5dfccb0359462dbb7da776293fa679 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:13.764681 589 consensus_queue.cc:1035] T de5dfccb0359462dbb7da776293fa679 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:13.772545 865 raft_consensus.cc:2932] T de5dfccb0359462dbb7da776293fa679 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:13:13.773677 998 tablet_bootstrap.cc:492] T 7f2743ab5fa24c9899b67ea52f48f9d8 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:13.774039 998 tablet_bootstrap.cc:492] T 7f2743ab5fa24c9899b67ea52f48f9d8 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap complete.
I20240503 08:13:13.773382 28133 raft_consensus.cc:2932] T de5dfccb0359462dbb7da776293fa679 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:13:13.774262 998 ts_tablet_manager.cc:1367] T 7f2743ab5fa24c9899b67ea52f48f9d8 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20240503 08:13:13.774891 998 raft_consensus.cc:348] T 7f2743ab5fa24c9899b67ea52f48f9d8 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:13.775099 998 raft_consensus.cc:729] T 7f2743ab5fa24c9899b67ea52f48f9d8 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:13:13.775323 998 consensus_queue.cc:260] T 7f2743ab5fa24c9899b67ea52f48f9d8 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:13.777138 998 ts_tablet_manager.cc:1398] T 7f2743ab5fa24c9899b67ea52f48f9d8 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:13:13.777909 30783 tablet_copy_service.cc:342] P 6d1d7de19459435fbde1999e3dfd226b: Request end of tablet copy session d09a11a03989421b9f28019dc1c14cc2-7f2743ab5fa24c9899b67ea52f48f9d8 received from {username='slave'} at 127.20.230.68:50161
I20240503 08:13:13.778064 30783 tablet_copy_service.cc:433] P 6d1d7de19459435fbde1999e3dfd226b: ending tablet copy session d09a11a03989421b9f28019dc1c14cc2-7f2743ab5fa24c9899b67ea52f48f9d8 on tablet 7f2743ab5fa24c9899b67ea52f48f9d8 with peer d09a11a03989421b9f28019dc1c14cc2
I20240503 08:13:13.781973 28002 raft_consensus.cc:2932] T de5dfccb0359462dbb7da776293fa679 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:13:13.783233 27862 catalog_manager.cc:5484] T de5dfccb0359462dbb7da776293fa679 P 1b411c6bb25549368041773d7b710d48 reported cstate change: config changed from index -1 to 2, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) added. New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:13:13.788648 1008 tablet_copy_client.cc:308] T c9055668b86e4dc5aa321a4d29beaf74 P 833582ff68c94269be364ebd187a75db: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.70:33747
I20240503 08:13:13.790428 30915 tablet_copy_service.cc:140] P ce1496a47d084365ad786cd09e472a2c: Received BeginTabletCopySession request for tablet c9055668b86e4dc5aa321a4d29beaf74 from peer 833582ff68c94269be364ebd187a75db ({username='slave'} at 127.20.230.65:60309)
I20240503 08:13:13.790665 30915 tablet_copy_service.cc:161] P ce1496a47d084365ad786cd09e472a2c: Beginning new tablet copy session on tablet c9055668b86e4dc5aa321a4d29beaf74 from peer 833582ff68c94269be364ebd187a75db at {username='slave'} at 127.20.230.65:60309: session id = 833582ff68c94269be364ebd187a75db-c9055668b86e4dc5aa321a4d29beaf74
I20240503 08:13:13.794580 30915 tablet_copy_source_session.cc:205] T c9055668b86e4dc5aa321a4d29beaf74 P ce1496a47d084365ad786cd09e472a2c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:13.796821 1008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9055668b86e4dc5aa321a4d29beaf74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:13.803066 1008 tablet_copy_client.cc:783] T c9055668b86e4dc5aa321a4d29beaf74 P 833582ff68c94269be364ebd187a75db: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:13.803318 1008 tablet_copy_client.cc:647] T c9055668b86e4dc5aa321a4d29beaf74 P 833582ff68c94269be364ebd187a75db: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:13.806181 1008 tablet_copy_client.cc:523] T c9055668b86e4dc5aa321a4d29beaf74 P 833582ff68c94269be364ebd187a75db: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:13.809953 1008 tablet_bootstrap.cc:492] T c9055668b86e4dc5aa321a4d29beaf74 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:13:13.811957 1028 ts_tablet_manager.cc:902] T 3042212758bf4aef8be5c5d12d10113c P ce1496a47d084365ad786cd09e472a2c: Initiating tablet copy from peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:13:13.815794 1008 tablet_bootstrap.cc:492] T c9055668b86e4dc5aa321a4d29beaf74 P 833582ff68c94269be364ebd187a75db: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:13.816355 1008 tablet_bootstrap.cc:492] T c9055668b86e4dc5aa321a4d29beaf74 P 833582ff68c94269be364ebd187a75db: Bootstrap complete.
I20240503 08:13:13.816551 1008 ts_tablet_manager.cc:1367] T c9055668b86e4dc5aa321a4d29beaf74 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20240503 08:13:13.817135 1008 raft_consensus.cc:348] T c9055668b86e4dc5aa321a4d29beaf74 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:13.817330 1008 raft_consensus.cc:729] T c9055668b86e4dc5aa321a4d29beaf74 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:13:13.817579 1008 consensus_queue.cc:260] T c9055668b86e4dc5aa321a4d29beaf74 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:13.818176 1008 ts_tablet_manager.cc:1398] T c9055668b86e4dc5aa321a4d29beaf74 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:13.819137 30915 tablet_copy_service.cc:342] P ce1496a47d084365ad786cd09e472a2c: Request end of tablet copy session 833582ff68c94269be364ebd187a75db-c9055668b86e4dc5aa321a4d29beaf74 received from {username='slave'} at 127.20.230.65:60309
I20240503 08:13:13.819291 30915 tablet_copy_service.cc:433] P ce1496a47d084365ad786cd09e472a2c: ending tablet copy session 833582ff68c94269be364ebd187a75db-c9055668b86e4dc5aa321a4d29beaf74 on tablet c9055668b86e4dc5aa321a4d29beaf74 with peer 833582ff68c94269be364ebd187a75db
I20240503 08:13:13.821517 1028 tablet_copy_client.cc:308] T 3042212758bf4aef8be5c5d12d10113c P ce1496a47d084365ad786cd09e472a2c: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.65:39081
I20240503 08:13:13.822342 28023 tablet_copy_service.cc:140] P 833582ff68c94269be364ebd187a75db: Received BeginTabletCopySession request for tablet 3042212758bf4aef8be5c5d12d10113c from peer ce1496a47d084365ad786cd09e472a2c ({username='slave'} at 127.20.230.70:49991)
I20240503 08:13:13.822479 28023 tablet_copy_service.cc:161] P 833582ff68c94269be364ebd187a75db: Beginning new tablet copy session on tablet 3042212758bf4aef8be5c5d12d10113c from peer ce1496a47d084365ad786cd09e472a2c at {username='slave'} at 127.20.230.70:49991: session id = ce1496a47d084365ad786cd09e472a2c-3042212758bf4aef8be5c5d12d10113c
I20240503 08:13:13.825136 28023 tablet_copy_source_session.cc:205] T 3042212758bf4aef8be5c5d12d10113c P 833582ff68c94269be364ebd187a75db: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:13.827491 1028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3042212758bf4aef8be5c5d12d10113c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:13.833552 1028 tablet_copy_client.cc:783] T 3042212758bf4aef8be5c5d12d10113c P ce1496a47d084365ad786cd09e472a2c: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:13.833936 1028 tablet_copy_client.cc:647] T 3042212758bf4aef8be5c5d12d10113c P ce1496a47d084365ad786cd09e472a2c: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:13.836066 1028 tablet_copy_client.cc:523] T 3042212758bf4aef8be5c5d12d10113c P ce1496a47d084365ad786cd09e472a2c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:13.839423 1028 tablet_bootstrap.cc:492] T 3042212758bf4aef8be5c5d12d10113c P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:13:13.839605 28132 consensus_queue.cc:237] T 7a101bee3af34166848ee7fa7c733892 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:13.842411 28390 raft_consensus.cc:1261] T 7a101bee3af34166848ee7fa7c733892 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:13.842429 28262 raft_consensus.cc:1261] T 7a101bee3af34166848ee7fa7c733892 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:13.843043 324 consensus_queue.cc:1035] T 7a101bee3af34166848ee7fa7c733892 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:13.843451 1040 consensus_queue.cc:1035] T 7a101bee3af34166848ee7fa7c733892 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:13:13.843860 28066 consensus_peers.cc:485] T 7a101bee3af34166848ee7fa7c733892 P 1b411c6bb25549368041773d7b710d48 -> Peer ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): Couldn't send request to peer ce1496a47d084365ad786cd09e472a2c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7a101bee3af34166848ee7fa7c733892. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:13.846343 324 raft_consensus.cc:2932] T 7a101bee3af34166848ee7fa7c733892 P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:13.847020 28262 raft_consensus.cc:2932] T 7a101bee3af34166848ee7fa7c733892 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:13.849985 28390 raft_consensus.cc:2932] T 7a101bee3af34166848ee7fa7c733892 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:13.855266 27861 catalog_manager.cc:5484] T 7a101bee3af34166848ee7fa7c733892 P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: config changed from index -1 to 4, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New cstate: current_term: 1 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:13.887054 1028 tablet_bootstrap.cc:492] T 3042212758bf4aef8be5c5d12d10113c P ce1496a47d084365ad786cd09e472a2c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:13.887533 1028 tablet_bootstrap.cc:492] T 3042212758bf4aef8be5c5d12d10113c P ce1496a47d084365ad786cd09e472a2c: Bootstrap complete.
I20240503 08:13:13.887836 1028 ts_tablet_manager.cc:1367] T 3042212758bf4aef8be5c5d12d10113c P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.048s user 0.046s sys 0.001s
I20240503 08:13:13.888748 1028 raft_consensus.cc:348] T 3042212758bf4aef8be5c5d12d10113c P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:13.889043 1028 raft_consensus.cc:729] T 3042212758bf4aef8be5c5d12d10113c P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:13:13.889374 1028 consensus_queue.cc:260] T 3042212758bf4aef8be5c5d12d10113c P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:13.889966 1028 ts_tablet_manager.cc:1398] T 3042212758bf4aef8be5c5d12d10113c P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 08:13:13.890712 28023 tablet_copy_service.cc:342] P 833582ff68c94269be364ebd187a75db: Request end of tablet copy session ce1496a47d084365ad786cd09e472a2c-3042212758bf4aef8be5c5d12d10113c received from {username='slave'} at 127.20.230.70:49991
I20240503 08:13:13.890815 28023 tablet_copy_service.cc:433] P 833582ff68c94269be364ebd187a75db: ending tablet copy session ce1496a47d084365ad786cd09e472a2c-3042212758bf4aef8be5c5d12d10113c on tablet 3042212758bf4aef8be5c5d12d10113c with peer ce1496a47d084365ad786cd09e472a2c
I20240503 08:13:13.923774 1008 ts_tablet_manager.cc:902] T f9b4f775b9914bdc98b9b582072cb1ab P 833582ff68c94269be364ebd187a75db: Initiating tablet copy from peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:13:13.924322 28262 consensus_queue.cc:237] T 8cc6ae30aa60495d8a4364a11d5bef79 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:13.927410 28390 raft_consensus.cc:1261] T 8cc6ae30aa60495d8a4364a11d5bef79 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:13.927939 1008 tablet_copy_client.cc:308] T f9b4f775b9914bdc98b9b582072cb1ab P 833582ff68c94269be364ebd187a75db: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.69:34725
W20240503 08:13:13.928458 28198 consensus_peers.cc:485] T 8cc6ae30aa60495d8a4364a11d5bef79 P aae860d7ae2b4002bf7794c278f9cc56 -> Peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): Couldn't send request to peer 6d1d7de19459435fbde1999e3dfd226b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8cc6ae30aa60495d8a4364a11d5bef79. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:13.928741 30783 tablet_copy_service.cc:140] P 6d1d7de19459435fbde1999e3dfd226b: Received BeginTabletCopySession request for tablet f9b4f775b9914bdc98b9b582072cb1ab from peer 833582ff68c94269be364ebd187a75db ({username='slave'} at 127.20.230.65:42993)
I20240503 08:13:13.928897 30783 tablet_copy_service.cc:161] P 6d1d7de19459435fbde1999e3dfd226b: Beginning new tablet copy session on tablet f9b4f775b9914bdc98b9b582072cb1ab from peer 833582ff68c94269be364ebd187a75db at {username='slave'} at 127.20.230.65:42993: session id = 833582ff68c94269be364ebd187a75db-f9b4f775b9914bdc98b9b582072cb1ab
I20240503 08:13:13.930899 30783 tablet_copy_source_session.cc:205] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:13.927932 28002 raft_consensus.cc:1261] T 8cc6ae30aa60495d8a4364a11d5bef79 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:13.932003 1008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9b4f775b9914bdc98b9b582072cb1ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:13.932261 792 consensus_queue.cc:1035] T 8cc6ae30aa60495d8a4364a11d5bef79 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:13.933905 792 consensus_queue.cc:1035] T 8cc6ae30aa60495d8a4364a11d5bef79 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:13.940529 1008 tablet_copy_client.cc:783] T f9b4f775b9914bdc98b9b582072cb1ab P 833582ff68c94269be364ebd187a75db: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:13.940918 1008 tablet_copy_client.cc:647] T f9b4f775b9914bdc98b9b582072cb1ab P 833582ff68c94269be364ebd187a75db: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:13.944993 1008 tablet_copy_client.cc:523] T f9b4f775b9914bdc98b9b582072cb1ab P 833582ff68c94269be364ebd187a75db: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:13.947988 1008 tablet_bootstrap.cc:492] T f9b4f775b9914bdc98b9b582072cb1ab P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:13:13.948062 865 raft_consensus.cc:2932] T 8cc6ae30aa60495d8a4364a11d5bef79 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:13.949115 28391 raft_consensus.cc:2932] T 8cc6ae30aa60495d8a4364a11d5bef79 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:13.957127 27861 catalog_manager.cc:5484] T 8cc6ae30aa60495d8a4364a11d5bef79 P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: config changed from index -1 to 4, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) added. New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:13.960026 966 ts_tablet_manager.cc:902] T 2222ff7fb5184d6c86010e1475d6c1f7 P 1b411c6bb25549368041773d7b710d48: Initiating tablet copy from peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:13:13.966393 28002 raft_consensus.cc:2932] T 8cc6ae30aa60495d8a4364a11d5bef79 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:13.979775 1008 tablet_bootstrap.cc:492] T f9b4f775b9914bdc98b9b582072cb1ab P 833582ff68c94269be364ebd187a75db: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:13.980118 1008 tablet_bootstrap.cc:492] T f9b4f775b9914bdc98b9b582072cb1ab P 833582ff68c94269be364ebd187a75db: Bootstrap complete.
I20240503 08:13:13.980329 1008 ts_tablet_manager.cc:1367] T f9b4f775b9914bdc98b9b582072cb1ab P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.017s
I20240503 08:13:13.980979 1008 raft_consensus.cc:348] T f9b4f775b9914bdc98b9b582072cb1ab P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:13.981166 1008 raft_consensus.cc:729] T f9b4f775b9914bdc98b9b582072cb1ab P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:13:13.981356 1008 consensus_queue.cc:260] T f9b4f775b9914bdc98b9b582072cb1ab P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:13.984674 1008 ts_tablet_manager.cc:1398] T f9b4f775b9914bdc98b9b582072cb1ab P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:13:13.985311 30783 tablet_copy_service.cc:342] P 6d1d7de19459435fbde1999e3dfd226b: Request end of tablet copy session 833582ff68c94269be364ebd187a75db-f9b4f775b9914bdc98b9b582072cb1ab received from {username='slave'} at 127.20.230.65:42993
I20240503 08:13:13.985476 30783 tablet_copy_service.cc:433] P 6d1d7de19459435fbde1999e3dfd226b: ending tablet copy session 833582ff68c94269be364ebd187a75db-f9b4f775b9914bdc98b9b582072cb1ab on tablet f9b4f775b9914bdc98b9b582072cb1ab with peer 833582ff68c94269be364ebd187a75db
I20240503 08:13:14.027614 966 tablet_copy_client.cc:308] T 2222ff7fb5184d6c86010e1475d6c1f7 P 1b411c6bb25549368041773d7b710d48: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.67:39919
I20240503 08:13:14.028524 28283 tablet_copy_service.cc:140] P aae860d7ae2b4002bf7794c278f9cc56: Received BeginTabletCopySession request for tablet 2222ff7fb5184d6c86010e1475d6c1f7 from peer 1b411c6bb25549368041773d7b710d48 ({username='slave'} at 127.20.230.66:43521)
I20240503 08:13:14.028746 28283 tablet_copy_service.cc:161] P aae860d7ae2b4002bf7794c278f9cc56: Beginning new tablet copy session on tablet 2222ff7fb5184d6c86010e1475d6c1f7 from peer 1b411c6bb25549368041773d7b710d48 at {username='slave'} at 127.20.230.66:43521: session id = 1b411c6bb25549368041773d7b710d48-2222ff7fb5184d6c86010e1475d6c1f7
I20240503 08:13:14.032413 28283 tablet_copy_source_session.cc:205] T 2222ff7fb5184d6c86010e1475d6c1f7 P aae860d7ae2b4002bf7794c278f9cc56: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:14.035849 966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2222ff7fb5184d6c86010e1475d6c1f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:14.041581 966 tablet_copy_client.cc:783] T 2222ff7fb5184d6c86010e1475d6c1f7 P 1b411c6bb25549368041773d7b710d48: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:14.041960 966 tablet_copy_client.cc:647] T 2222ff7fb5184d6c86010e1475d6c1f7 P 1b411c6bb25549368041773d7b710d48: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:14.043892 966 tablet_copy_client.cc:523] T 2222ff7fb5184d6c86010e1475d6c1f7 P 1b411c6bb25549368041773d7b710d48: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:14.047945 966 tablet_bootstrap.cc:492] T 2222ff7fb5184d6c86010e1475d6c1f7 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:13:14.054878 966 tablet_bootstrap.cc:492] T 2222ff7fb5184d6c86010e1475d6c1f7 P 1b411c6bb25549368041773d7b710d48: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:14.055207 966 tablet_bootstrap.cc:492] T 2222ff7fb5184d6c86010e1475d6c1f7 P 1b411c6bb25549368041773d7b710d48: Bootstrap complete.
I20240503 08:13:14.055420 966 ts_tablet_manager.cc:1367] T 2222ff7fb5184d6c86010e1475d6c1f7 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.001s
I20240503 08:13:14.056253 966 raft_consensus.cc:348] T 2222ff7fb5184d6c86010e1475d6c1f7 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:14.056473 966 raft_consensus.cc:729] T 2222ff7fb5184d6c86010e1475d6c1f7 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:13:14.056787 966 consensus_queue.cc:260] T 2222ff7fb5184d6c86010e1475d6c1f7 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:14.059123 966 ts_tablet_manager.cc:1398] T 2222ff7fb5184d6c86010e1475d6c1f7 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:13:14.060127 28283 tablet_copy_service.cc:342] P aae860d7ae2b4002bf7794c278f9cc56: Request end of tablet copy session 1b411c6bb25549368041773d7b710d48-2222ff7fb5184d6c86010e1475d6c1f7 received from {username='slave'} at 127.20.230.66:43521
I20240503 08:13:14.060269 28283 tablet_copy_service.cc:433] P aae860d7ae2b4002bf7794c278f9cc56: ending tablet copy session 1b411c6bb25549368041773d7b710d48-2222ff7fb5184d6c86010e1475d6c1f7 on tablet 2222ff7fb5184d6c86010e1475d6c1f7 with peer 1b411c6bb25549368041773d7b710d48
I20240503 08:13:14.091750 28133 raft_consensus.cc:1203] T 417cedc3fa5c4a85a7a130005291d64b P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:14.094622 1028 ts_tablet_manager.cc:902] T d3da6397e0c34c8f96933963dd4fa65e P ce1496a47d084365ad786cd09e472a2c: Initiating tablet copy from peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:13:14.105412 966 ts_tablet_manager.cc:902] T 7df30eb244914dcabc6965324eced236 P 1b411c6bb25549368041773d7b710d48: Initiating tablet copy from peer ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:13:14.151185 966 tablet_copy_client.cc:308] T 7df30eb244914dcabc6965324eced236 P 1b411c6bb25549368041773d7b710d48: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.71:41915
I20240503 08:13:14.152240 31048 tablet_copy_service.cc:140] P ced21ca983b749d0b4d4f6a0e2426bf8: Received BeginTabletCopySession request for tablet 7df30eb244914dcabc6965324eced236 from peer 1b411c6bb25549368041773d7b710d48 ({username='slave'} at 127.20.230.66:37577)
I20240503 08:13:14.152465 31048 tablet_copy_service.cc:161] P ced21ca983b749d0b4d4f6a0e2426bf8: Beginning new tablet copy session on tablet 7df30eb244914dcabc6965324eced236 from peer 1b411c6bb25549368041773d7b710d48 at {username='slave'} at 127.20.230.66:37577: session id = 1b411c6bb25549368041773d7b710d48-7df30eb244914dcabc6965324eced236
I20240503 08:13:14.156333 31048 tablet_copy_source_session.cc:205] T 7df30eb244914dcabc6965324eced236 P ced21ca983b749d0b4d4f6a0e2426bf8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:14.159453 966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7df30eb244914dcabc6965324eced236. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:14.161829 1028 tablet_copy_client.cc:308] T d3da6397e0c34c8f96933963dd4fa65e P ce1496a47d084365ad786cd09e472a2c: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.65:39081
I20240503 08:13:14.162708 28023 tablet_copy_service.cc:140] P 833582ff68c94269be364ebd187a75db: Received BeginTabletCopySession request for tablet d3da6397e0c34c8f96933963dd4fa65e from peer ce1496a47d084365ad786cd09e472a2c ({username='slave'} at 127.20.230.70:49991)
I20240503 08:13:14.162840 28023 tablet_copy_service.cc:161] P 833582ff68c94269be364ebd187a75db: Beginning new tablet copy session on tablet d3da6397e0c34c8f96933963dd4fa65e from peer ce1496a47d084365ad786cd09e472a2c at {username='slave'} at 127.20.230.70:49991: session id = ce1496a47d084365ad786cd09e472a2c-d3da6397e0c34c8f96933963dd4fa65e
I20240503 08:13:14.164813 28023 tablet_copy_source_session.cc:205] T d3da6397e0c34c8f96933963dd4fa65e P 833582ff68c94269be364ebd187a75db: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:14.178445 1028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3da6397e0c34c8f96933963dd4fa65e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:14.182816 28262 consensus_queue.cc:237] T 1cac39d5831d4b6887250f7a9d63dcd5 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:14.184959 966 tablet_copy_client.cc:783] T 7df30eb244914dcabc6965324eced236 P 1b411c6bb25549368041773d7b710d48: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:14.185168 966 tablet_copy_client.cc:647] T 7df30eb244914dcabc6965324eced236 P 1b411c6bb25549368041773d7b710d48: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:14.195360 30760 raft_consensus.cc:1261] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:14.196007 966 tablet_copy_client.cc:523] T 7df30eb244914dcabc6965324eced236 P 1b411c6bb25549368041773d7b710d48: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240503 08:13:14.196080 28198 consensus_peers.cc:485] T 1cac39d5831d4b6887250f7a9d63dcd5 P aae860d7ae2b4002bf7794c278f9cc56 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1cac39d5831d4b6887250f7a9d63dcd5. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:14.196234 589 consensus_queue.cc:1035] T 1cac39d5831d4b6887250f7a9d63dcd5 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:14.196949 31159 raft_consensus.cc:1261] T 1cac39d5831d4b6887250f7a9d63dcd5 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:14.198086 589 consensus_queue.cc:1035] T 1cac39d5831d4b6887250f7a9d63dcd5 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:14.199591 966 tablet_bootstrap.cc:492] T 7df30eb244914dcabc6965324eced236 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:13:14.201514 998 ts_tablet_manager.cc:902] T 013e00c67a1c4ae9b3c1a050b966e9a5 P d09a11a03989421b9f28019dc1c14cc2: Initiating tablet copy from peer b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:13:14.204510 1028 tablet_copy_client.cc:783] T d3da6397e0c34c8f96933963dd4fa65e P ce1496a47d084365ad786cd09e472a2c: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:14.204728 1028 tablet_copy_client.cc:647] T d3da6397e0c34c8f96933963dd4fa65e P ce1496a47d084365ad786cd09e472a2c: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:14.206447 1028 tablet_copy_client.cc:523] T d3da6397e0c34c8f96933963dd4fa65e P ce1496a47d084365ad786cd09e472a2c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:14.216266 1028 tablet_bootstrap.cc:492] T d3da6397e0c34c8f96933963dd4fa65e P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:13:14.223839 966 tablet_bootstrap.cc:492] T 7df30eb244914dcabc6965324eced236 P 1b411c6bb25549368041773d7b710d48: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:14.224167 966 tablet_bootstrap.cc:492] T 7df30eb244914dcabc6965324eced236 P 1b411c6bb25549368041773d7b710d48: Bootstrap complete.
I20240503 08:13:14.224372 966 ts_tablet_manager.cc:1367] T 7df30eb244914dcabc6965324eced236 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.001s
I20240503 08:13:14.224972 966 raft_consensus.cc:348] T 7df30eb244914dcabc6965324eced236 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:14.225160 966 raft_consensus.cc:729] T 7df30eb244914dcabc6965324eced236 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:13:14.225373 966 consensus_queue.cc:260] T 7df30eb244914dcabc6965324eced236 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:14.229904 966 ts_tablet_manager.cc:1398] T 7df30eb244914dcabc6965324eced236 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240503 08:13:14.235662 31048 tablet_copy_service.cc:342] P ced21ca983b749d0b4d4f6a0e2426bf8: Request end of tablet copy session 1b411c6bb25549368041773d7b710d48-7df30eb244914dcabc6965324eced236 received from {username='slave'} at 127.20.230.66:37577
I20240503 08:13:14.235841 31048 tablet_copy_service.cc:433] P ced21ca983b749d0b4d4f6a0e2426bf8: ending tablet copy session 1b411c6bb25549368041773d7b710d48-7df30eb244914dcabc6965324eced236 on tablet 7df30eb244914dcabc6965324eced236 with peer 1b411c6bb25549368041773d7b710d48
I20240503 08:13:14.241832 589 raft_consensus.cc:2932] T 1cac39d5831d4b6887250f7a9d63dcd5 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:14.243410 30761 raft_consensus.cc:2932] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:14.248917 31159 raft_consensus.cc:2932] T 1cac39d5831d4b6887250f7a9d63dcd5 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:14.258792 27861 catalog_manager.cc:5484] T 1cac39d5831d4b6887250f7a9d63dcd5 P b00f244c69df401997ea99d492218a98 reported cstate change: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:14.267684 28002 raft_consensus.cc:1203] T c9055668b86e4dc5aa321a4d29beaf74 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:14.290030 998 tablet_copy_client.cc:308] T 013e00c67a1c4ae9b3c1a050b966e9a5 P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.72:40893
I20240503 08:13:14.290791 31181 tablet_copy_service.cc:140] P b00f244c69df401997ea99d492218a98: Received BeginTabletCopySession request for tablet 013e00c67a1c4ae9b3c1a050b966e9a5 from peer d09a11a03989421b9f28019dc1c14cc2 ({username='slave'} at 127.20.230.68:37433)
I20240503 08:13:14.290995 31181 tablet_copy_service.cc:161] P b00f244c69df401997ea99d492218a98: Beginning new tablet copy session on tablet 013e00c67a1c4ae9b3c1a050b966e9a5 from peer d09a11a03989421b9f28019dc1c14cc2 at {username='slave'} at 127.20.230.68:37433: session id = d09a11a03989421b9f28019dc1c14cc2-013e00c67a1c4ae9b3c1a050b966e9a5
I20240503 08:13:14.294013 31181 tablet_copy_source_session.cc:205] T 013e00c67a1c4ae9b3c1a050b966e9a5 P b00f244c69df401997ea99d492218a98: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:14.295370 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 013e00c67a1c4ae9b3c1a050b966e9a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:14.299814 998 tablet_copy_client.cc:783] T 013e00c67a1c4ae9b3c1a050b966e9a5 P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:14.300011 998 tablet_copy_client.cc:647] T 013e00c67a1c4ae9b3c1a050b966e9a5 P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:14.302054 998 tablet_copy_client.cc:523] T 013e00c67a1c4ae9b3c1a050b966e9a5 P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:14.305074 998 tablet_bootstrap.cc:492] T 013e00c67a1c4ae9b3c1a050b966e9a5 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:13:14.305945 28002 raft_consensus.cc:1203] T f9b4f775b9914bdc98b9b582072cb1ab P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:14.312213 998 tablet_bootstrap.cc:492] T 013e00c67a1c4ae9b3c1a050b966e9a5 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:14.312631 998 tablet_bootstrap.cc:492] T 013e00c67a1c4ae9b3c1a050b966e9a5 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap complete.
I20240503 08:13:14.312860 998 ts_tablet_manager.cc:1367] T 013e00c67a1c4ae9b3c1a050b966e9a5 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.001s
I20240503 08:13:14.313525 998 raft_consensus.cc:348] T 013e00c67a1c4ae9b3c1a050b966e9a5 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:14.313838 998 raft_consensus.cc:729] T 013e00c67a1c4ae9b3c1a050b966e9a5 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:13:14.314044 998 consensus_queue.cc:260] T 013e00c67a1c4ae9b3c1a050b966e9a5 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:14.318138 998 ts_tablet_manager.cc:1398] T 013e00c67a1c4ae9b3c1a050b966e9a5 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20240503 08:13:14.319888 31181 tablet_copy_service.cc:342] P b00f244c69df401997ea99d492218a98: Request end of tablet copy session d09a11a03989421b9f28019dc1c14cc2-013e00c67a1c4ae9b3c1a050b966e9a5 received from {username='slave'} at 127.20.230.68:37433
I20240503 08:13:14.320134 31181 tablet_copy_service.cc:433] P b00f244c69df401997ea99d492218a98: ending tablet copy session d09a11a03989421b9f28019dc1c14cc2-013e00c67a1c4ae9b3c1a050b966e9a5 on tablet 013e00c67a1c4ae9b3c1a050b966e9a5 with peer d09a11a03989421b9f28019dc1c14cc2
I20240503 08:13:14.321449 1028 tablet_bootstrap.cc:492] T d3da6397e0c34c8f96933963dd4fa65e P ce1496a47d084365ad786cd09e472a2c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:14.326507 1028 tablet_bootstrap.cc:492] T d3da6397e0c34c8f96933963dd4fa65e P ce1496a47d084365ad786cd09e472a2c: Bootstrap complete.
I20240503 08:13:14.326882 1028 ts_tablet_manager.cc:1367] T d3da6397e0c34c8f96933963dd4fa65e P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.112s user 0.043s sys 0.006s
I20240503 08:13:14.327795 1028 raft_consensus.cc:348] T d3da6397e0c34c8f96933963dd4fa65e P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:14.328063 1028 raft_consensus.cc:729] T d3da6397e0c34c8f96933963dd4fa65e P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:13:14.328572 1028 consensus_queue.cc:260] T d3da6397e0c34c8f96933963dd4fa65e P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:14.329213 1028 ts_tablet_manager.cc:1398] T d3da6397e0c34c8f96933963dd4fa65e P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:14.329892 1113 ts_tablet_manager.cc:902] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8: Initiating tablet copy from peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:13:14.331215 28023 tablet_copy_service.cc:342] P 833582ff68c94269be364ebd187a75db: Request end of tablet copy session ce1496a47d084365ad786cd09e472a2c-d3da6397e0c34c8f96933963dd4fa65e received from {username='slave'} at 127.20.230.70:49991
I20240503 08:13:14.331352 28023 tablet_copy_service.cc:433] P 833582ff68c94269be364ebd187a75db: ending tablet copy session ce1496a47d084365ad786cd09e472a2c-d3da6397e0c34c8f96933963dd4fa65e on tablet d3da6397e0c34c8f96933963dd4fa65e with peer ce1496a47d084365ad786cd09e472a2c
I20240503 08:13:14.338333 28391 raft_consensus.cc:1203] T 7f2743ab5fa24c9899b67ea52f48f9d8 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:14.347786 1113 tablet_copy_client.cc:308] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.67:39919
I20240503 08:13:14.348523 28283 tablet_copy_service.cc:140] P aae860d7ae2b4002bf7794c278f9cc56: Received BeginTabletCopySession request for tablet de5dfccb0359462dbb7da776293fa679 from peer ced21ca983b749d0b4d4f6a0e2426bf8 ({username='slave'} at 127.20.230.71:55281)
I20240503 08:13:14.348702 28283 tablet_copy_service.cc:161] P aae860d7ae2b4002bf7794c278f9cc56: Beginning new tablet copy session on tablet de5dfccb0359462dbb7da776293fa679 from peer ced21ca983b749d0b4d4f6a0e2426bf8 at {username='slave'} at 127.20.230.71:55281: session id = ced21ca983b749d0b4d4f6a0e2426bf8-de5dfccb0359462dbb7da776293fa679
I20240503 08:13:14.351333 28283 tablet_copy_source_session.cc:205] T de5dfccb0359462dbb7da776293fa679 P aae860d7ae2b4002bf7794c278f9cc56: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:14.352972 1113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de5dfccb0359462dbb7da776293fa679. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:14.358842 1028 ts_tablet_manager.cc:902] T 7a101bee3af34166848ee7fa7c733892 P ce1496a47d084365ad786cd09e472a2c: Initiating tablet copy from peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:13:14.361172 1028 tablet_copy_client.cc:308] T 7a101bee3af34166848ee7fa7c733892 P ce1496a47d084365ad786cd09e472a2c: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.66:37143
I20240503 08:13:14.361868 28153 tablet_copy_service.cc:140] P 1b411c6bb25549368041773d7b710d48: Received BeginTabletCopySession request for tablet 7a101bee3af34166848ee7fa7c733892 from peer ce1496a47d084365ad786cd09e472a2c ({username='slave'} at 127.20.230.70:50767)
I20240503 08:13:14.362035 28153 tablet_copy_service.cc:161] P 1b411c6bb25549368041773d7b710d48: Beginning new tablet copy session on tablet 7a101bee3af34166848ee7fa7c733892 from peer ce1496a47d084365ad786cd09e472a2c at {username='slave'} at 127.20.230.70:50767: session id = ce1496a47d084365ad786cd09e472a2c-7a101bee3af34166848ee7fa7c733892
I20240503 08:13:14.365021 1113 tablet_copy_client.cc:783] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:14.365411 1113 tablet_copy_client.cc:647] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:14.365836 28153 tablet_copy_source_session.cc:205] T 7a101bee3af34166848ee7fa7c733892 P 1b411c6bb25549368041773d7b710d48: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:14.367386 1028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a101bee3af34166848ee7fa7c733892. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:14.369208 1113 tablet_copy_client.cc:523] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:14.373220 1113 tablet_bootstrap.cc:492] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:13:14.375125 1028 tablet_copy_client.cc:783] T 7a101bee3af34166848ee7fa7c733892 P ce1496a47d084365ad786cd09e472a2c: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:14.375402 1028 tablet_copy_client.cc:647] T 7a101bee3af34166848ee7fa7c733892 P ce1496a47d084365ad786cd09e472a2c: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:14.379071 1028 tablet_copy_client.cc:523] T 7a101bee3af34166848ee7fa7c733892 P ce1496a47d084365ad786cd09e472a2c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:14.382967 1028 tablet_bootstrap.cc:492] T 7a101bee3af34166848ee7fa7c733892 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:13:14.387377 1113 tablet_bootstrap.cc:492] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:14.387974 1113 tablet_bootstrap.cc:492] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap complete.
I20240503 08:13:14.388238 1113 ts_tablet_manager.cc:1367] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.013s
I20240503 08:13:14.390285 1113 raft_consensus.cc:348] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:14.390605 1113 raft_consensus.cc:729] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:13:14.390942 1113 consensus_queue.cc:260] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:14.392472 1113 ts_tablet_manager.cc:1398] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 08:13:14.393591 1134 ts_tablet_manager.cc:902] T 8cc6ae30aa60495d8a4364a11d5bef79 P 6d1d7de19459435fbde1999e3dfd226b: Initiating tablet copy from peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:13:14.395300 30895 raft_consensus.cc:1203] T 3042212758bf4aef8be5c5d12d10113c P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:14.396224 28283 tablet_copy_service.cc:342] P aae860d7ae2b4002bf7794c278f9cc56: Request end of tablet copy session ced21ca983b749d0b4d4f6a0e2426bf8-de5dfccb0359462dbb7da776293fa679 received from {username='slave'} at 127.20.230.71:55281
I20240503 08:13:14.396348 28283 tablet_copy_service.cc:433] P aae860d7ae2b4002bf7794c278f9cc56: ending tablet copy session ced21ca983b749d0b4d4f6a0e2426bf8-de5dfccb0359462dbb7da776293fa679 on tablet de5dfccb0359462dbb7da776293fa679 with peer ced21ca983b749d0b4d4f6a0e2426bf8
I20240503 08:13:14.402379 28262 consensus_queue.cc:237] T 8450e62352a74d2ea1471166d4cb1132 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:14.404913 28132 raft_consensus.cc:1261] T 8450e62352a74d2ea1471166d4cb1132 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:14.405151 28002 raft_consensus.cc:1261] T 8450e62352a74d2ea1471166d4cb1132 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:13:14.405670 28197 consensus_peers.cc:485] T 8450e62352a74d2ea1471166d4cb1132 P aae860d7ae2b4002bf7794c278f9cc56 -> Peer ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): Couldn't send request to peer ced21ca983b749d0b4d4f6a0e2426bf8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8450e62352a74d2ea1471166d4cb1132. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:14.405750 589 consensus_queue.cc:1035] T 8450e62352a74d2ea1471166d4cb1132 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:14.406993 1103 consensus_queue.cc:1035] T 8450e62352a74d2ea1471166d4cb1132 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:14.411821 1103 raft_consensus.cc:2932] T 8450e62352a74d2ea1471166d4cb1132 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:13:14.412600 28002 raft_consensus.cc:2932] T 8450e62352a74d2ea1471166d4cb1132 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:13:14.418603 1134 tablet_copy_client.cc:308] T 8cc6ae30aa60495d8a4364a11d5bef79 P 6d1d7de19459435fbde1999e3dfd226b: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.67:39919
I20240503 08:13:14.418975 28132 raft_consensus.cc:1203] T 2222ff7fb5184d6c86010e1475d6c1f7 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:14.419749 28283 tablet_copy_service.cc:140] P aae860d7ae2b4002bf7794c278f9cc56: Received BeginTabletCopySession request for tablet 8cc6ae30aa60495d8a4364a11d5bef79 from peer 6d1d7de19459435fbde1999e3dfd226b ({username='slave'} at 127.20.230.69:54309)
I20240503 08:13:14.419904 28283 tablet_copy_service.cc:161] P aae860d7ae2b4002bf7794c278f9cc56: Beginning new tablet copy session on tablet 8cc6ae30aa60495d8a4364a11d5bef79 from peer 6d1d7de19459435fbde1999e3dfd226b at {username='slave'} at 127.20.230.69:54309: session id = 6d1d7de19459435fbde1999e3dfd226b-8cc6ae30aa60495d8a4364a11d5bef79
I20240503 08:13:14.419988 27858 catalog_manager.cc:5484] T 8450e62352a74d2ea1471166d4cb1132 P 833582ff68c94269be364ebd187a75db reported cstate change: config changed from index -1 to 4, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) added. New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:13:14.422925 28283 tablet_copy_source_session.cc:205] T 8cc6ae30aa60495d8a4364a11d5bef79 P aae860d7ae2b4002bf7794c278f9cc56: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:14.425173 1134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cc6ae30aa60495d8a4364a11d5bef79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:14.425743 28132 raft_consensus.cc:2932] T 8450e62352a74d2ea1471166d4cb1132 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:13:14.437041 1134 tablet_copy_client.cc:783] T 8cc6ae30aa60495d8a4364a11d5bef79 P 6d1d7de19459435fbde1999e3dfd226b: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:14.437458 1134 tablet_copy_client.cc:647] T 8cc6ae30aa60495d8a4364a11d5bef79 P 6d1d7de19459435fbde1999e3dfd226b: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:14.444855 1134 tablet_copy_client.cc:523] T 8cc6ae30aa60495d8a4364a11d5bef79 P 6d1d7de19459435fbde1999e3dfd226b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:14.449604 1134 tablet_bootstrap.cc:492] T 8cc6ae30aa60495d8a4364a11d5bef79 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:13:14.451640 1028 tablet_bootstrap.cc:492] T 7a101bee3af34166848ee7fa7c733892 P ce1496a47d084365ad786cd09e472a2c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:14.452143 1028 tablet_bootstrap.cc:492] T 7a101bee3af34166848ee7fa7c733892 P ce1496a47d084365ad786cd09e472a2c: Bootstrap complete.
I20240503 08:13:14.452720 1028 ts_tablet_manager.cc:1367] T 7a101bee3af34166848ee7fa7c733892 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.070s user 0.037s sys 0.016s
I20240503 08:13:14.453574 1028 raft_consensus.cc:348] T 7a101bee3af34166848ee7fa7c733892 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:14.453888 1028 raft_consensus.cc:729] T 7a101bee3af34166848ee7fa7c733892 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:13:14.454178 1028 consensus_queue.cc:260] T 7a101bee3af34166848ee7fa7c733892 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:14.456377 1028 ts_tablet_manager.cc:1398] T 7a101bee3af34166848ee7fa7c733892 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:13:14.457022 28153 tablet_copy_service.cc:342] P 1b411c6bb25549368041773d7b710d48: Request end of tablet copy session ce1496a47d084365ad786cd09e472a2c-7a101bee3af34166848ee7fa7c733892 received from {username='slave'} at 127.20.230.70:50767
I20240503 08:13:14.457167 28153 tablet_copy_service.cc:433] P 1b411c6bb25549368041773d7b710d48: ending tablet copy session ce1496a47d084365ad786cd09e472a2c-7a101bee3af34166848ee7fa7c733892 on tablet 7a101bee3af34166848ee7fa7c733892 with peer ce1496a47d084365ad786cd09e472a2c
I20240503 08:13:14.507476 1134 tablet_bootstrap.cc:492] T 8cc6ae30aa60495d8a4364a11d5bef79 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:14.507939 1134 tablet_bootstrap.cc:492] T 8cc6ae30aa60495d8a4364a11d5bef79 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap complete.
I20240503 08:13:14.508222 1134 ts_tablet_manager.cc:1367] T 8cc6ae30aa60495d8a4364a11d5bef79 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.059s user 0.043s sys 0.006s
I20240503 08:13:14.509029 1134 raft_consensus.cc:348] T 8cc6ae30aa60495d8a4364a11d5bef79 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:14.509261 1134 raft_consensus.cc:729] T 8cc6ae30aa60495d8a4364a11d5bef79 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:13:14.509467 1134 consensus_queue.cc:260] T 8cc6ae30aa60495d8a4364a11d5bef79 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:14.511804 1134 ts_tablet_manager.cc:1398] T 8cc6ae30aa60495d8a4364a11d5bef79 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20240503 08:13:14.513247 28283 tablet_copy_service.cc:342] P aae860d7ae2b4002bf7794c278f9cc56: Request end of tablet copy session 6d1d7de19459435fbde1999e3dfd226b-8cc6ae30aa60495d8a4364a11d5bef79 received from {username='slave'} at 127.20.230.69:54309
I20240503 08:13:14.513384 28283 tablet_copy_service.cc:433] P aae860d7ae2b4002bf7794c278f9cc56: ending tablet copy session 6d1d7de19459435fbde1999e3dfd226b-8cc6ae30aa60495d8a4364a11d5bef79 on tablet 8cc6ae30aa60495d8a4364a11d5bef79 with peer 6d1d7de19459435fbde1999e3dfd226b
I20240503 08:13:14.542380 30895 consensus_queue.cc:237] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:14.556198 28132 raft_consensus.cc:1261] T 362590d3d0834d0b8a672ab552f8c209 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:14.556630 28391 raft_consensus.cc:1261] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:14.557240 553 consensus_queue.cc:1035] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:14.557792 1054 consensus_queue.cc:1035] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 08:13:14.559224 30828 consensus_peers.cc:485] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 362590d3d0834d0b8a672ab552f8c209. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:14.575263 553 raft_consensus.cc:2932] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:14.575984 28391 raft_consensus.cc:2932] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:14.581444 28132 raft_consensus.cc:2932] T 362590d3d0834d0b8a672ab552f8c209 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:14.584635 27861 catalog_manager.cc:5484] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New cstate: current_term: 1 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:14.606025 30895 raft_consensus.cc:1203] T d3da6397e0c34c8f96933963dd4fa65e P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:14.678490 28262 consensus_queue.cc:237] T 8fc87f7f3c724cd4a270583ddebd87aa P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:14.680860 28132 raft_consensus.cc:1261] T 8fc87f7f3c724cd4a270583ddebd87aa P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:14.681042 28391 raft_consensus.cc:1261] T 8fc87f7f3c724cd4a270583ddebd87aa P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:14.681526 589 consensus_queue.cc:1035] T 8fc87f7f3c724cd4a270583ddebd87aa P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:13:14.682715 28196 consensus_peers.cc:485] T 8fc87f7f3c724cd4a270583ddebd87aa P aae860d7ae2b4002bf7794c278f9cc56 -> Peer b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): Couldn't send request to peer b00f244c69df401997ea99d492218a98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8fc87f7f3c724cd4a270583ddebd87aa. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:14.683205 1103 consensus_queue.cc:1035] T 8fc87f7f3c724cd4a270583ddebd87aa P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:14.688643 1103 raft_consensus.cc:2932] T 8fc87f7f3c724cd4a270583ddebd87aa P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:13:14.691273 28391 raft_consensus.cc:2932] T 8fc87f7f3c724cd4a270583ddebd87aa P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:13:14.693502 28132 raft_consensus.cc:1203] T 7df30eb244914dcabc6965324eced236 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:14.695396 28132 raft_consensus.cc:2932] T 8fc87f7f3c724cd4a270583ddebd87aa P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:13:14.701297 27861 catalog_manager.cc:5484] T 8fc87f7f3c724cd4a270583ddebd87aa P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: config changed from index -1 to 4, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) added. New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:13:14.761956 31028 consensus_queue.cc:237] T 50e4d2c569344dde915eec12f6f3d747 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:14.764828 31159 raft_consensus.cc:1261] T 50e4d2c569344dde915eec12f6f3d747 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer ced21ca983b749d0b4d4f6a0e2426bf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:13:14.764945 30961 consensus_peers.cc:485] T 50e4d2c569344dde915eec12f6f3d747 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 50e4d2c569344dde915eec12f6f3d747. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:14.765028 30761 raft_consensus.cc:1261] T 50e4d2c569344dde915eec12f6f3d747 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer ced21ca983b749d0b4d4f6a0e2426bf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:14.765643 985 consensus_queue.cc:1035] T 50e4d2c569344dde915eec12f6f3d747 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:14.766039 850 consensus_queue.cc:1035] T 50e4d2c569344dde915eec12f6f3d747 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:14.771135 850 raft_consensus.cc:2932] T 50e4d2c569344dde915eec12f6f3d747 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:14.771713 30761 raft_consensus.cc:2932] T 50e4d2c569344dde915eec12f6f3d747 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:14.779943 27861 catalog_manager.cc:5484] T 50e4d2c569344dde915eec12f6f3d747 P ced21ca983b749d0b4d4f6a0e2426bf8 reported cstate change: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New cstate: current_term: 1 leader_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:14.780476 28391 raft_consensus.cc:1203] T 013e00c67a1c4ae9b3c1a050b966e9a5 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:14.775126 31159 raft_consensus.cc:2932] T 50e4d2c569344dde915eec12f6f3d747 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:14.800961 1205 ts_tablet_manager.cc:902] T 1cac39d5831d4b6887250f7a9d63dcd5 P 1b411c6bb25549368041773d7b710d48: Initiating tablet copy from peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:13:14.812914 1205 tablet_copy_client.cc:308] T 1cac39d5831d4b6887250f7a9d63dcd5 P 1b411c6bb25549368041773d7b710d48: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.67:39919
I20240503 08:13:14.813632 28283 tablet_copy_service.cc:140] P aae860d7ae2b4002bf7794c278f9cc56: Received BeginTabletCopySession request for tablet 1cac39d5831d4b6887250f7a9d63dcd5 from peer 1b411c6bb25549368041773d7b710d48 ({username='slave'} at 127.20.230.66:43521)
I20240503 08:13:14.813838 28283 tablet_copy_service.cc:161] P aae860d7ae2b4002bf7794c278f9cc56: Beginning new tablet copy session on tablet 1cac39d5831d4b6887250f7a9d63dcd5 from peer 1b411c6bb25549368041773d7b710d48 at {username='slave'} at 127.20.230.66:43521: session id = 1b411c6bb25549368041773d7b710d48-1cac39d5831d4b6887250f7a9d63dcd5
I20240503 08:13:14.816421 28283 tablet_copy_source_session.cc:205] T 1cac39d5831d4b6887250f7a9d63dcd5 P aae860d7ae2b4002bf7794c278f9cc56: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:14.818310 1205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cac39d5831d4b6887250f7a9d63dcd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:14.823130 1205 tablet_copy_client.cc:783] T 1cac39d5831d4b6887250f7a9d63dcd5 P 1b411c6bb25549368041773d7b710d48: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:14.823364 1205 tablet_copy_client.cc:647] T 1cac39d5831d4b6887250f7a9d63dcd5 P 1b411c6bb25549368041773d7b710d48: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:14.828187 1205 tablet_copy_client.cc:523] T 1cac39d5831d4b6887250f7a9d63dcd5 P 1b411c6bb25549368041773d7b710d48: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:14.831701 1205 tablet_bootstrap.cc:492] T 1cac39d5831d4b6887250f7a9d63dcd5 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:13:14.831710 28390 consensus_queue.cc:237] T d808d27e71a5486f9773ad4b239d1cfe P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:14.838354 1205 tablet_bootstrap.cc:492] T 1cac39d5831d4b6887250f7a9d63dcd5 P 1b411c6bb25549368041773d7b710d48: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:14.838697 1205 tablet_bootstrap.cc:492] T 1cac39d5831d4b6887250f7a9d63dcd5 P 1b411c6bb25549368041773d7b710d48: Bootstrap complete.
I20240503 08:13:14.838932 1205 ts_tablet_manager.cc:1367] T 1cac39d5831d4b6887250f7a9d63dcd5 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.001s
W20240503 08:13:14.839444 28329 consensus_peers.cc:485] T d808d27e71a5486f9773ad4b239d1cfe P d09a11a03989421b9f28019dc1c14cc2 -> Peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Couldn't send request to peer 833582ff68c94269be364ebd187a75db. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d808d27e71a5486f9773ad4b239d1cfe. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:14.839486 1205 raft_consensus.cc:348] T 1cac39d5831d4b6887250f7a9d63dcd5 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:14.839670 1205 raft_consensus.cc:729] T 1cac39d5831d4b6887250f7a9d63dcd5 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:13:14.839885 1205 consensus_queue.cc:260] T 1cac39d5831d4b6887250f7a9d63dcd5 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:14.840382 30895 raft_consensus.cc:1261] T d808d27e71a5486f9773ad4b239d1cfe P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:14.840912 31159 raft_consensus.cc:1261] T d808d27e71a5486f9773ad4b239d1cfe P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:14.840950 580 consensus_queue.cc:1035] T d808d27e71a5486f9773ad4b239d1cfe P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:14.841600 580 consensus_queue.cc:1035] T d808d27e71a5486f9773ad4b239d1cfe P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:14.843134 1205 ts_tablet_manager.cc:1398] T 1cac39d5831d4b6887250f7a9d63dcd5 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:13:14.843848 28283 tablet_copy_service.cc:342] P aae860d7ae2b4002bf7794c278f9cc56: Request end of tablet copy session 1b411c6bb25549368041773d7b710d48-1cac39d5831d4b6887250f7a9d63dcd5 received from {username='slave'} at 127.20.230.66:43521
I20240503 08:13:14.843973 28283 tablet_copy_service.cc:433] P aae860d7ae2b4002bf7794c278f9cc56: ending tablet copy session 1b411c6bb25549368041773d7b710d48-1cac39d5831d4b6887250f7a9d63dcd5 on tablet 1cac39d5831d4b6887250f7a9d63dcd5 with peer 1b411c6bb25549368041773d7b710d48
I20240503 08:13:14.850519 976 raft_consensus.cc:2932] T d808d27e71a5486f9773ad4b239d1cfe P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:14.852356 31160 raft_consensus.cc:2932] T d808d27e71a5486f9773ad4b239d1cfe P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:14.854593 30895 raft_consensus.cc:2932] T d808d27e71a5486f9773ad4b239d1cfe P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:14.858333 31028 raft_consensus.cc:1203] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:14.860812 27862 catalog_manager.cc:5484] T d808d27e71a5486f9773ad4b239d1cfe P ce1496a47d084365ad786cd09e472a2c reported cstate change: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:14.903512 28390 consensus_queue.cc:237] T 45e7a03f838d4beaae2080d523319d58 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:14.906248 31028 raft_consensus.cc:1261] T 45e7a03f838d4beaae2080d523319d58 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:14.906872 31160 raft_consensus.cc:1261] T 45e7a03f838d4beaae2080d523319d58 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:14.907433 580 consensus_queue.cc:1035] T 45e7a03f838d4beaae2080d523319d58 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:14.908761 1222 consensus_queue.cc:1035] T 45e7a03f838d4beaae2080d523319d58 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 08:13:14.910887 28327 consensus_peers.cc:485] T 45e7a03f838d4beaae2080d523319d58 P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 45e7a03f838d4beaae2080d523319d58. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:14.911805 30895 raft_consensus.cc:1203] T 7a101bee3af34166848ee7fa7c733892 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:14.915547 976 raft_consensus.cc:2932] T 45e7a03f838d4beaae2080d523319d58 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:14.918524 31028 raft_consensus.cc:2932] T 45e7a03f838d4beaae2080d523319d58 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:14.921986 27858 catalog_manager.cc:5484] T 45e7a03f838d4beaae2080d523319d58 P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:14.925066 31160 raft_consensus.cc:2932] T 45e7a03f838d4beaae2080d523319d58 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:14.967579 1255 ts_tablet_manager.cc:902] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8: Initiating tablet copy from peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:13:14.968955 1255 tablet_copy_client.cc:308] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.67:39919
I20240503 08:13:14.969585 28283 tablet_copy_service.cc:140] P aae860d7ae2b4002bf7794c278f9cc56: Received BeginTabletCopySession request for tablet 8450e62352a74d2ea1471166d4cb1132 from peer ced21ca983b749d0b4d4f6a0e2426bf8 ({username='slave'} at 127.20.230.71:55281)
I20240503 08:13:14.969745 28283 tablet_copy_service.cc:161] P aae860d7ae2b4002bf7794c278f9cc56: Beginning new tablet copy session on tablet 8450e62352a74d2ea1471166d4cb1132 from peer ced21ca983b749d0b4d4f6a0e2426bf8 at {username='slave'} at 127.20.230.71:55281: session id = ced21ca983b749d0b4d4f6a0e2426bf8-8450e62352a74d2ea1471166d4cb1132
I20240503 08:13:14.972450 28283 tablet_copy_source_session.cc:205] T 8450e62352a74d2ea1471166d4cb1132 P aae860d7ae2b4002bf7794c278f9cc56: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:14.974077 1255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8450e62352a74d2ea1471166d4cb1132. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:14.979210 1255 tablet_copy_client.cc:783] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:14.979559 1255 tablet_copy_client.cc:647] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:14.981995 1255 tablet_copy_client.cc:523] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:14.984514 1255 tablet_bootstrap.cc:492] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:13:14.986871 30761 raft_consensus.cc:1203] T 8cc6ae30aa60495d8a4364a11d5bef79 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:15.044670 1255 tablet_bootstrap.cc:492] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:15.045199 1255 tablet_bootstrap.cc:492] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap complete.
I20240503 08:13:15.045594 1255 ts_tablet_manager.cc:1367] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.061s user 0.049s sys 0.005s
I20240503 08:13:15.046639 1255 raft_consensus.cc:348] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:15.046905 1255 raft_consensus.cc:729] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:13:15.047144 1255 consensus_queue.cc:260] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:15.047731 1255 ts_tablet_manager.cc:1398] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:15.049573 28283 tablet_copy_service.cc:342] P aae860d7ae2b4002bf7794c278f9cc56: Request end of tablet copy session ced21ca983b749d0b4d4f6a0e2426bf8-8450e62352a74d2ea1471166d4cb1132 received from {username='slave'} at 127.20.230.71:55281
I20240503 08:13:15.049706 28283 tablet_copy_service.cc:433] P aae860d7ae2b4002bf7794c278f9cc56: ending tablet copy session ced21ca983b749d0b4d4f6a0e2426bf8-8450e62352a74d2ea1471166d4cb1132 on tablet 8450e62352a74d2ea1471166d4cb1132 with peer ced21ca983b749d0b4d4f6a0e2426bf8
I20240503 08:13:15.055744 1271 ts_tablet_manager.cc:902] T 362590d3d0834d0b8a672ab552f8c209 P aae860d7ae2b4002bf7794c278f9cc56: Initiating tablet copy from peer ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:13:15.058936 1271 tablet_copy_client.cc:308] T 362590d3d0834d0b8a672ab552f8c209 P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.70:33747
I20240503 08:13:15.059780 30915 tablet_copy_service.cc:140] P ce1496a47d084365ad786cd09e472a2c: Received BeginTabletCopySession request for tablet 362590d3d0834d0b8a672ab552f8c209 from peer aae860d7ae2b4002bf7794c278f9cc56 ({username='slave'} at 127.20.230.67:50141)
I20240503 08:13:15.059944 30915 tablet_copy_service.cc:161] P ce1496a47d084365ad786cd09e472a2c: Beginning new tablet copy session on tablet 362590d3d0834d0b8a672ab552f8c209 from peer aae860d7ae2b4002bf7794c278f9cc56 at {username='slave'} at 127.20.230.67:50141: session id = aae860d7ae2b4002bf7794c278f9cc56-362590d3d0834d0b8a672ab552f8c209
I20240503 08:13:15.062091 30915 tablet_copy_source_session.cc:205] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:15.064507 1271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 362590d3d0834d0b8a672ab552f8c209. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:15.071914 1271 tablet_copy_client.cc:783] T 362590d3d0834d0b8a672ab552f8c209 P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:15.072260 1271 tablet_copy_client.cc:647] T 362590d3d0834d0b8a672ab552f8c209 P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:15.075580 1271 tablet_copy_client.cc:523] T 362590d3d0834d0b8a672ab552f8c209 P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:15.080755 1271 tablet_bootstrap.cc:492] T 362590d3d0834d0b8a672ab552f8c209 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:13:15.093107 1271 tablet_bootstrap.cc:492] T 362590d3d0834d0b8a672ab552f8c209 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:15.093528 1271 tablet_bootstrap.cc:492] T 362590d3d0834d0b8a672ab552f8c209 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap complete.
I20240503 08:13:15.093940 1271 ts_tablet_manager.cc:1367] T 362590d3d0834d0b8a672ab552f8c209 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.002s
I20240503 08:13:15.094906 1271 raft_consensus.cc:348] T 362590d3d0834d0b8a672ab552f8c209 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:15.095156 1271 raft_consensus.cc:729] T 362590d3d0834d0b8a672ab552f8c209 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:13:15.095559 1271 consensus_queue.cc:260] T 362590d3d0834d0b8a672ab552f8c209 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:15.096310 1271 ts_tablet_manager.cc:1398] T 362590d3d0834d0b8a672ab552f8c209 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:13:15.099943 30915 tablet_copy_service.cc:342] P ce1496a47d084365ad786cd09e472a2c: Request end of tablet copy session aae860d7ae2b4002bf7794c278f9cc56-362590d3d0834d0b8a672ab552f8c209 received from {username='slave'} at 127.20.230.67:50141
I20240503 08:13:15.100109 30915 tablet_copy_service.cc:433] P ce1496a47d084365ad786cd09e472a2c: ending tablet copy session aae860d7ae2b4002bf7794c278f9cc56-362590d3d0834d0b8a672ab552f8c209 on tablet 362590d3d0834d0b8a672ab552f8c209 with peer aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:13:15.179121 1293 ts_tablet_manager.cc:902] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98: Initiating tablet copy from peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:13:15.181538 1205 ts_tablet_manager.cc:902] T 50e4d2c569344dde915eec12f6f3d747 P 1b411c6bb25549368041773d7b710d48: Initiating tablet copy from peer ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:13:15.234833 1293 tablet_copy_client.cc:308] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.67:39919
I20240503 08:13:15.236347 28283 tablet_copy_service.cc:140] P aae860d7ae2b4002bf7794c278f9cc56: Received BeginTabletCopySession request for tablet 8fc87f7f3c724cd4a270583ddebd87aa from peer b00f244c69df401997ea99d492218a98 ({username='slave'} at 127.20.230.72:55725)
I20240503 08:13:15.236524 28283 tablet_copy_service.cc:161] P aae860d7ae2b4002bf7794c278f9cc56: Beginning new tablet copy session on tablet 8fc87f7f3c724cd4a270583ddebd87aa from peer b00f244c69df401997ea99d492218a98 at {username='slave'} at 127.20.230.72:55725: session id = b00f244c69df401997ea99d492218a98-8fc87f7f3c724cd4a270583ddebd87aa
I20240503 08:13:15.238510 28283 tablet_copy_source_session.cc:205] T 8fc87f7f3c724cd4a270583ddebd87aa P aae860d7ae2b4002bf7794c278f9cc56: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:15.241235 1293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fc87f7f3c724cd4a270583ddebd87aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:15.251241 1293 tablet_copy_client.cc:783] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:15.252018 1293 tablet_copy_client.cc:647] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:15.254096 1293 tablet_copy_client.cc:523] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:15.258491 1293 tablet_bootstrap.cc:492] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:13:15.266649 1205 tablet_copy_client.cc:308] T 50e4d2c569344dde915eec12f6f3d747 P 1b411c6bb25549368041773d7b710d48: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.71:41915
I20240503 08:13:15.267379 31048 tablet_copy_service.cc:140] P ced21ca983b749d0b4d4f6a0e2426bf8: Received BeginTabletCopySession request for tablet 50e4d2c569344dde915eec12f6f3d747 from peer 1b411c6bb25549368041773d7b710d48 ({username='slave'} at 127.20.230.66:37577)
I20240503 08:13:15.267591 31048 tablet_copy_service.cc:161] P ced21ca983b749d0b4d4f6a0e2426bf8: Beginning new tablet copy session on tablet 50e4d2c569344dde915eec12f6f3d747 from peer 1b411c6bb25549368041773d7b710d48 at {username='slave'} at 127.20.230.66:37577: session id = 1b411c6bb25549368041773d7b710d48-50e4d2c569344dde915eec12f6f3d747
I20240503 08:13:15.273444 31048 tablet_copy_source_session.cc:205] T 50e4d2c569344dde915eec12f6f3d747 P ced21ca983b749d0b4d4f6a0e2426bf8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:15.275723 1205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50e4d2c569344dde915eec12f6f3d747. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:15.283387 1205 tablet_copy_client.cc:783] T 50e4d2c569344dde915eec12f6f3d747 P 1b411c6bb25549368041773d7b710d48: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:15.283763 1205 tablet_copy_client.cc:647] T 50e4d2c569344dde915eec12f6f3d747 P 1b411c6bb25549368041773d7b710d48: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:15.290562 1205 tablet_copy_client.cc:523] T 50e4d2c569344dde915eec12f6f3d747 P 1b411c6bb25549368041773d7b710d48: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:15.293912 1205 tablet_bootstrap.cc:492] T 50e4d2c569344dde915eec12f6f3d747 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:13:15.333742 1205 tablet_bootstrap.cc:492] T 50e4d2c569344dde915eec12f6f3d747 P 1b411c6bb25549368041773d7b710d48: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:15.334146 1205 tablet_bootstrap.cc:492] T 50e4d2c569344dde915eec12f6f3d747 P 1b411c6bb25549368041773d7b710d48: Bootstrap complete.
I20240503 08:13:15.334458 1205 ts_tablet_manager.cc:1367] T 50e4d2c569344dde915eec12f6f3d747 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.041s user 0.009s sys 0.015s
I20240503 08:13:15.335198 1205 raft_consensus.cc:348] T 50e4d2c569344dde915eec12f6f3d747 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:15.335547 1205 raft_consensus.cc:729] T 50e4d2c569344dde915eec12f6f3d747 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:13:15.335810 1205 consensus_queue.cc:260] T 50e4d2c569344dde915eec12f6f3d747 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:15.338218 1205 ts_tablet_manager.cc:1398] T 50e4d2c569344dde915eec12f6f3d747 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:13:15.339075 31048 tablet_copy_service.cc:342] P ced21ca983b749d0b4d4f6a0e2426bf8: Request end of tablet copy session 1b411c6bb25549368041773d7b710d48-50e4d2c569344dde915eec12f6f3d747 received from {username='slave'} at 127.20.230.66:37577
I20240503 08:13:15.339238 31048 tablet_copy_service.cc:433] P ced21ca983b749d0b4d4f6a0e2426bf8: ending tablet copy session 1b411c6bb25549368041773d7b710d48-50e4d2c569344dde915eec12f6f3d747 on tablet 50e4d2c569344dde915eec12f6f3d747 with peer 1b411c6bb25549368041773d7b710d48
I20240503 08:13:15.365146 1293 tablet_bootstrap.cc:492] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:15.365649 1293 tablet_bootstrap.cc:492] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98: Bootstrap complete.
I20240503 08:13:15.365973 1293 ts_tablet_manager.cc:1367] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.108s user 0.054s sys 0.005s
I20240503 08:13:15.366874 1293 raft_consensus.cc:348] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:15.367147 1293 raft_consensus.cc:729] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:13:15.367383 1293 consensus_queue.cc:260] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:15.368719 1293 ts_tablet_manager.cc:1398] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:13:15.373608 28283 tablet_copy_service.cc:342] P aae860d7ae2b4002bf7794c278f9cc56: Request end of tablet copy session b00f244c69df401997ea99d492218a98-8fc87f7f3c724cd4a270583ddebd87aa received from {username='slave'} at 127.20.230.72:55725
I20240503 08:13:15.373723 28283 tablet_copy_service.cc:433] P aae860d7ae2b4002bf7794c278f9cc56: ending tablet copy session b00f244c69df401997ea99d492218a98-8fc87f7f3c724cd4a270583ddebd87aa on tablet 8fc87f7f3c724cd4a270583ddebd87aa with peer b00f244c69df401997ea99d492218a98
I20240503 08:13:15.376051 31028 raft_consensus.cc:1203] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:15.392159 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:15.396705 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:15.404711 28133 raft_consensus.cc:1203] T 1cac39d5831d4b6887250f7a9d63dcd5 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:15.438278 1338 ts_tablet_manager.cc:902] T d808d27e71a5486f9773ad4b239d1cfe P 833582ff68c94269be364ebd187a75db: Initiating tablet copy from peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:13:15.445072 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:15.450377 28243 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:15.461472 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:15.476490 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:15.484004 1338 tablet_copy_client.cc:308] T d808d27e71a5486f9773ad4b239d1cfe P 833582ff68c94269be364ebd187a75db: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.68:46211
I20240503 08:13:15.484856 28413 tablet_copy_service.cc:140] P d09a11a03989421b9f28019dc1c14cc2: Received BeginTabletCopySession request for tablet d808d27e71a5486f9773ad4b239d1cfe from peer 833582ff68c94269be364ebd187a75db ({username='slave'} at 127.20.230.65:36529)
I20240503 08:13:15.485059 28413 tablet_copy_service.cc:161] P d09a11a03989421b9f28019dc1c14cc2: Beginning new tablet copy session on tablet d808d27e71a5486f9773ad4b239d1cfe from peer 833582ff68c94269be364ebd187a75db at {username='slave'} at 127.20.230.65:36529: session id = 833582ff68c94269be364ebd187a75db-d808d27e71a5486f9773ad4b239d1cfe
I20240503 08:13:15.488322 28413 tablet_copy_source_session.cc:205] T d808d27e71a5486f9773ad4b239d1cfe P d09a11a03989421b9f28019dc1c14cc2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:15.490379 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:15.490522 1338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d808d27e71a5486f9773ad4b239d1cfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:15.501144 1338 tablet_copy_client.cc:783] T d808d27e71a5486f9773ad4b239d1cfe P 833582ff68c94269be364ebd187a75db: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:15.501576 1338 tablet_copy_client.cc:647] T d808d27e71a5486f9773ad4b239d1cfe P 833582ff68c94269be364ebd187a75db: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:15.503547 1338 tablet_copy_client.cc:523] T d808d27e71a5486f9773ad4b239d1cfe P 833582ff68c94269be364ebd187a75db: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:15.507293 1338 tablet_bootstrap.cc:492] T d808d27e71a5486f9773ad4b239d1cfe P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:13:15.510162 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:15.526965 1338 tablet_bootstrap.cc:492] T d808d27e71a5486f9773ad4b239d1cfe P 833582ff68c94269be364ebd187a75db: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:15.527405 1338 tablet_bootstrap.cc:492] T d808d27e71a5486f9773ad4b239d1cfe P 833582ff68c94269be364ebd187a75db: Bootstrap complete.
I20240503 08:13:15.527751 1338 ts_tablet_manager.cc:1367] T d808d27e71a5486f9773ad4b239d1cfe P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.006s
I20240503 08:13:15.534085 1338 raft_consensus.cc:348] T d808d27e71a5486f9773ad4b239d1cfe P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:15.534325 1338 raft_consensus.cc:729] T d808d27e71a5486f9773ad4b239d1cfe P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:13:15.534569 1338 consensus_queue.cc:260] T d808d27e71a5486f9773ad4b239d1cfe P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:15.535216 1338 ts_tablet_manager.cc:1398] T d808d27e71a5486f9773ad4b239d1cfe P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240503 08:13:15.535950 28413 tablet_copy_service.cc:342] P d09a11a03989421b9f28019dc1c14cc2: Request end of tablet copy session 833582ff68c94269be364ebd187a75db-d808d27e71a5486f9773ad4b239d1cfe received from {username='slave'} at 127.20.230.65:36529
I20240503 08:13:15.536094 28413 tablet_copy_service.cc:433] P d09a11a03989421b9f28019dc1c14cc2: ending tablet copy session 833582ff68c94269be364ebd187a75db-d808d27e71a5486f9773ad4b239d1cfe on tablet d808d27e71a5486f9773ad4b239d1cfe with peer 833582ff68c94269be364ebd187a75db
I20240503 08:13:15.556499 28262 raft_consensus.cc:1203] T 362590d3d0834d0b8a672ab552f8c209 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240503 08:13:15.588325 28199 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.65:39081, user_credentials={real_user=slave}} blocked reactor thread for 137415us
I20240503 08:13:15.602408 1271 ts_tablet_manager.cc:902] T 45e7a03f838d4beaae2080d523319d58 P aae860d7ae2b4002bf7794c278f9cc56: Initiating tablet copy from peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:13:15.617936 31159 raft_consensus.cc:1203] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:15.673108 1271 tablet_copy_client.cc:308] T 45e7a03f838d4beaae2080d523319d58 P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.68:46211
I20240503 08:13:15.674621 28413 tablet_copy_service.cc:140] P d09a11a03989421b9f28019dc1c14cc2: Received BeginTabletCopySession request for tablet 45e7a03f838d4beaae2080d523319d58 from peer aae860d7ae2b4002bf7794c278f9cc56 ({username='slave'} at 127.20.230.67:39219)
I20240503 08:13:15.674767 28413 tablet_copy_service.cc:161] P d09a11a03989421b9f28019dc1c14cc2: Beginning new tablet copy session on tablet 45e7a03f838d4beaae2080d523319d58 from peer aae860d7ae2b4002bf7794c278f9cc56 at {username='slave'} at 127.20.230.67:39219: session id = aae860d7ae2b4002bf7794c278f9cc56-45e7a03f838d4beaae2080d523319d58
I20240503 08:13:15.683158 28132 raft_consensus.cc:1203] T 50e4d2c569344dde915eec12f6f3d747 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:15.677733 28413 tablet_copy_source_session.cc:205] T 45e7a03f838d4beaae2080d523319d58 P d09a11a03989421b9f28019dc1c14cc2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:15.695192 1271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45e7a03f838d4beaae2080d523319d58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:15.720417 1271 tablet_copy_client.cc:783] T 45e7a03f838d4beaae2080d523319d58 P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:15.720918 1271 tablet_copy_client.cc:647] T 45e7a03f838d4beaae2080d523319d58 P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:15.723484 1271 tablet_copy_client.cc:523] T 45e7a03f838d4beaae2080d523319d58 P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:15.728947 1271 tablet_bootstrap.cc:492] T 45e7a03f838d4beaae2080d523319d58 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
W20240503 08:13:15.735183 30961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.66:37143, user_credentials={real_user=slave}} blocked reactor thread for 47030.1us
I20240503 08:13:15.755875 1271 tablet_bootstrap.cc:492] T 45e7a03f838d4beaae2080d523319d58 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:15.756201 1271 tablet_bootstrap.cc:492] T 45e7a03f838d4beaae2080d523319d58 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap complete.
I20240503 08:13:15.756439 1271 ts_tablet_manager.cc:1367] T 45e7a03f838d4beaae2080d523319d58 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.001s
I20240503 08:13:15.757134 1271 raft_consensus.cc:348] T 45e7a03f838d4beaae2080d523319d58 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:15.757347 1271 raft_consensus.cc:729] T 45e7a03f838d4beaae2080d523319d58 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:13:15.757550 1271 consensus_queue.cc:260] T 45e7a03f838d4beaae2080d523319d58 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:15.758167 1271 ts_tablet_manager.cc:1398] T 45e7a03f838d4beaae2080d523319d58 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 08:13:15.759009 28413 tablet_copy_service.cc:342] P d09a11a03989421b9f28019dc1c14cc2: Request end of tablet copy session aae860d7ae2b4002bf7794c278f9cc56-45e7a03f838d4beaae2080d523319d58 received from {username='slave'} at 127.20.230.67:39219
I20240503 08:13:15.759130 28413 tablet_copy_service.cc:433] P d09a11a03989421b9f28019dc1c14cc2: ending tablet copy session aae860d7ae2b4002bf7794c278f9cc56-45e7a03f838d4beaae2080d523319d58 on tablet 45e7a03f838d4beaae2080d523319d58 with peer aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:13:15.885952 28002 raft_consensus.cc:1203] T d808d27e71a5486f9773ad4b239d1cfe P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:16.042562 28261 raft_consensus.cc:1203] T 45e7a03f838d4beaae2080d523319d58 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:16.068454 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:16.117244 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:16.137275 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:16.227092 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:16.236742 28243 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:16.236752 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:16.252254 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:16.297453 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:16.415733 31026 consensus_queue.cc:237] T 50e4d2c569344dde915eec12f6f3d747 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:16.418035 28133 raft_consensus.cc:1261] T 50e4d2c569344dde915eec12f6f3d747 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer ced21ca983b749d0b4d4f6a0e2426bf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:16.418428 31160 raft_consensus.cc:1261] T 50e4d2c569344dde915eec12f6f3d747 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer ced21ca983b749d0b4d4f6a0e2426bf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:16.418612 1352 consensus_queue.cc:1035] T 50e4d2c569344dde915eec12f6f3d747 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:16.419154 985 consensus_queue.cc:1035] T 50e4d2c569344dde915eec12f6f3d747 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:16.435343 1352 raft_consensus.cc:2932] T 50e4d2c569344dde915eec12f6f3d747 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:16.436048 31160 raft_consensus.cc:2932] T 50e4d2c569344dde915eec12f6f3d747 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:16.441939 27862 catalog_manager.cc:5484] T 50e4d2c569344dde915eec12f6f3d747 P b00f244c69df401997ea99d492218a98 reported cstate change: config changed from index 2 to 3, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New cstate: current_term: 1 leader_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:16.448485 28133 raft_consensus.cc:2932] T 50e4d2c569344dde915eec12f6f3d747 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:16.468251 30742 tablet_service.cc:1514] Processing DeleteTablet for tablet 50e4d2c569344dde915eec12f6f3d747 with delete_type TABLET_DATA_TOMBSTONED (TS 6d1d7de19459435fbde1999e3dfd226b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46794
I20240503 08:13:16.468948 1451 tablet_replica.cc:330] T 50e4d2c569344dde915eec12f6f3d747 P 6d1d7de19459435fbde1999e3dfd226b: stopping tablet replica
I20240503 08:13:16.469166 1451 raft_consensus.cc:2229] T 50e4d2c569344dde915eec12f6f3d747 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:16.469424 1451 raft_consensus.cc:2258] T 50e4d2c569344dde915eec12f6f3d747 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:16.470165 1451 ts_tablet_manager.cc:1875] T 50e4d2c569344dde915eec12f6f3d747 P 6d1d7de19459435fbde1999e3dfd226b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:16.475265 1451 ts_tablet_manager.cc:1888] T 50e4d2c569344dde915eec12f6f3d747 P 6d1d7de19459435fbde1999e3dfd226b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:13:16.475378 1451 log.cc:1196] T 50e4d2c569344dde915eec12f6f3d747 P 6d1d7de19459435fbde1999e3dfd226b: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal/wals/50e4d2c569344dde915eec12f6f3d747
I20240503 08:13:16.476213 27848 catalog_manager.cc:4830] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): tablet 50e4d2c569344dde915eec12f6f3d747 (table rebalancer_extra_table_18 [id=e4d6c0dac75042f984dc1aa8fa801cee]) successfully deleted
I20240503 08:13:16.848294 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:16.853340 28243 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:16.879621 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:16.883782 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:16.888967 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:16.898483 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:16.928181 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:16.967892 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:17.135331 28391 consensus_queue.cc:237] T 45e7a03f838d4beaae2080d523319d58 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:17.137499 28262 raft_consensus.cc:1261] T 45e7a03f838d4beaae2080d523319d58 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:17.137851 31160 raft_consensus.cc:1261] T 45e7a03f838d4beaae2080d523319d58 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:17.138142 1225 consensus_queue.cc:1035] T 45e7a03f838d4beaae2080d523319d58 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:17.138381 1414 consensus_queue.cc:1035] T 45e7a03f838d4beaae2080d523319d58 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:17.141026 1414 raft_consensus.cc:2932] T 45e7a03f838d4beaae2080d523319d58 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:17.146438 31160 raft_consensus.cc:2932] T 45e7a03f838d4beaae2080d523319d58 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:17.157418 27858 catalog_manager.cc:5484] T 45e7a03f838d4beaae2080d523319d58 P b00f244c69df401997ea99d492218a98 reported cstate change: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:17.160612 28261 raft_consensus.cc:2932] T 45e7a03f838d4beaae2080d523319d58 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:17.184777 31006 tablet_service.cc:1514] Processing DeleteTablet for tablet 45e7a03f838d4beaae2080d523319d58 with delete_type TABLET_DATA_TOMBSTONED (TS ced21ca983b749d0b4d4f6a0e2426bf8 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47020
I20240503 08:13:17.185523 1539 tablet_replica.cc:330] T 45e7a03f838d4beaae2080d523319d58 P ced21ca983b749d0b4d4f6a0e2426bf8: stopping tablet replica
I20240503 08:13:17.185724 1539 raft_consensus.cc:2229] T 45e7a03f838d4beaae2080d523319d58 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:17.186398 1539 raft_consensus.cc:2258] T 45e7a03f838d4beaae2080d523319d58 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:17.188009 1539 ts_tablet_manager.cc:1875] T 45e7a03f838d4beaae2080d523319d58 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:17.194168 1539 ts_tablet_manager.cc:1888] T 45e7a03f838d4beaae2080d523319d58 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:13:17.194284 1539 log.cc:1196] T 45e7a03f838d4beaae2080d523319d58 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal/wals/45e7a03f838d4beaae2080d523319d58
I20240503 08:13:17.195029 27847 catalog_manager.cc:4830] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): tablet 45e7a03f838d4beaae2080d523319d58 (table rebalancer_extra_table_64 [id=00d43ce433a44235afa9ed3a6510e586]) successfully deleted
I20240503 08:13:17.472532 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:17.519654 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:17.525168 28243 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:17.541738 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:17.586818 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:17.599498 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:17.607044 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:17.608752 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:17.802032 30760 consensus_queue.cc:237] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:17.804090 28262 raft_consensus.cc:1261] T f9b4f775b9914bdc98b9b582072cb1ab P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:17.804112 28001 raft_consensus.cc:1261] T f9b4f775b9914bdc98b9b582072cb1ab P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:17.804602 1394 consensus_queue.cc:1035] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:17.804893 1126 consensus_queue.cc:1035] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:17.807621 1537 raft_consensus.cc:2932] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:17.808225 28001 raft_consensus.cc:2932] T f9b4f775b9914bdc98b9b582072cb1ab P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:17.810705 28262 raft_consensus.cc:2932] T f9b4f775b9914bdc98b9b582072cb1ab P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:17.814729 27860 catalog_manager.cc:5484] T f9b4f775b9914bdc98b9b582072cb1ab P 833582ff68c94269be364ebd187a75db reported cstate change: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:17.831897 31006 tablet_service.cc:1514] Processing DeleteTablet for tablet f9b4f775b9914bdc98b9b582072cb1ab with delete_type TABLET_DATA_TOMBSTONED (TS ced21ca983b749d0b4d4f6a0e2426bf8 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47020
I20240503 08:13:17.832537 1621 tablet_replica.cc:330] T f9b4f775b9914bdc98b9b582072cb1ab P ced21ca983b749d0b4d4f6a0e2426bf8: stopping tablet replica
I20240503 08:13:17.832927 1621 raft_consensus.cc:2229] T f9b4f775b9914bdc98b9b582072cb1ab P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:17.833163 1621 raft_consensus.cc:2258] T f9b4f775b9914bdc98b9b582072cb1ab P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:17.833724 1621 ts_tablet_manager.cc:1875] T f9b4f775b9914bdc98b9b582072cb1ab P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:17.839212 1621 ts_tablet_manager.cc:1888] T f9b4f775b9914bdc98b9b582072cb1ab P ced21ca983b749d0b4d4f6a0e2426bf8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:13:17.839344 1621 log.cc:1196] T f9b4f775b9914bdc98b9b582072cb1ab P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal/wals/f9b4f775b9914bdc98b9b582072cb1ab
I20240503 08:13:17.840091 27847 catalog_manager.cc:4830] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): tablet f9b4f775b9914bdc98b9b582072cb1ab (table rebalancer_extra_table_32 [id=6e8613a79fc542daa3011dfac68e972f]) successfully deleted
I20240503 08:13:18.089547 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:18.103801 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:18.115871 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:18.182552 28243 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:18.199105 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:18.199643 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:18.202343 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:18.260365 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:18.394049 28001 consensus_queue.cc:237] T 3042212758bf4aef8be5c5d12d10113c P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:18.396116 30893 raft_consensus.cc:1261] T 3042212758bf4aef8be5c5d12d10113c P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:18.396624 28390 raft_consensus.cc:1261] T 3042212758bf4aef8be5c5d12d10113c P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:18.396873 1276 consensus_queue.cc:1035] T 3042212758bf4aef8be5c5d12d10113c P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:18.397804 1276 consensus_queue.cc:1035] T 3042212758bf4aef8be5c5d12d10113c P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:18.400626 1276 raft_consensus.cc:2932] T 3042212758bf4aef8be5c5d12d10113c P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:18.402159 30893 raft_consensus.cc:2932] T 3042212758bf4aef8be5c5d12d10113c P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:18.407989 28390 raft_consensus.cc:2932] T 3042212758bf4aef8be5c5d12d10113c P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:18.415592 27856 catalog_manager.cc:5484] T 3042212758bf4aef8be5c5d12d10113c P 833582ff68c94269be364ebd187a75db reported cstate change: config changed from index 4 to 5, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) evicted. New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:18.440357 28243 tablet_service.cc:1514] Processing DeleteTablet for tablet 3042212758bf4aef8be5c5d12d10113c with delete_type TABLET_DATA_TOMBSTONED (TS aae860d7ae2b4002bf7794c278f9cc56 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41960
I20240503 08:13:18.441013 1703 tablet_replica.cc:330] T 3042212758bf4aef8be5c5d12d10113c P aae860d7ae2b4002bf7794c278f9cc56: stopping tablet replica
I20240503 08:13:18.441291 1703 raft_consensus.cc:2229] T 3042212758bf4aef8be5c5d12d10113c P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:18.441637 1703 raft_consensus.cc:2258] T 3042212758bf4aef8be5c5d12d10113c P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:18.442227 1703 ts_tablet_manager.cc:1875] T 3042212758bf4aef8be5c5d12d10113c P aae860d7ae2b4002bf7794c278f9cc56: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:18.447189 1703 ts_tablet_manager.cc:1888] T 3042212758bf4aef8be5c5d12d10113c P aae860d7ae2b4002bf7794c278f9cc56: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:13:18.447331 1703 log.cc:1196] T 3042212758bf4aef8be5c5d12d10113c P aae860d7ae2b4002bf7794c278f9cc56: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal/wals/3042212758bf4aef8be5c5d12d10113c
I20240503 08:13:18.447938 27847 catalog_manager.cc:4830] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): tablet 3042212758bf4aef8be5c5d12d10113c (table rebalance_test_table_11 [id=e00af55ec1c448bdb2c617816b10a90f]) successfully deleted
I20240503 08:13:18.813612 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:18.828032 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:18.841235 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:18.847704 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:18.861783 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:18.895289 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:18.945281 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:18.968371 28243 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:19.097081 31159 consensus_queue.cc:237] T 013e00c67a1c4ae9b3c1a050b966e9a5 P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:19.099184 28391 raft_consensus.cc:1261] T 013e00c67a1c4ae9b3c1a050b966e9a5 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:19.099458 31028 raft_consensus.cc:1261] T 013e00c67a1c4ae9b3c1a050b966e9a5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:19.099720 959 consensus_queue.cc:1035] T 013e00c67a1c4ae9b3c1a050b966e9a5 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:19.101123 959 consensus_queue.cc:1035] T 013e00c67a1c4ae9b3c1a050b966e9a5 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:19.102203 473 raft_consensus.cc:2932] T 013e00c67a1c4ae9b3c1a050b966e9a5 P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:19.104182 31028 raft_consensus.cc:2932] T 013e00c67a1c4ae9b3c1a050b966e9a5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:19.110864 27860 catalog_manager.cc:5484] T 013e00c67a1c4ae9b3c1a050b966e9a5 P b00f244c69df401997ea99d492218a98 reported cstate change: config changed from index 2 to 3, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New cstate: current_term: 1 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:19.112180 28391 raft_consensus.cc:2932] T 013e00c67a1c4ae9b3c1a050b966e9a5 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:19.128015 30742 tablet_service.cc:1514] Processing DeleteTablet for tablet 013e00c67a1c4ae9b3c1a050b966e9a5 with delete_type TABLET_DATA_TOMBSTONED (TS 6d1d7de19459435fbde1999e3dfd226b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46794
I20240503 08:13:19.129329 1794 tablet_replica.cc:330] T 013e00c67a1c4ae9b3c1a050b966e9a5 P 6d1d7de19459435fbde1999e3dfd226b: stopping tablet replica
I20240503 08:13:19.129586 1794 raft_consensus.cc:2229] T 013e00c67a1c4ae9b3c1a050b966e9a5 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:19.129953 1794 raft_consensus.cc:2258] T 013e00c67a1c4ae9b3c1a050b966e9a5 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:19.130637 1794 ts_tablet_manager.cc:1875] T 013e00c67a1c4ae9b3c1a050b966e9a5 P 6d1d7de19459435fbde1999e3dfd226b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:19.136987 1794 ts_tablet_manager.cc:1888] T 013e00c67a1c4ae9b3c1a050b966e9a5 P 6d1d7de19459435fbde1999e3dfd226b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:13:19.137158 1794 log.cc:1196] T 013e00c67a1c4ae9b3c1a050b966e9a5 P 6d1d7de19459435fbde1999e3dfd226b: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal/wals/013e00c67a1c4ae9b3c1a050b966e9a5
I20240503 08:13:19.137890 27848 catalog_manager.cc:4830] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): tablet 013e00c67a1c4ae9b3c1a050b966e9a5 (table rebalancer_extra_table_0 [id=10122b5266cf4446bb541a06acf3a16f]) successfully deleted
I20240503 08:13:19.416559 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:19.423264 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:19.423684 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:19.431713 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:19.469446 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:13:19.515276 30962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.65:39081, user_credentials={real_user=slave}} blocked reactor thread for 91138.1us
I20240503 08:13:19.530095 28243 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:19.553625 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:19.638527 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:19.752000 28261 consensus_queue.cc:237] T 1cac39d5831d4b6887250f7a9d63dcd5 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:19.754024 28131 raft_consensus.cc:1261] T 1cac39d5831d4b6887250f7a9d63dcd5 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:19.754053 30760 raft_consensus.cc:1261] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:19.754572 1743 consensus_queue.cc:1035] T 1cac39d5831d4b6887250f7a9d63dcd5 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:19.755095 1743 consensus_queue.cc:1035] T 1cac39d5831d4b6887250f7a9d63dcd5 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:19.758148 1348 raft_consensus.cc:2932] T 1cac39d5831d4b6887250f7a9d63dcd5 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:19.759223 28131 raft_consensus.cc:2932] T 1cac39d5831d4b6887250f7a9d63dcd5 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:19.761965 30761 raft_consensus.cc:2932] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:19.765014 27855 catalog_manager.cc:5484] T 1cac39d5831d4b6887250f7a9d63dcd5 P 1b411c6bb25549368041773d7b710d48 reported cstate change: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:19.772773 31141 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cac39d5831d4b6887250f7a9d63dcd5 with delete_type TABLET_DATA_TOMBSTONED (TS b00f244c69df401997ea99d492218a98 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45814
I20240503 08:13:19.773840 1876 tablet_replica.cc:330] T 1cac39d5831d4b6887250f7a9d63dcd5 P b00f244c69df401997ea99d492218a98: stopping tablet replica
I20240503 08:13:19.774143 1876 raft_consensus.cc:2229] T 1cac39d5831d4b6887250f7a9d63dcd5 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:19.774468 1876 raft_consensus.cc:2258] T 1cac39d5831d4b6887250f7a9d63dcd5 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:19.775884 1876 ts_tablet_manager.cc:1875] T 1cac39d5831d4b6887250f7a9d63dcd5 P b00f244c69df401997ea99d492218a98: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:19.782828 1876 ts_tablet_manager.cc:1888] T 1cac39d5831d4b6887250f7a9d63dcd5 P b00f244c69df401997ea99d492218a98: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:13:19.782953 1876 log.cc:1196] T 1cac39d5831d4b6887250f7a9d63dcd5 P b00f244c69df401997ea99d492218a98: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal/wals/1cac39d5831d4b6887250f7a9d63dcd5
I20240503 08:13:19.783648 27846 catalog_manager.cc:4830] TS b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): tablet 1cac39d5831d4b6887250f7a9d63dcd5 (table rebalancer_extra_table_30 [id=b0813f15eaf04597949cf586c648712c]) successfully deleted
I20240503 08:13:20.031701 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:20.032151 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:20.033998 28243 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:20.072264 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:20.090725 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:20.125911 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:20.152968 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:20.199510 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:20.295562 30760 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7f2743ab5fa24c9899b67ea52f48f9d8"
dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35852
I20240503 08:13:20.295799 30760 raft_consensus.cc:595] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Received request to transfer leadership
I20240503 08:13:20.412132 1875 raft_consensus.cc:979] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b: : Instructing follower 833582ff68c94269be364ebd187a75db to start an election
I20240503 08:13:20.412328 1980 raft_consensus.cc:1067] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Signalling peer 833582ff68c94269be364ebd187a75db to start an election
I20240503 08:13:20.417307 28001 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7f2743ab5fa24c9899b67ea52f48f9d8"
dest_uuid: "833582ff68c94269be364ebd187a75db"
from {username='slave'} at 127.20.230.69:32907
I20240503 08:13:20.417557 28001 raft_consensus.cc:482] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:20.417656 28001 raft_consensus.cc:3037] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:20.423842 28001 raft_consensus.cc:504] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:20.425822 30760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f2743ab5fa24c9899b67ea52f48f9d8" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:13:20.426031 30760 raft_consensus.cc:3032] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:20.426131 30760 raft_consensus.cc:729] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Running, Role: LEADER
I20240503 08:13:20.426434 30760 consensus_queue.cc:260] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:20.426939 30760 raft_consensus.cc:3037] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:20.429617 30760 raft_consensus.cc:2454] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 2.
I20240503 08:13:20.430354 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f2743ab5fa24c9899b67ea52f48f9d8" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
I20240503 08:13:20.430554 28262 raft_consensus.cc:3037] T 7f2743ab5fa24c9899b67ea52f48f9d8 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:20.433274 28262 raft_consensus.cc:2454] T 7f2743ab5fa24c9899b67ea52f48f9d8 P aae860d7ae2b4002bf7794c278f9cc56 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 2.
I20240503 08:13:20.434870 27938 leader_election.cc:302] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, 833582ff68c94269be364ebd187a75db, aae860d7ae2b4002bf7794c278f9cc56; no voters:
I20240503 08:13:20.435238 1700 raft_consensus.cc:2781] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:20.435353 1700 raft_consensus.cc:686] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 833582ff68c94269be364ebd187a75db [term 2 LEADER]: Becoming Leader. State: Replica: 833582ff68c94269be364ebd187a75db, State: Running, Role: LEADER
I20240503 08:13:20.435598 1700 consensus_queue.cc:237] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:20.441936 28389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f2743ab5fa24c9899b67ea52f48f9d8" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:13:20.442118 28389 raft_consensus.cc:3037] T 7f2743ab5fa24c9899b67ea52f48f9d8 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:20.444696 28389 raft_consensus.cc:2454] T 7f2743ab5fa24c9899b67ea52f48f9d8 P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 2.
I20240503 08:13:20.450856 27853 catalog_manager.cc:5484] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 833582ff68c94269be364ebd187a75db reported cstate change: term changed from 1 to 2, leader changed from 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) to 833582ff68c94269be364ebd187a75db (127.20.230.65). New cstate: current_term: 2 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:20.436406 28001 leader_election.cc:288] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 election: Requested vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:13:20.543881 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:20.565438 30742 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 08:13:20.591465 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:20.602391 27983 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 08:13:20.626068 28243 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:20.714663 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:20.750874 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:20.794512 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:21.001336 28262 raft_consensus.cc:1261] T 7f2743ab5fa24c9899b67ea52f48f9d8 P aae860d7ae2b4002bf7794c278f9cc56 [term 2 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:13:21.001884 1699 consensus_queue.cc:1035] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:21.027519 28389 raft_consensus.cc:1261] T 7f2743ab5fa24c9899b67ea52f48f9d8 P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:13:21.031970 1700 consensus_queue.cc:1035] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:21.049525 30760 raft_consensus.cc:1261] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b [term 2 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:13:21.050169 1699 consensus_queue.cc:1035] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:21.189138 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:21.238945 30742 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 08:13:21.259209 30829 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.69:34725, user_credentials={real_user=slave}} blocked reactor thread for 55937.4us
I20240503 08:13:21.283984 27983 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 08:13:21.329538 28243 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:21.371062 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:21.386881 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:21.413408 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:21.480877 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:21.610205 28002 consensus_queue.cc:237] T 417cedc3fa5c4a85a7a130005291d64b P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:21.612226 31026 raft_consensus.cc:1261] T 417cedc3fa5c4a85a7a130005291d64b P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:21.612286 28131 raft_consensus.cc:1261] T 417cedc3fa5c4a85a7a130005291d64b P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:21.612677 2048 consensus_queue.cc:1035] T 417cedc3fa5c4a85a7a130005291d64b P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:21.612941 1699 consensus_queue.cc:1035] T 417cedc3fa5c4a85a7a130005291d64b P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:21.616356 2048 raft_consensus.cc:2932] T 417cedc3fa5c4a85a7a130005291d64b P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:21.616778 28131 raft_consensus.cc:2932] T 417cedc3fa5c4a85a7a130005291d64b P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:21.624086 27858 catalog_manager.cc:5484] T 417cedc3fa5c4a85a7a130005291d64b P 833582ff68c94269be364ebd187a75db reported cstate change: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:21.629106 31027 raft_consensus.cc:2932] T 417cedc3fa5c4a85a7a130005291d64b P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:21.662295 31141 tablet_service.cc:1514] Processing DeleteTablet for tablet 417cedc3fa5c4a85a7a130005291d64b with delete_type TABLET_DATA_TOMBSTONED (TS b00f244c69df401997ea99d492218a98 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45814
I20240503 08:13:21.671455 2100 tablet_replica.cc:330] T 417cedc3fa5c4a85a7a130005291d64b P b00f244c69df401997ea99d492218a98: stopping tablet replica
I20240503 08:13:21.671733 2100 raft_consensus.cc:2229] T 417cedc3fa5c4a85a7a130005291d64b P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:21.672000 2100 raft_consensus.cc:2258] T 417cedc3fa5c4a85a7a130005291d64b P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:21.672668 2100 ts_tablet_manager.cc:1875] T 417cedc3fa5c4a85a7a130005291d64b P b00f244c69df401997ea99d492218a98: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:21.684963 2100 ts_tablet_manager.cc:1888] T 417cedc3fa5c4a85a7a130005291d64b P b00f244c69df401997ea99d492218a98: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:13:21.685150 2100 log.cc:1196] T 417cedc3fa5c4a85a7a130005291d64b P b00f244c69df401997ea99d492218a98: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal/wals/417cedc3fa5c4a85a7a130005291d64b
I20240503 08:13:21.685832 27846 catalog_manager.cc:4830] TS b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): tablet 417cedc3fa5c4a85a7a130005291d64b (table rebalancer_extra_table_34 [id=505d74ce858645248ed95157f710daf1]) successfully deleted
I20240503 08:13:21.903259 27983 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 08:13:21.927011 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:21.955447 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:21.966056 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:21.968343 30742 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 08:13:21.982800 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:22.000476 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:22.017253 28243 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:22.183895 31027 consensus_queue.cc:237] T 7df30eb244914dcabc6965324eced236 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:22.185910 30895 raft_consensus.cc:1261] T 7df30eb244914dcabc6965324eced236 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer ced21ca983b749d0b4d4f6a0e2426bf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:22.185966 28131 raft_consensus.cc:1261] T 7df30eb244914dcabc6965324eced236 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer ced21ca983b749d0b4d4f6a0e2426bf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:22.186424 1938 consensus_queue.cc:1035] T 7df30eb244914dcabc6965324eced236 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:22.186919 1938 consensus_queue.cc:1035] T 7df30eb244914dcabc6965324eced236 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:22.189214 2139 raft_consensus.cc:2932] T 7df30eb244914dcabc6965324eced236 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:22.190481 30895 raft_consensus.cc:2932] T 7df30eb244914dcabc6965324eced236 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:22.194195 28131 raft_consensus.cc:2932] T 7df30eb244914dcabc6965324eced236 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:22.199585 27856 catalog_manager.cc:5484] T 7df30eb244914dcabc6965324eced236 P ce1496a47d084365ad786cd09e472a2c reported cstate change: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New cstate: current_term: 1 leader_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:22.213304 31141 tablet_service.cc:1514] Processing DeleteTablet for tablet 7df30eb244914dcabc6965324eced236 with delete_type TABLET_DATA_TOMBSTONED (TS b00f244c69df401997ea99d492218a98 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45814
I20240503 08:13:22.214672 2180 tablet_replica.cc:330] T 7df30eb244914dcabc6965324eced236 P b00f244c69df401997ea99d492218a98: stopping tablet replica
I20240503 08:13:22.214875 2180 raft_consensus.cc:2229] T 7df30eb244914dcabc6965324eced236 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:22.215652 2180 raft_consensus.cc:2258] T 7df30eb244914dcabc6965324eced236 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:22.216322 2180 ts_tablet_manager.cc:1875] T 7df30eb244914dcabc6965324eced236 P b00f244c69df401997ea99d492218a98: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:22.223230 2180 ts_tablet_manager.cc:1888] T 7df30eb244914dcabc6965324eced236 P b00f244c69df401997ea99d492218a98: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:13:22.223399 2180 log.cc:1196] T 7df30eb244914dcabc6965324eced236 P b00f244c69df401997ea99d492218a98: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal/wals/7df30eb244914dcabc6965324eced236
I20240503 08:13:22.224174 27846 catalog_manager.cc:4830] TS b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): tablet 7df30eb244914dcabc6965324eced236 (table rebalancer_extra_table_10 [id=9ee47b3ce8574777976ff3cb1ea03e9c]) successfully deleted
I20240503 08:13:22.467723 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:22.493693 28243 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:22.496850 27983 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 08:13:22.515046 30742 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 08:13:22.577365 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:22.618170 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:22.637001 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:22.735898 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:22.828007 28262 consensus_queue.cc:237] T 2222ff7fb5184d6c86010e1475d6c1f7 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:22.829735 28131 raft_consensus.cc:1261] T 2222ff7fb5184d6c86010e1475d6c1f7 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:22.829880 30760 raft_consensus.cc:1261] T 2222ff7fb5184d6c86010e1475d6c1f7 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:22.830217 2242 consensus_queue.cc:1035] T 2222ff7fb5184d6c86010e1475d6c1f7 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:22.830468 1743 consensus_queue.cc:1035] T 2222ff7fb5184d6c86010e1475d6c1f7 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:22.832839 2242 raft_consensus.cc:2932] T 2222ff7fb5184d6c86010e1475d6c1f7 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:22.833396 28131 raft_consensus.cc:2932] T 2222ff7fb5184d6c86010e1475d6c1f7 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:22.833705 30760 raft_consensus.cc:2932] T 2222ff7fb5184d6c86010e1475d6c1f7 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:22.838680 27856 catalog_manager.cc:5484] T 2222ff7fb5184d6c86010e1475d6c1f7 P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:22.849560 31141 tablet_service.cc:1514] Processing DeleteTablet for tablet 2222ff7fb5184d6c86010e1475d6c1f7 with delete_type TABLET_DATA_TOMBSTONED (TS b00f244c69df401997ea99d492218a98 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45814
I20240503 08:13:22.850179 2267 tablet_replica.cc:330] T 2222ff7fb5184d6c86010e1475d6c1f7 P b00f244c69df401997ea99d492218a98: stopping tablet replica
I20240503 08:13:22.850464 2267 raft_consensus.cc:2229] T 2222ff7fb5184d6c86010e1475d6c1f7 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:22.851415 2267 raft_consensus.cc:2258] T 2222ff7fb5184d6c86010e1475d6c1f7 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:22.852207 2267 ts_tablet_manager.cc:1875] T 2222ff7fb5184d6c86010e1475d6c1f7 P b00f244c69df401997ea99d492218a98: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:22.859678 2267 ts_tablet_manager.cc:1888] T 2222ff7fb5184d6c86010e1475d6c1f7 P b00f244c69df401997ea99d492218a98: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:13:22.859808 2267 log.cc:1196] T 2222ff7fb5184d6c86010e1475d6c1f7 P b00f244c69df401997ea99d492218a98: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal/wals/2222ff7fb5184d6c86010e1475d6c1f7
I20240503 08:13:22.860467 27846 catalog_manager.cc:4830] TS b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): tablet 2222ff7fb5184d6c86010e1475d6c1f7 (table rebalancer_extra_table_28 [id=49b49624ca8542a3a7429b4b6e32095d]) successfully deleted
I20240503 08:13:23.095850 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:23.113387 28243 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:23.140256 30742 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 08:13:23.148506 27983 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 08:13:23.193002 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:23.203490 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:23.235172 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:23.261073 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:23.387447 28262 consensus_queue.cc:237] T 8fc87f7f3c724cd4a270583ddebd87aa P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:23.389226 28133 raft_consensus.cc:1261] T 8fc87f7f3c724cd4a270583ddebd87aa P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:23.389710 1743 consensus_queue.cc:1035] T 8fc87f7f3c724cd4a270583ddebd87aa P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:23.389973 31160 raft_consensus.cc:1261] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:23.390414 1743 consensus_queue.cc:1035] T 8fc87f7f3c724cd4a270583ddebd87aa P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:23.392195 2242 raft_consensus.cc:2932] T 8fc87f7f3c724cd4a270583ddebd87aa P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:13:23.392818 28133 raft_consensus.cc:2932] T 8fc87f7f3c724cd4a270583ddebd87aa P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:13:23.393100 31160 raft_consensus.cc:2932] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:13:23.398037 27862 catalog_manager.cc:5484] T 8fc87f7f3c724cd4a270583ddebd87aa P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: config changed from index 4 to 5, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) evicted. New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:13:23.402623 28373 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fc87f7f3c724cd4a270583ddebd87aa with delete_type TABLET_DATA_TOMBSTONED (TS d09a11a03989421b9f28019dc1c14cc2 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34786
I20240503 08:13:23.403404 2348 tablet_replica.cc:330] T 8fc87f7f3c724cd4a270583ddebd87aa P d09a11a03989421b9f28019dc1c14cc2: stopping tablet replica
I20240503 08:13:23.403611 2348 raft_consensus.cc:2229] T 8fc87f7f3c724cd4a270583ddebd87aa P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:23.403892 2348 raft_consensus.cc:2258] T 8fc87f7f3c724cd4a270583ddebd87aa P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:23.404598 2348 ts_tablet_manager.cc:1875] T 8fc87f7f3c724cd4a270583ddebd87aa P d09a11a03989421b9f28019dc1c14cc2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:23.410410 2348 ts_tablet_manager.cc:1888] T 8fc87f7f3c724cd4a270583ddebd87aa P d09a11a03989421b9f28019dc1c14cc2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:13:23.410569 2348 log.cc:1196] T 8fc87f7f3c724cd4a270583ddebd87aa P d09a11a03989421b9f28019dc1c14cc2: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/wals/8fc87f7f3c724cd4a270583ddebd87aa
I20240503 08:13:23.411504 27849 catalog_manager.cc:4830] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): tablet 8fc87f7f3c724cd4a270583ddebd87aa (table rebalance_test_table_11 [id=e00af55ec1c448bdb2c617816b10a90f]) successfully deleted
I20240503 08:13:23.745790 30742 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 08:13:23.772405 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:23.772758 28243 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:23.807384 27983 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 08:13:23.815145 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:23.815625 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:23.838229 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:23.856493 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:23.966532 30894 consensus_queue.cc:237] T c9055668b86e4dc5aa321a4d29beaf74 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:23.968406 28389 raft_consensus.cc:1261] T c9055668b86e4dc5aa321a4d29beaf74 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:23.968421 28001 raft_consensus.cc:1261] T c9055668b86e4dc5aa321a4d29beaf74 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:23.968904 2405 consensus_queue.cc:1035] T c9055668b86e4dc5aa321a4d29beaf74 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:23.969177 2061 consensus_queue.cc:1035] T c9055668b86e4dc5aa321a4d29beaf74 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:23.982304 1696 raft_consensus.cc:2932] T c9055668b86e4dc5aa321a4d29beaf74 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:23.983244 28002 raft_consensus.cc:2932] T c9055668b86e4dc5aa321a4d29beaf74 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:23.989069 27862 catalog_manager.cc:5484] T c9055668b86e4dc5aa321a4d29beaf74 P ce1496a47d084365ad786cd09e472a2c reported cstate change: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New cstate: current_term: 1 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:23.989603 28389 raft_consensus.cc:2932] T c9055668b86e4dc5aa321a4d29beaf74 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:24.002748 31006 tablet_service.cc:1514] Processing DeleteTablet for tablet c9055668b86e4dc5aa321a4d29beaf74 with delete_type TABLET_DATA_TOMBSTONED (TS ced21ca983b749d0b4d4f6a0e2426bf8 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47020
I20240503 08:13:24.017479 2440 tablet_replica.cc:330] T c9055668b86e4dc5aa321a4d29beaf74 P ced21ca983b749d0b4d4f6a0e2426bf8: stopping tablet replica
I20240503 08:13:24.017670 2440 raft_consensus.cc:2229] T c9055668b86e4dc5aa321a4d29beaf74 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:24.018054 2440 raft_consensus.cc:2258] T c9055668b86e4dc5aa321a4d29beaf74 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:24.018903 2440 ts_tablet_manager.cc:1875] T c9055668b86e4dc5aa321a4d29beaf74 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:24.032660 2440 ts_tablet_manager.cc:1888] T c9055668b86e4dc5aa321a4d29beaf74 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:13:24.032850 2440 log.cc:1196] T c9055668b86e4dc5aa321a4d29beaf74 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal/wals/c9055668b86e4dc5aa321a4d29beaf74
I20240503 08:13:24.033565 27847 catalog_manager.cc:4830] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): tablet c9055668b86e4dc5aa321a4d29beaf74 (table rebalancer_extra_table_22 [id=11f85d35fe5f401d990c665eefacf247]) successfully deleted
I20240503 08:13:24.282194 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:24.319489 30742 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 08:13:24.323303 27983 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 08:13:24.360071 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:24.367815 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:24.368420 28243 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:24.369212 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:24.448303 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:24.557228 28002 consensus_queue.cc:237] T d3da6397e0c34c8f96933963dd4fa65e P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:24.559450 28390 raft_consensus.cc:1261] T d3da6397e0c34c8f96933963dd4fa65e P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:24.559949 2048 consensus_queue.cc:1035] T d3da6397e0c34c8f96933963dd4fa65e P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:24.560596 30894 raft_consensus.cc:1261] T d3da6397e0c34c8f96933963dd4fa65e P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:24.561051 2496 consensus_queue.cc:1035] T d3da6397e0c34c8f96933963dd4fa65e P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:24.562387 2048 raft_consensus.cc:2932] T d3da6397e0c34c8f96933963dd4fa65e P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:24.564241 30894 raft_consensus.cc:2932] T d3da6397e0c34c8f96933963dd4fa65e P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:24.566253 28390 raft_consensus.cc:2932] T d3da6397e0c34c8f96933963dd4fa65e P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:24.569830 27862 catalog_manager.cc:5484] T d3da6397e0c34c8f96933963dd4fa65e P ce1496a47d084365ad786cd09e472a2c reported cstate change: config changed from index 4 to 5, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) evicted. New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:24.593786 28243 tablet_service.cc:1514] Processing DeleteTablet for tablet d3da6397e0c34c8f96933963dd4fa65e with delete_type TABLET_DATA_TOMBSTONED (TS aae860d7ae2b4002bf7794c278f9cc56 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41960
I20240503 08:13:24.595227 2523 tablet_replica.cc:330] T d3da6397e0c34c8f96933963dd4fa65e P aae860d7ae2b4002bf7794c278f9cc56: stopping tablet replica
I20240503 08:13:24.595480 2523 raft_consensus.cc:2229] T d3da6397e0c34c8f96933963dd4fa65e P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:24.595810 2523 raft_consensus.cc:2258] T d3da6397e0c34c8f96933963dd4fa65e P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:24.596518 2523 ts_tablet_manager.cc:1875] T d3da6397e0c34c8f96933963dd4fa65e P aae860d7ae2b4002bf7794c278f9cc56: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:24.604154 2523 ts_tablet_manager.cc:1888] T d3da6397e0c34c8f96933963dd4fa65e P aae860d7ae2b4002bf7794c278f9cc56: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:13:24.604245 2523 log.cc:1196] T d3da6397e0c34c8f96933963dd4fa65e P aae860d7ae2b4002bf7794c278f9cc56: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal/wals/d3da6397e0c34c8f96933963dd4fa65e
I20240503 08:13:24.604828 27847 catalog_manager.cc:4830] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): tablet d3da6397e0c34c8f96933963dd4fa65e (table rebalance_test_table_3 [id=8e9aac1c8668451485cb809d15f7c27d]) successfully deleted
I20240503 08:13:24.877699 27983 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 08:13:24.901981 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:24.902172 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:24.918155 28243 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:24.921114 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:24.934032 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:24.959949 30742 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 08:13:24.971076 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:25.150033 28261 consensus_queue.cc:237] T de5dfccb0359462dbb7da776293fa679 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:25.152496 31027 raft_consensus.cc:1261] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:25.152998 2492 consensus_queue.cc:1035] T de5dfccb0359462dbb7da776293fa679 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:25.153986 28133 raft_consensus.cc:1261] T de5dfccb0359462dbb7da776293fa679 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:25.154556 1743 consensus_queue.cc:1035] T de5dfccb0359462dbb7da776293fa679 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:25.157635 2242 raft_consensus.cc:2932] T de5dfccb0359462dbb7da776293fa679 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:13:25.160122 28131 raft_consensus.cc:2932] T de5dfccb0359462dbb7da776293fa679 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:13:25.162590 27862 catalog_manager.cc:5484] T de5dfccb0359462dbb7da776293fa679 P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: config changed from index 2 to 3, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) evicted. New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:13:25.164415 31027 raft_consensus.cc:2932] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:13:25.180531 27983 tablet_service.cc:1514] Processing DeleteTablet for tablet de5dfccb0359462dbb7da776293fa679 with delete_type TABLET_DATA_TOMBSTONED (TS 833582ff68c94269be364ebd187a75db not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:55610
I20240503 08:13:25.182895 2616 tablet_replica.cc:330] T de5dfccb0359462dbb7da776293fa679 P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:13:25.183095 2616 raft_consensus.cc:2229] T de5dfccb0359462dbb7da776293fa679 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:25.183331 2616 raft_consensus.cc:2258] T de5dfccb0359462dbb7da776293fa679 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:25.183938 2616 ts_tablet_manager.cc:1875] T de5dfccb0359462dbb7da776293fa679 P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:25.190588 2616 ts_tablet_manager.cc:1888] T de5dfccb0359462dbb7da776293fa679 P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:13:25.190691 2616 log.cc:1196] T de5dfccb0359462dbb7da776293fa679 P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/de5dfccb0359462dbb7da776293fa679
I20240503 08:13:25.191233 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet de5dfccb0359462dbb7da776293fa679 (table rebalancer_extra_table_50 [id=c50031f182e6429695396372dc04b4e9]) successfully deleted
I20240503 08:13:25.408793 30742 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 08:13:25.415211 28243 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:25.415745 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:25.462725 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:25.488772 27983 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 08:13:25.535367 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:25.592082 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:25.604050 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:25.746529 28261 consensus_queue.cc:237] T 8450e62352a74d2ea1471166d4cb1132 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:25.748428 28133 raft_consensus.cc:1261] T 8450e62352a74d2ea1471166d4cb1132 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:25.749128 2242 consensus_queue.cc:1035] T 8450e62352a74d2ea1471166d4cb1132 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:25.749874 31027 raft_consensus.cc:1261] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:25.750867 2492 consensus_queue.cc:1035] T 8450e62352a74d2ea1471166d4cb1132 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:25.751751 2492 raft_consensus.cc:2932] T 8450e62352a74d2ea1471166d4cb1132 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:13:25.752312 28133 raft_consensus.cc:2932] T 8450e62352a74d2ea1471166d4cb1132 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:13:25.753695 31027 raft_consensus.cc:2932] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:13:25.757875 27858 catalog_manager.cc:5484] T 8450e62352a74d2ea1471166d4cb1132 P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: config changed from index 4 to 5, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) evicted. New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:13:25.765614 27983 tablet_service.cc:1514] Processing DeleteTablet for tablet 8450e62352a74d2ea1471166d4cb1132 with delete_type TABLET_DATA_TOMBSTONED (TS 833582ff68c94269be364ebd187a75db not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:55610
I20240503 08:13:25.766713 2701 tablet_replica.cc:330] T 8450e62352a74d2ea1471166d4cb1132 P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:13:25.766973 2701 raft_consensus.cc:2229] T 8450e62352a74d2ea1471166d4cb1132 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:25.767249 2701 raft_consensus.cc:2258] T 8450e62352a74d2ea1471166d4cb1132 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:25.767980 2701 ts_tablet_manager.cc:1875] T 8450e62352a74d2ea1471166d4cb1132 P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:25.773774 2701 ts_tablet_manager.cc:1888] T 8450e62352a74d2ea1471166d4cb1132 P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:13:25.773947 2701 log.cc:1196] T 8450e62352a74d2ea1471166d4cb1132 P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/8450e62352a74d2ea1471166d4cb1132
I20240503 08:13:25.774806 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet 8450e62352a74d2ea1471166d4cb1132 (table rebalance_test_table_3 [id=8e9aac1c8668451485cb809d15f7c27d]) successfully deleted
I20240503 08:13:25.997932 28243 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:26.010468 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:26.041437 30742 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 08:13:26.046249 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:26.070183 27983 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 08:13:26.075784 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:26.099588 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:26.106741 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:26.284929 28131 consensus_queue.cc:237] T 7a101bee3af34166848ee7fa7c733892 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:26.287182 28391 raft_consensus.cc:1261] T 7a101bee3af34166848ee7fa7c733892 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:26.287182 30895 raft_consensus.cc:1261] T 7a101bee3af34166848ee7fa7c733892 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:26.287813 2237 consensus_queue.cc:1035] T 7a101bee3af34166848ee7fa7c733892 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:26.288133 2316 consensus_queue.cc:1035] T 7a101bee3af34166848ee7fa7c733892 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:26.291014 2316 raft_consensus.cc:2932] T 7a101bee3af34166848ee7fa7c733892 P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:26.294162 28391 raft_consensus.cc:2932] T 7a101bee3af34166848ee7fa7c733892 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:26.294463 30895 raft_consensus.cc:2932] T 7a101bee3af34166848ee7fa7c733892 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:26.299293 27853 catalog_manager.cc:5484] T 7a101bee3af34166848ee7fa7c733892 P 1b411c6bb25549368041773d7b710d48 reported cstate change: config changed from index 4 to 5, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) evicted. New cstate: current_term: 1 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:26.312623 28243 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a101bee3af34166848ee7fa7c733892 with delete_type TABLET_DATA_TOMBSTONED (TS aae860d7ae2b4002bf7794c278f9cc56 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41960
I20240503 08:13:26.326119 2781 tablet_replica.cc:330] T 7a101bee3af34166848ee7fa7c733892 P aae860d7ae2b4002bf7794c278f9cc56: stopping tablet replica
I20240503 08:13:26.326370 2781 raft_consensus.cc:2229] T 7a101bee3af34166848ee7fa7c733892 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:26.326644 2781 raft_consensus.cc:2258] T 7a101bee3af34166848ee7fa7c733892 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:26.327306 2781 ts_tablet_manager.cc:1875] T 7a101bee3af34166848ee7fa7c733892 P aae860d7ae2b4002bf7794c278f9cc56: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:26.333349 2781 ts_tablet_manager.cc:1888] T 7a101bee3af34166848ee7fa7c733892 P aae860d7ae2b4002bf7794c278f9cc56: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:13:26.333460 2781 log.cc:1196] T 7a101bee3af34166848ee7fa7c733892 P aae860d7ae2b4002bf7794c278f9cc56: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal/wals/7a101bee3af34166848ee7fa7c733892
I20240503 08:13:26.334118 27847 catalog_manager.cc:4830] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): tablet 7a101bee3af34166848ee7fa7c733892 (table rebalance_test_table_13 [id=6e8538506685489f8a1e931a2a781746]) successfully deleted
I20240503 08:13:26.560971 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:26.576975 30742 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 08:13:26.584491 27983 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 08:13:26.604615 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:26.622519 28243 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:26.641372 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:26.700341 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:26.755101 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:26.847209 28261 consensus_queue.cc:237] T 8cc6ae30aa60495d8a4364a11d5bef79 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:26.849650 30761 raft_consensus.cc:1261] T 8cc6ae30aa60495d8a4364a11d5bef79 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:26.849711 28001 raft_consensus.cc:1261] T 8cc6ae30aa60495d8a4364a11d5bef79 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:26.850121 1743 consensus_queue.cc:1035] T 8cc6ae30aa60495d8a4364a11d5bef79 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:26.850337 2492 consensus_queue.cc:1035] T 8cc6ae30aa60495d8a4364a11d5bef79 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:26.853384 2492 raft_consensus.cc:2932] T 8cc6ae30aa60495d8a4364a11d5bef79 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:26.853838 28003 raft_consensus.cc:2932] T 8cc6ae30aa60495d8a4364a11d5bef79 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:26.857784 30761 raft_consensus.cc:2932] T 8cc6ae30aa60495d8a4364a11d5bef79 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:26.859647 27854 catalog_manager.cc:5484] T 8cc6ae30aa60495d8a4364a11d5bef79 P 833582ff68c94269be364ebd187a75db reported cstate change: config changed from index 4 to 5, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) evicted. New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:26.867534 28373 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cc6ae30aa60495d8a4364a11d5bef79 with delete_type TABLET_DATA_TOMBSTONED (TS d09a11a03989421b9f28019dc1c14cc2 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34786
I20240503 08:13:26.868906 2862 tablet_replica.cc:330] T 8cc6ae30aa60495d8a4364a11d5bef79 P d09a11a03989421b9f28019dc1c14cc2: stopping tablet replica
I20240503 08:13:26.869096 2862 raft_consensus.cc:2229] T 8cc6ae30aa60495d8a4364a11d5bef79 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:26.869395 2862 raft_consensus.cc:2258] T 8cc6ae30aa60495d8a4364a11d5bef79 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:26.870117 2862 ts_tablet_manager.cc:1875] T 8cc6ae30aa60495d8a4364a11d5bef79 P d09a11a03989421b9f28019dc1c14cc2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:26.877529 2862 ts_tablet_manager.cc:1888] T 8cc6ae30aa60495d8a4364a11d5bef79 P d09a11a03989421b9f28019dc1c14cc2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:13:26.877645 2862 log.cc:1196] T 8cc6ae30aa60495d8a4364a11d5bef79 P d09a11a03989421b9f28019dc1c14cc2: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/wals/8cc6ae30aa60495d8a4364a11d5bef79
I20240503 08:13:26.878307 27849 catalog_manager.cc:4830] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): tablet 8cc6ae30aa60495d8a4364a11d5bef79 (table rebalance_test_table_7 [id=f65dccb36e67468c8e28e95e66b5e4ef]) successfully deleted
I20240503 08:13:27.183475 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:27.210295 27983 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 08:13:27.230037 30742 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 08:13:27.255281 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:27.265417 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:27.312494 28243 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:27.381701 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:27.412070 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:27.473850 30894 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "362590d3d0834d0b8a672ab552f8c209"
dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46380
I20240503 08:13:27.474100 30894 raft_consensus.cc:595] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Received request to transfer leadership
I20240503 08:13:27.615145 2909 raft_consensus.cc:979] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c: : Instructing follower aae860d7ae2b4002bf7794c278f9cc56 to start an election
I20240503 08:13:27.615310 2909 raft_consensus.cc:1067] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Signalling peer aae860d7ae2b4002bf7794c278f9cc56 to start an election
I20240503 08:13:27.617312 28261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "362590d3d0834d0b8a672ab552f8c209"
dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
from {username='slave'} at 127.20.230.70:57835
I20240503 08:13:27.617542 28261 raft_consensus.cc:482] T 362590d3d0834d0b8a672ab552f8c209 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:27.617640 28261 raft_consensus.cc:3037] T 362590d3d0834d0b8a672ab552f8c209 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:27.620903 28261 raft_consensus.cc:504] T 362590d3d0834d0b8a672ab552f8c209 P aae860d7ae2b4002bf7794c278f9cc56 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:27.622166 30894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "362590d3d0834d0b8a672ab552f8c209" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:13:27.622184 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "362590d3d0834d0b8a672ab552f8c209" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:13:27.622356 28391 raft_consensus.cc:3037] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:27.622360 30894 raft_consensus.cc:3032] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:27.622423 30894 raft_consensus.cc:729] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Running, Role: LEADER
I20240503 08:13:27.622687 30894 consensus_queue.cc:260] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:27.622967 30894 raft_consensus.cc:3037] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:27.624720 30894 raft_consensus.cc:2454] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 2.
I20240503 08:13:27.625069 28391 raft_consensus.cc:2454] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 2.
I20240503 08:13:27.625926 28133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "362590d3d0834d0b8a672ab552f8c209" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:13:27.626132 28133 raft_consensus.cc:3037] T 362590d3d0834d0b8a672ab552f8c209 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:27.627961 28133 raft_consensus.cc:2454] T 362590d3d0834d0b8a672ab552f8c209 P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 2.
I20240503 08:13:27.621749 28261 leader_election.cc:288] T 362590d3d0834d0b8a672ab552f8c209 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 2 election: Requested vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:13:27.625793 28199 leader_election.cc:302] T 362590d3d0834d0b8a672ab552f8c209 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, ce1496a47d084365ad786cd09e472a2c, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:13:27.633031 2492 raft_consensus.cc:2781] T 362590d3d0834d0b8a672ab552f8c209 P aae860d7ae2b4002bf7794c278f9cc56 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:27.633133 2492 raft_consensus.cc:686] T 362590d3d0834d0b8a672ab552f8c209 P aae860d7ae2b4002bf7794c278f9cc56 [term 2 LEADER]: Becoming Leader. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Running, Role: LEADER
I20240503 08:13:27.633352 2492 consensus_queue.cc:237] T 362590d3d0834d0b8a672ab552f8c209 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:27.657841 27861 catalog_manager.cc:5484] T 362590d3d0834d0b8a672ab552f8c209 P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: term changed from 1 to 2, leader changed from ce1496a47d084365ad786cd09e472a2c (127.20.230.70) to aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67). New cstate: current_term: 2 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:27.857591 27983 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 08:13:27.867954 30742 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 08:13:27.883929 28243 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:27.907677 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:27.918619 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:27.919066 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:27.971613 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:28.005403 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:28.047302 30894 raft_consensus.cc:1261] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:13:28.047852 28133 raft_consensus.cc:1261] T 362590d3d0834d0b8a672ab552f8c209 P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:13:28.049839 1743 consensus_queue.cc:1035] T 362590d3d0834d0b8a672ab552f8c209 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:28.050347 1743 consensus_queue.cc:1035] T 362590d3d0834d0b8a672ab552f8c209 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:28.065459 28391 raft_consensus.cc:1261] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:13:28.067124 1743 consensus_queue.cc:1035] T 362590d3d0834d0b8a672ab552f8c209 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:28.555853 30742 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 08:13:28.560773 28243 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:28.571846 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:28.574307 27983 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 08:13:28.582154 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:28.646404 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:28.687597 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:28.698015 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:28.829774 28390 consensus_queue.cc:237] T d808d27e71a5486f9773ad4b239d1cfe P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:28.831593 31159 raft_consensus.cc:1261] T d808d27e71a5486f9773ad4b239d1cfe P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:28.831579 28003 raft_consensus.cc:1261] T d808d27e71a5486f9773ad4b239d1cfe P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:28.832018 2923 consensus_queue.cc:1035] T d808d27e71a5486f9773ad4b239d1cfe P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:28.832291 3003 consensus_queue.cc:1035] T d808d27e71a5486f9773ad4b239d1cfe P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:28.834589 3003 raft_consensus.cc:2932] T d808d27e71a5486f9773ad4b239d1cfe P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:28.835135 28003 raft_consensus.cc:2932] T d808d27e71a5486f9773ad4b239d1cfe P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:28.840005 27862 catalog_manager.cc:5484] T d808d27e71a5486f9773ad4b239d1cfe P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: config changed from index 2 to 3, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:28.842090 31159 raft_consensus.cc:2932] T d808d27e71a5486f9773ad4b239d1cfe P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:28.849025 30874 tablet_service.cc:1514] Processing DeleteTablet for tablet d808d27e71a5486f9773ad4b239d1cfe with delete_type TABLET_DATA_TOMBSTONED (TS ce1496a47d084365ad786cd09e472a2c not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:34426
I20240503 08:13:28.849610 3100 tablet_replica.cc:330] T d808d27e71a5486f9773ad4b239d1cfe P ce1496a47d084365ad786cd09e472a2c: stopping tablet replica
I20240503 08:13:28.849920 3100 raft_consensus.cc:2229] T d808d27e71a5486f9773ad4b239d1cfe P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:28.850142 3100 raft_consensus.cc:2258] T d808d27e71a5486f9773ad4b239d1cfe P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:28.850703 3100 ts_tablet_manager.cc:1875] T d808d27e71a5486f9773ad4b239d1cfe P ce1496a47d084365ad786cd09e472a2c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:28.858487 3100 ts_tablet_manager.cc:1888] T d808d27e71a5486f9773ad4b239d1cfe P ce1496a47d084365ad786cd09e472a2c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:13:28.858673 3100 log.cc:1196] T d808d27e71a5486f9773ad4b239d1cfe P ce1496a47d084365ad786cd09e472a2c: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal/wals/d808d27e71a5486f9773ad4b239d1cfe
I20240503 08:13:28.859470 27846 catalog_manager.cc:4830] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): tablet d808d27e71a5486f9773ad4b239d1cfe (table rebalancer_extra_table_2 [id=7358500a74314a5a8795ddb89337f666]) successfully deleted
I20240503 08:13:29.258136 28243 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:29.281042 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:29.281977 30742 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 08:13:29.282452 27983 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 08:13:29.290558 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:29.304857 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:29.311007 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:29.329255 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:29.457607 28001 consensus_queue.cc:237] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:29.459457 28262 raft_consensus.cc:1261] T 7f2743ab5fa24c9899b67ea52f48f9d8 P aae860d7ae2b4002bf7794c278f9cc56 [term 2 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:29.459875 28390 raft_consensus.cc:1261] T 7f2743ab5fa24c9899b67ea52f48f9d8 P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:29.460258 2496 consensus_queue.cc:1035] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:29.460706 2496 consensus_queue.cc:1035] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:29.462880 2496 raft_consensus.cc:2932] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 833582ff68c94269be364ebd187a75db [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:29.464242 28262 raft_consensus.cc:2932] T 7f2743ab5fa24c9899b67ea52f48f9d8 P aae860d7ae2b4002bf7794c278f9cc56 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:29.471110 27857 catalog_manager.cc:5484] T 7f2743ab5fa24c9899b67ea52f48f9d8 P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: config changed from index 2 to 4, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New cstate: current_term: 2 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:29.491719 30742 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f2743ab5fa24c9899b67ea52f48f9d8 with delete_type TABLET_DATA_TOMBSTONED (TS 6d1d7de19459435fbde1999e3dfd226b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46794
I20240503 08:13:29.505409 3211 tablet_replica.cc:330] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b: stopping tablet replica
I20240503 08:13:29.505784 3211 raft_consensus.cc:2229] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:29.506083 3211 raft_consensus.cc:2258] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:29.506759 3211 ts_tablet_manager.cc:1875] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:29.511623 28391 raft_consensus.cc:2932] T 7f2743ab5fa24c9899b67ea52f48f9d8 P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:29.512533 3211 ts_tablet_manager.cc:1888] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240503 08:13:29.512636 3211 log.cc:1196] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 6d1d7de19459435fbde1999e3dfd226b: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal/wals/7f2743ab5fa24c9899b67ea52f48f9d8
I20240503 08:13:29.513267 27848 catalog_manager.cc:4830] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): tablet 7f2743ab5fa24c9899b67ea52f48f9d8 (table rebalancer_extra_table_62 [id=7adc395c24f24d63aa9f9ceed2c44c01]) successfully deleted
I20240503 08:13:29.902915 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:29.926903 27983 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 08:13:29.938079 28243 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:29.945333 30742 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 08:13:29.954211 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:29.974154 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:29.999017 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:30.027935 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240503 08:13:30.032649 28197 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.71:41915, user_credentials={real_user=slave}} blocked reactor thread for 56669.8us
I20240503 08:13:30.184707 28262 consensus_queue.cc:237] T 362590d3d0834d0b8a672ab552f8c209 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:30.186981 28131 raft_consensus.cc:1261] T 362590d3d0834d0b8a672ab552f8c209 P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:30.187062 28391 raft_consensus.cc:1261] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:30.187425 3317 consensus_queue.cc:1035] T 362590d3d0834d0b8a672ab552f8c209 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:30.187651 2492 consensus_queue.cc:1035] T 362590d3d0834d0b8a672ab552f8c209 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:30.190716 2492 raft_consensus.cc:2932] T 362590d3d0834d0b8a672ab552f8c209 P aae860d7ae2b4002bf7794c278f9cc56 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:30.191088 28391 raft_consensus.cc:2932] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:30.191188 28131 raft_consensus.cc:2932] T 362590d3d0834d0b8a672ab552f8c209 P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:30.196718 27854 catalog_manager.cc:5484] T 362590d3d0834d0b8a672ab552f8c209 P 1b411c6bb25549368041773d7b710d48 reported cstate change: config changed from index 2 to 4, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New cstate: current_term: 2 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:30.201663 30874 tablet_service.cc:1514] Processing DeleteTablet for tablet 362590d3d0834d0b8a672ab552f8c209 with delete_type TABLET_DATA_TOMBSTONED (TS ce1496a47d084365ad786cd09e472a2c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34426
I20240503 08:13:30.202526 3341 tablet_replica.cc:330] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c: stopping tablet replica
I20240503 08:13:30.202760 3341 raft_consensus.cc:2229] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:30.202952 3341 raft_consensus.cc:2258] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:30.203497 3341 ts_tablet_manager.cc:1875] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:30.211537 3341 ts_tablet_manager.cc:1888] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240503 08:13:30.211710 3341 log.cc:1196] T 362590d3d0834d0b8a672ab552f8c209 P ce1496a47d084365ad786cd09e472a2c: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal/wals/362590d3d0834d0b8a672ab552f8c209
I20240503 08:13:30.212428 27846 catalog_manager.cc:4830] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): tablet 362590d3d0834d0b8a672ab552f8c209 (table rebalancer_extra_table_36 [id=ae8d4f3f001646efbeb5154b3ce01aac]) successfully deleted
I20240503 08:13:30.237278 28262 consensus_queue.cc:237] T ca38671a760d4534ab903579e0487772 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:30.240103 28001 raft_consensus.cc:1261] T ca38671a760d4534ab903579e0487772 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:30.240598 2492 consensus_queue.cc:1035] T ca38671a760d4534ab903579e0487772 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:30.244273 28133 raft_consensus.cc:1261] T ca38671a760d4534ab903579e0487772 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:30.245082 2492 consensus_queue.cc:1035] T ca38671a760d4534ab903579e0487772 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:13:30.245661 28196 consensus_peers.cc:485] T ca38671a760d4534ab903579e0487772 P aae860d7ae2b4002bf7794c278f9cc56 -> Peer b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): Couldn't send request to peer b00f244c69df401997ea99d492218a98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ca38671a760d4534ab903579e0487772. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:30.262785 3007 raft_consensus.cc:2932] T ca38671a760d4534ab903579e0487772 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:13:30.263325 28133 raft_consensus.cc:2932] T ca38671a760d4534ab903579e0487772 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:13:30.268882 27857 catalog_manager.cc:5484] T ca38671a760d4534ab903579e0487772 P 1b411c6bb25549368041773d7b710d48 reported cstate change: config changed from index -1 to 4, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) added. New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:13:30.270291 28001 raft_consensus.cc:2932] T ca38671a760d4534ab903579e0487772 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:13:30.298171 30895 consensus_queue.cc:237] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:30.300220 30761 raft_consensus.cc:1261] T f53acf725b62447382e9bae95e6e74d5 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:30.301043 3164 consensus_queue.cc:1035] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:30.301388 31026 raft_consensus.cc:1261] T f53acf725b62447382e9bae95e6e74d5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:13:30.301891 30828 consensus_peers.cc:485] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f53acf725b62447382e9bae95e6e74d5. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:30.302173 3164 consensus_queue.cc:1035] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:30.304844 2909 raft_consensus.cc:2932] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:30.306061 31026 raft_consensus.cc:2932] T f53acf725b62447382e9bae95e6e74d5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:30.308387 30762 raft_consensus.cc:2932] T f53acf725b62447382e9bae95e6e74d5 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:30.311360 27854 catalog_manager.cc:5484] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c reported cstate change: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New cstate: current_term: 1 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:30.359274 31026 consensus_queue.cc:237] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:30.361673 28262 raft_consensus.cc:1261] T b12ceaf7f8d04fad8b07b36459da7116 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer ced21ca983b749d0b4d4f6a0e2426bf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:30.361711 28003 raft_consensus.cc:1261] T b12ceaf7f8d04fad8b07b36459da7116 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer ced21ca983b749d0b4d4f6a0e2426bf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:13:30.361826 30962 consensus_peers.cc:485] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): Couldn't send request to peer d09a11a03989421b9f28019dc1c14cc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b12ceaf7f8d04fad8b07b36459da7116. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:30.362149 3319 consensus_queue.cc:1035] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:30.362380 3303 consensus_queue.cc:1035] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:30.364691 3319 raft_consensus.cc:2932] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:30.365260 28003 raft_consensus.cc:2932] T b12ceaf7f8d04fad8b07b36459da7116 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:30.368350 28262 raft_consensus.cc:2932] T b12ceaf7f8d04fad8b07b36459da7116 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:30.371318 27854 catalog_manager.cc:5484] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8 reported cstate change: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New cstate: current_term: 1 leader_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:30.401279 28262 consensus_queue.cc:237] T cdad6ef53b6f413ebfda67f9b42c9c23 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:30.403229 28133 raft_consensus.cc:1261] T cdad6ef53b6f413ebfda67f9b42c9c23 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:30.403687 3317 consensus_queue.cc:1035] T cdad6ef53b6f413ebfda67f9b42c9c23 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:30.403898 28391 raft_consensus.cc:1261] T cdad6ef53b6f413ebfda67f9b42c9c23 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:13:30.403946 28198 consensus_peers.cc:485] T cdad6ef53b6f413ebfda67f9b42c9c23 P aae860d7ae2b4002bf7794c278f9cc56 -> Peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): Couldn't send request to peer 6d1d7de19459435fbde1999e3dfd226b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cdad6ef53b6f413ebfda67f9b42c9c23. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:30.404378 3317 consensus_queue.cc:1035] T cdad6ef53b6f413ebfda67f9b42c9c23 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:30.406893 3007 raft_consensus.cc:2932] T cdad6ef53b6f413ebfda67f9b42c9c23 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:30.407389 28133 raft_consensus.cc:2932] T cdad6ef53b6f413ebfda67f9b42c9c23 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:30.410106 28391 raft_consensus.cc:2932] T cdad6ef53b6f413ebfda67f9b42c9c23 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:30.412317 27857 catalog_manager.cc:5484] T cdad6ef53b6f413ebfda67f9b42c9c23 P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: config changed from index -1 to 4, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) added. New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:30.446957 28391 consensus_queue.cc:237] T 8425cfe5c6134a60852e12f9af55227b P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:30.449227 28003 raft_consensus.cc:1261] T 8425cfe5c6134a60852e12f9af55227b P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:13:30.449518 28327 consensus_peers.cc:485] T 8425cfe5c6134a60852e12f9af55227b P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8425cfe5c6134a60852e12f9af55227b. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:30.449710 31159 raft_consensus.cc:1261] T 8425cfe5c6134a60852e12f9af55227b P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:30.450206 3003 consensus_queue.cc:1035] T 8425cfe5c6134a60852e12f9af55227b P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:30.450453 2923 consensus_queue.cc:1035] T 8425cfe5c6134a60852e12f9af55227b P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:30.463652 3003 raft_consensus.cc:2932] T 8425cfe5c6134a60852e12f9af55227b P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:30.464278 31159 raft_consensus.cc:2932] T 8425cfe5c6134a60852e12f9af55227b P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:30.468429 28003 raft_consensus.cc:2932] T 8425cfe5c6134a60852e12f9af55227b P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:30.469667 27857 catalog_manager.cc:5484] T 8425cfe5c6134a60852e12f9af55227b P b00f244c69df401997ea99d492218a98 reported cstate change: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:30.513347 31159 consensus_queue.cc:237] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:30.516032 31026 raft_consensus.cc:1261] T 4a23c8186c4d4b23b62d7474088b45c1 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:30.516044 28262 raft_consensus.cc:1261] T 4a23c8186c4d4b23b62d7474088b45c1 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:13:30.516439 31094 consensus_peers.cc:485] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98 -> Peer ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): Couldn't send request to peer ce1496a47d084365ad786cd09e472a2c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4a23c8186c4d4b23b62d7474088b45c1. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:30.516608 2924 consensus_queue.cc:1035] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:30.516870 3316 consensus_queue.cc:1035] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:30.533931 3316 raft_consensus.cc:2932] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:30.537613 31027 raft_consensus.cc:2932] T 4a23c8186c4d4b23b62d7474088b45c1 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:30.541631 28262 raft_consensus.cc:2932] T 4a23c8186c4d4b23b62d7474088b45c1 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:30.543292 27854 catalog_manager.cc:5484] T 4a23c8186c4d4b23b62d7474088b45c1 P ced21ca983b749d0b4d4f6a0e2426bf8 reported cstate change: config changed from index -1 to 2, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New cstate: current_term: 1 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:30.596266 28003 consensus_queue.cc:237] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:30.598349 30762 raft_consensus.cc:1261] T df31bd9e544a44f280db2b0d021000c2 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:30.598408 30894 raft_consensus.cc:1261] T df31bd9e544a44f280db2b0d021000c2 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:13:30.598534 27939 consensus_peers.cc:485] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db -> Peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): Couldn't send request to peer d09a11a03989421b9f28019dc1c14cc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df31bd9e544a44f280db2b0d021000c2. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:30.598992 3358 consensus_queue.cc:1035] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:30.599599 3475 consensus_queue.cc:1035] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:30.602761 3358 raft_consensus.cc:2932] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:30.603302 30762 raft_consensus.cc:2932] T df31bd9e544a44f280db2b0d021000c2 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:30.606148 30894 raft_consensus.cc:2932] T df31bd9e544a44f280db2b0d021000c2 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:30.608337 27854 catalog_manager.cc:5484] T df31bd9e544a44f280db2b0d021000c2 P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:30.682688 28262 consensus_queue.cc:237] T 08fb64977309479b831635344969c662 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:30.684758 28003 raft_consensus.cc:1261] T 08fb64977309479b831635344969c662 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:13:30.684816 28198 consensus_peers.cc:485] T 08fb64977309479b831635344969c662 P aae860d7ae2b4002bf7794c278f9cc56 -> Peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): Couldn't send request to peer 6d1d7de19459435fbde1999e3dfd226b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 08fb64977309479b831635344969c662. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:30.684898 28133 raft_consensus.cc:1261] T 08fb64977309479b831635344969c662 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:30.685246 3417 consensus_queue.cc:1035] T 08fb64977309479b831635344969c662 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:30.685498 3007 consensus_queue.cc:1035] T 08fb64977309479b831635344969c662 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:30.688846 3007 raft_consensus.cc:2932] T 08fb64977309479b831635344969c662 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:30.691787 28133 raft_consensus.cc:2932] T 08fb64977309479b831635344969c662 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:30.694213 28003 raft_consensus.cc:2932] T 08fb64977309479b831635344969c662 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:30.700924 27857 catalog_manager.cc:5484] T 08fb64977309479b831635344969c662 P 1b411c6bb25549368041773d7b710d48 reported cstate change: config changed from index -1 to 4, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) added. New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:30.754168 3502 ts_tablet_manager.cc:902] T ca38671a760d4534ab903579e0487772 P b00f244c69df401997ea99d492218a98: Initiating tablet copy from peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:13:30.755683 3502 tablet_copy_client.cc:308] T ca38671a760d4534ab903579e0487772 P b00f244c69df401997ea99d492218a98: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.67:39919
I20240503 08:13:30.756245 28283 tablet_copy_service.cc:140] P aae860d7ae2b4002bf7794c278f9cc56: Received BeginTabletCopySession request for tablet ca38671a760d4534ab903579e0487772 from peer b00f244c69df401997ea99d492218a98 ({username='slave'} at 127.20.230.72:55725)
I20240503 08:13:30.756376 28283 tablet_copy_service.cc:161] P aae860d7ae2b4002bf7794c278f9cc56: Beginning new tablet copy session on tablet ca38671a760d4534ab903579e0487772 from peer b00f244c69df401997ea99d492218a98 at {username='slave'} at 127.20.230.72:55725: session id = b00f244c69df401997ea99d492218a98-ca38671a760d4534ab903579e0487772
I20240503 08:13:30.759021 28283 tablet_copy_source_session.cc:205] T ca38671a760d4534ab903579e0487772 P aae860d7ae2b4002bf7794c278f9cc56: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:30.760099 3502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca38671a760d4534ab903579e0487772. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:30.765655 3502 tablet_copy_client.cc:783] T ca38671a760d4534ab903579e0487772 P b00f244c69df401997ea99d492218a98: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:30.765882 3502 tablet_copy_client.cc:647] T ca38671a760d4534ab903579e0487772 P b00f244c69df401997ea99d492218a98: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:30.768013 3502 tablet_copy_client.cc:523] T ca38671a760d4534ab903579e0487772 P b00f244c69df401997ea99d492218a98: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:30.769219 31159 consensus_queue.cc:237] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:30.771237 3502 tablet_bootstrap.cc:492] T ca38671a760d4534ab903579e0487772 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:13:30.771656 31026 raft_consensus.cc:1261] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:13:30.771770 31095 consensus_peers.cc:485] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 17c6e3293aad4591a9c1395ebffff2e6. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:30.771977 28133 raft_consensus.cc:1261] T 17c6e3293aad4591a9c1395ebffff2e6 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:30.772912 3455 consensus_queue.cc:1035] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:30.773154 2924 consensus_queue.cc:1035] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:30.782953 2924 raft_consensus.cc:2932] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:30.783783 28133 raft_consensus.cc:2932] T 17c6e3293aad4591a9c1395ebffff2e6 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:30.787122 31026 raft_consensus.cc:2932] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:30.792445 27854 catalog_manager.cc:5484] T 17c6e3293aad4591a9c1395ebffff2e6 P 1b411c6bb25549368041773d7b710d48 reported cstate change: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New cstate: current_term: 1 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:30.822938 3502 tablet_bootstrap.cc:492] T ca38671a760d4534ab903579e0487772 P b00f244c69df401997ea99d492218a98: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:30.823271 3502 tablet_bootstrap.cc:492] T ca38671a760d4534ab903579e0487772 P b00f244c69df401997ea99d492218a98: Bootstrap complete.
I20240503 08:13:30.823551 3502 ts_tablet_manager.cc:1367] T ca38671a760d4534ab903579e0487772 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.052s user 0.038s sys 0.001s
I20240503 08:13:30.824677 3502 raft_consensus.cc:348] T ca38671a760d4534ab903579e0487772 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:30.824982 3502 raft_consensus.cc:729] T ca38671a760d4534ab903579e0487772 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:13:30.825268 3502 consensus_queue.cc:260] T ca38671a760d4534ab903579e0487772 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:30.825964 3502 ts_tablet_manager.cc:1398] T ca38671a760d4534ab903579e0487772 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 08:13:30.826758 28283 tablet_copy_service.cc:342] P aae860d7ae2b4002bf7794c278f9cc56: Request end of tablet copy session b00f244c69df401997ea99d492218a98-ca38671a760d4534ab903579e0487772 received from {username='slave'} at 127.20.230.72:55725
I20240503 08:13:30.826880 28283 tablet_copy_service.cc:433] P aae860d7ae2b4002bf7794c278f9cc56: ending tablet copy session b00f244c69df401997ea99d492218a98-ca38671a760d4534ab903579e0487772 on tablet ca38671a760d4534ab903579e0487772 with peer b00f244c69df401997ea99d492218a98
I20240503 08:13:30.843490 31159 consensus_queue.cc:237] T 9fc39354ba064392be1af1769513f738 P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:30.845865 30762 raft_consensus.cc:1261] T 9fc39354ba064392be1af1769513f738 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:13:30.846074 31095 consensus_peers.cc:485] T 9fc39354ba064392be1af1769513f738 P b00f244c69df401997ea99d492218a98 -> Peer ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): Couldn't send request to peer ced21ca983b749d0b4d4f6a0e2426bf8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9fc39354ba064392be1af1769513f738. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:30.846247 30894 raft_consensus.cc:1261] T 9fc39354ba064392be1af1769513f738 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:30.846942 3455 consensus_queue.cc:1035] T 9fc39354ba064392be1af1769513f738 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:30.847472 3316 consensus_queue.cc:1035] T 9fc39354ba064392be1af1769513f738 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:30.851730 3538 ts_tablet_manager.cc:902] T 8425cfe5c6134a60852e12f9af55227b P aae860d7ae2b4002bf7794c278f9cc56: Initiating tablet copy from peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:13:30.852057 3316 raft_consensus.cc:2932] T 9fc39354ba064392be1af1769513f738 P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:13:30.856499 30894 raft_consensus.cc:2932] T 9fc39354ba064392be1af1769513f738 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:13:30.864140 30762 raft_consensus.cc:2932] T 9fc39354ba064392be1af1769513f738 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:13:30.864746 27857 catalog_manager.cc:5484] T 9fc39354ba064392be1af1769513f738 P b00f244c69df401997ea99d492218a98 reported cstate change: config changed from index -1 to 2, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) added. New cstate: current_term: 1 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:13:30.871343 3536 ts_tablet_manager.cc:902] T cdad6ef53b6f413ebfda67f9b42c9c23 P 6d1d7de19459435fbde1999e3dfd226b: Initiating tablet copy from peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:13:30.880532 3538 tablet_copy_client.cc:308] T 8425cfe5c6134a60852e12f9af55227b P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.68:46211
I20240503 08:13:30.881188 28413 tablet_copy_service.cc:140] P d09a11a03989421b9f28019dc1c14cc2: Received BeginTabletCopySession request for tablet 8425cfe5c6134a60852e12f9af55227b from peer aae860d7ae2b4002bf7794c278f9cc56 ({username='slave'} at 127.20.230.67:39219)
I20240503 08:13:30.881338 28413 tablet_copy_service.cc:161] P d09a11a03989421b9f28019dc1c14cc2: Beginning new tablet copy session on tablet 8425cfe5c6134a60852e12f9af55227b from peer aae860d7ae2b4002bf7794c278f9cc56 at {username='slave'} at 127.20.230.67:39219: session id = aae860d7ae2b4002bf7794c278f9cc56-8425cfe5c6134a60852e12f9af55227b
I20240503 08:13:30.884037 28413 tablet_copy_source_session.cc:205] T 8425cfe5c6134a60852e12f9af55227b P d09a11a03989421b9f28019dc1c14cc2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:30.885557 3538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8425cfe5c6134a60852e12f9af55227b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:30.889825 3549 ts_tablet_manager.cc:902] T f53acf725b62447382e9bae95e6e74d5 P aae860d7ae2b4002bf7794c278f9cc56: Initiating tablet copy from peer ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:13:30.893687 3538 tablet_copy_client.cc:783] T 8425cfe5c6134a60852e12f9af55227b P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:30.893965 3538 tablet_copy_client.cc:647] T 8425cfe5c6134a60852e12f9af55227b P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:30.895537 3536 tablet_copy_client.cc:308] T cdad6ef53b6f413ebfda67f9b42c9c23 P 6d1d7de19459435fbde1999e3dfd226b: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.67:39919
I20240503 08:13:30.896010 3538 tablet_copy_client.cc:523] T 8425cfe5c6134a60852e12f9af55227b P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:30.896113 28283 tablet_copy_service.cc:140] P aae860d7ae2b4002bf7794c278f9cc56: Received BeginTabletCopySession request for tablet cdad6ef53b6f413ebfda67f9b42c9c23 from peer 6d1d7de19459435fbde1999e3dfd226b ({username='slave'} at 127.20.230.69:54309)
I20240503 08:13:30.896311 28283 tablet_copy_service.cc:161] P aae860d7ae2b4002bf7794c278f9cc56: Beginning new tablet copy session on tablet cdad6ef53b6f413ebfda67f9b42c9c23 from peer 6d1d7de19459435fbde1999e3dfd226b at {username='slave'} at 127.20.230.69:54309: session id = 6d1d7de19459435fbde1999e3dfd226b-cdad6ef53b6f413ebfda67f9b42c9c23
I20240503 08:13:30.900370 3538 tablet_bootstrap.cc:492] T 8425cfe5c6134a60852e12f9af55227b P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:13:30.900827 28283 tablet_copy_source_session.cc:205] T cdad6ef53b6f413ebfda67f9b42c9c23 P aae860d7ae2b4002bf7794c278f9cc56: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:30.902326 3536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdad6ef53b6f413ebfda67f9b42c9c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:30.908749 3538 tablet_bootstrap.cc:492] T 8425cfe5c6134a60852e12f9af55227b P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:30.909361 3538 tablet_bootstrap.cc:492] T 8425cfe5c6134a60852e12f9af55227b P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap complete.
I20240503 08:13:30.909581 3538 ts_tablet_manager.cc:1367] T 8425cfe5c6134a60852e12f9af55227b P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240503 08:13:30.912581 3538 raft_consensus.cc:348] T 8425cfe5c6134a60852e12f9af55227b P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:30.912808 3538 raft_consensus.cc:729] T 8425cfe5c6134a60852e12f9af55227b P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:13:30.913080 3538 consensus_queue.cc:260] T 8425cfe5c6134a60852e12f9af55227b P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:30.913614 3538 ts_tablet_manager.cc:1398] T 8425cfe5c6134a60852e12f9af55227b P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 08:13:30.914256 28413 tablet_copy_service.cc:342] P d09a11a03989421b9f28019dc1c14cc2: Request end of tablet copy session aae860d7ae2b4002bf7794c278f9cc56-8425cfe5c6134a60852e12f9af55227b received from {username='slave'} at 127.20.230.67:39219
I20240503 08:13:30.914366 28413 tablet_copy_service.cc:433] P d09a11a03989421b9f28019dc1c14cc2: ending tablet copy session aae860d7ae2b4002bf7794c278f9cc56-8425cfe5c6134a60852e12f9af55227b on tablet 8425cfe5c6134a60852e12f9af55227b with peer aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:13:30.915446 3561 ts_tablet_manager.cc:902] T b12ceaf7f8d04fad8b07b36459da7116 P d09a11a03989421b9f28019dc1c14cc2: Initiating tablet copy from peer ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:13:30.920851 3536 tablet_copy_client.cc:783] T cdad6ef53b6f413ebfda67f9b42c9c23 P 6d1d7de19459435fbde1999e3dfd226b: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:30.921070 3536 tablet_copy_client.cc:647] T cdad6ef53b6f413ebfda67f9b42c9c23 P 6d1d7de19459435fbde1999e3dfd226b: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:30.923765 3536 tablet_copy_client.cc:523] T cdad6ef53b6f413ebfda67f9b42c9c23 P 6d1d7de19459435fbde1999e3dfd226b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:30.927321 3536 tablet_bootstrap.cc:492] T cdad6ef53b6f413ebfda67f9b42c9c23 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:13:30.941900 3561 tablet_copy_client.cc:308] T b12ceaf7f8d04fad8b07b36459da7116 P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.71:41915
I20240503 08:13:30.942629 31048 tablet_copy_service.cc:140] P ced21ca983b749d0b4d4f6a0e2426bf8: Received BeginTabletCopySession request for tablet b12ceaf7f8d04fad8b07b36459da7116 from peer d09a11a03989421b9f28019dc1c14cc2 ({username='slave'} at 127.20.230.68:37753)
I20240503 08:13:30.942781 31048 tablet_copy_service.cc:161] P ced21ca983b749d0b4d4f6a0e2426bf8: Beginning new tablet copy session on tablet b12ceaf7f8d04fad8b07b36459da7116 from peer d09a11a03989421b9f28019dc1c14cc2 at {username='slave'} at 127.20.230.68:37753: session id = d09a11a03989421b9f28019dc1c14cc2-b12ceaf7f8d04fad8b07b36459da7116
I20240503 08:13:30.948468 31048 tablet_copy_source_session.cc:205] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:30.949905 3561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b12ceaf7f8d04fad8b07b36459da7116. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:30.954041 3549 tablet_copy_client.cc:308] T f53acf725b62447382e9bae95e6e74d5 P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.70:33747
I20240503 08:13:30.954659 30915 tablet_copy_service.cc:140] P ce1496a47d084365ad786cd09e472a2c: Received BeginTabletCopySession request for tablet f53acf725b62447382e9bae95e6e74d5 from peer aae860d7ae2b4002bf7794c278f9cc56 ({username='slave'} at 127.20.230.67:50141)
I20240503 08:13:30.954803 30915 tablet_copy_service.cc:161] P ce1496a47d084365ad786cd09e472a2c: Beginning new tablet copy session on tablet f53acf725b62447382e9bae95e6e74d5 from peer aae860d7ae2b4002bf7794c278f9cc56 at {username='slave'} at 127.20.230.67:50141: session id = aae860d7ae2b4002bf7794c278f9cc56-f53acf725b62447382e9bae95e6e74d5
I20240503 08:13:30.956416 3561 tablet_copy_client.cc:783] T b12ceaf7f8d04fad8b07b36459da7116 P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:30.956568 3561 tablet_copy_client.cc:647] T b12ceaf7f8d04fad8b07b36459da7116 P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:30.957202 30915 tablet_copy_source_session.cc:205] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:30.958084 3561 tablet_copy_client.cc:523] T b12ceaf7f8d04fad8b07b36459da7116 P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:30.961084 3561 tablet_bootstrap.cc:492] T b12ceaf7f8d04fad8b07b36459da7116 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:13:30.962090 3549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f53acf725b62447382e9bae95e6e74d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:30.971782 3549 tablet_copy_client.cc:783] T f53acf725b62447382e9bae95e6e74d5 P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:30.971978 3549 tablet_copy_client.cc:647] T f53acf725b62447382e9bae95e6e74d5 P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:30.986188 3549 tablet_copy_client.cc:523] T f53acf725b62447382e9bae95e6e74d5 P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:30.989456 3549 tablet_bootstrap.cc:492] T f53acf725b62447382e9bae95e6e74d5 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:13:30.993389 3536 tablet_bootstrap.cc:492] T cdad6ef53b6f413ebfda67f9b42c9c23 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:30.993724 3536 tablet_bootstrap.cc:492] T cdad6ef53b6f413ebfda67f9b42c9c23 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap complete.
I20240503 08:13:30.993964 3536 ts_tablet_manager.cc:1367] T cdad6ef53b6f413ebfda67f9b42c9c23 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.067s user 0.055s sys 0.000s
I20240503 08:13:30.994621 3536 raft_consensus.cc:348] T cdad6ef53b6f413ebfda67f9b42c9c23 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:30.994827 3536 raft_consensus.cc:729] T cdad6ef53b6f413ebfda67f9b42c9c23 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:13:30.995016 3536 consensus_queue.cc:260] T cdad6ef53b6f413ebfda67f9b42c9c23 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:30.995535 3536 ts_tablet_manager.cc:1398] T cdad6ef53b6f413ebfda67f9b42c9c23 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 08:13:30.996232 28283 tablet_copy_service.cc:342] P aae860d7ae2b4002bf7794c278f9cc56: Request end of tablet copy session 6d1d7de19459435fbde1999e3dfd226b-cdad6ef53b6f413ebfda67f9b42c9c23 received from {username='slave'} at 127.20.230.69:54309
I20240503 08:13:30.996330 28283 tablet_copy_service.cc:433] P aae860d7ae2b4002bf7794c278f9cc56: ending tablet copy session 6d1d7de19459435fbde1999e3dfd226b-cdad6ef53b6f413ebfda67f9b42c9c23 on tablet cdad6ef53b6f413ebfda67f9b42c9c23 with peer 6d1d7de19459435fbde1999e3dfd226b
I20240503 08:13:31.008708 3561 tablet_bootstrap.cc:492] T b12ceaf7f8d04fad8b07b36459da7116 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:31.009023 3561 tablet_bootstrap.cc:492] T b12ceaf7f8d04fad8b07b36459da7116 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap complete.
I20240503 08:13:31.009220 3561 ts_tablet_manager.cc:1367] T b12ceaf7f8d04fad8b07b36459da7116 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.048s user 0.014s sys 0.010s
I20240503 08:13:31.009847 3561 raft_consensus.cc:348] T b12ceaf7f8d04fad8b07b36459da7116 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:31.010051 3561 raft_consensus.cc:729] T b12ceaf7f8d04fad8b07b36459da7116 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:13:31.012758 3561 consensus_queue.cc:260] T b12ceaf7f8d04fad8b07b36459da7116 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:31.013757 3561 ts_tablet_manager.cc:1398] T b12ceaf7f8d04fad8b07b36459da7116 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:13:31.014856 31048 tablet_copy_service.cc:342] P ced21ca983b749d0b4d4f6a0e2426bf8: Request end of tablet copy session d09a11a03989421b9f28019dc1c14cc2-b12ceaf7f8d04fad8b07b36459da7116 received from {username='slave'} at 127.20.230.68:37753
I20240503 08:13:31.014969 31048 tablet_copy_service.cc:433] P ced21ca983b749d0b4d4f6a0e2426bf8: ending tablet copy session d09a11a03989421b9f28019dc1c14cc2-b12ceaf7f8d04fad8b07b36459da7116 on tablet b12ceaf7f8d04fad8b07b36459da7116 with peer d09a11a03989421b9f28019dc1c14cc2
I20240503 08:13:31.054077 3579 ts_tablet_manager.cc:902] T 4a23c8186c4d4b23b62d7474088b45c1 P ce1496a47d084365ad786cd09e472a2c: Initiating tablet copy from peer b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:13:31.056857 3549 tablet_bootstrap.cc:492] T f53acf725b62447382e9bae95e6e74d5 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:31.057227 3549 tablet_bootstrap.cc:492] T f53acf725b62447382e9bae95e6e74d5 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap complete.
I20240503 08:13:31.057471 3549 ts_tablet_manager.cc:1367] T f53acf725b62447382e9bae95e6e74d5 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.068s user 0.022s sys 0.017s
I20240503 08:13:31.057996 3579 tablet_copy_client.cc:308] T 4a23c8186c4d4b23b62d7474088b45c1 P ce1496a47d084365ad786cd09e472a2c: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.72:40893
I20240503 08:13:31.058143 3549 raft_consensus.cc:348] T f53acf725b62447382e9bae95e6e74d5 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:31.058331 3549 raft_consensus.cc:729] T f53acf725b62447382e9bae95e6e74d5 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:13:31.058617 31181 tablet_copy_service.cc:140] P b00f244c69df401997ea99d492218a98: Received BeginTabletCopySession request for tablet 4a23c8186c4d4b23b62d7474088b45c1 from peer ce1496a47d084365ad786cd09e472a2c ({username='slave'} at 127.20.230.70:40787)
I20240503 08:13:31.058744 31181 tablet_copy_service.cc:161] P b00f244c69df401997ea99d492218a98: Beginning new tablet copy session on tablet 4a23c8186c4d4b23b62d7474088b45c1 from peer ce1496a47d084365ad786cd09e472a2c at {username='slave'} at 127.20.230.70:40787: session id = ce1496a47d084365ad786cd09e472a2c-4a23c8186c4d4b23b62d7474088b45c1
I20240503 08:13:31.061424 31181 tablet_copy_source_session.cc:205] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:31.061967 3549 consensus_queue.cc:260] T f53acf725b62447382e9bae95e6e74d5 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:31.062906 31027 consensus_queue.cc:237] T 4ed8026daee840c6b31ca7d989ed25f0 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:31.063966 3549 ts_tablet_manager.cc:1398] T f53acf725b62447382e9bae95e6e74d5 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240503 08:13:31.064378 3579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a23c8186c4d4b23b62d7474088b45c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:31.065944 30915 tablet_copy_service.cc:342] P ce1496a47d084365ad786cd09e472a2c: Request end of tablet copy session aae860d7ae2b4002bf7794c278f9cc56-f53acf725b62447382e9bae95e6e74d5 received from {username='slave'} at 127.20.230.67:50141
I20240503 08:13:31.066040 30915 tablet_copy_service.cc:433] P ce1496a47d084365ad786cd09e472a2c: ending tablet copy session aae860d7ae2b4002bf7794c278f9cc56-f53acf725b62447382e9bae95e6e74d5 on tablet f53acf725b62447382e9bae95e6e74d5 with peer aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:13:31.067291 28001 raft_consensus.cc:1261] T 4ed8026daee840c6b31ca7d989ed25f0 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer ced21ca983b749d0b4d4f6a0e2426bf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:13:31.067649 30962 consensus_peers.cc:485] T 4ed8026daee840c6b31ca7d989ed25f0 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): Couldn't send request to peer d09a11a03989421b9f28019dc1c14cc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4ed8026daee840c6b31ca7d989ed25f0. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:31.068640 3319 consensus_queue.cc:1035] T 4ed8026daee840c6b31ca7d989ed25f0 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:31.068714 28133 raft_consensus.cc:1261] T 4ed8026daee840c6b31ca7d989ed25f0 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer ced21ca983b749d0b4d4f6a0e2426bf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:31.071000 3319 consensus_queue.cc:1035] T 4ed8026daee840c6b31ca7d989ed25f0 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:31.074270 3167 raft_consensus.cc:2932] T 4ed8026daee840c6b31ca7d989ed25f0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:31.074806 28001 raft_consensus.cc:2932] T 4ed8026daee840c6b31ca7d989ed25f0 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:31.078258 3579 tablet_copy_client.cc:783] T 4a23c8186c4d4b23b62d7474088b45c1 P ce1496a47d084365ad786cd09e472a2c: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:31.078500 3579 tablet_copy_client.cc:647] T 4a23c8186c4d4b23b62d7474088b45c1 P ce1496a47d084365ad786cd09e472a2c: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:31.081955 28133 raft_consensus.cc:2932] T 4ed8026daee840c6b31ca7d989ed25f0 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:31.088565 27857 catalog_manager.cc:5484] T 4ed8026daee840c6b31ca7d989ed25f0 P ced21ca983b749d0b4d4f6a0e2426bf8 reported cstate change: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New cstate: current_term: 1 leader_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:31.090240 3579 tablet_copy_client.cc:523] T 4a23c8186c4d4b23b62d7474088b45c1 P ce1496a47d084365ad786cd09e472a2c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:31.099133 3561 ts_tablet_manager.cc:902] T df31bd9e544a44f280db2b0d021000c2 P d09a11a03989421b9f28019dc1c14cc2: Initiating tablet copy from peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:13:31.106163 3579 tablet_bootstrap.cc:492] T 4a23c8186c4d4b23b62d7474088b45c1 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:13:31.129169 3579 tablet_bootstrap.cc:492] T 4a23c8186c4d4b23b62d7474088b45c1 P ce1496a47d084365ad786cd09e472a2c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:31.129578 3579 tablet_bootstrap.cc:492] T 4a23c8186c4d4b23b62d7474088b45c1 P ce1496a47d084365ad786cd09e472a2c: Bootstrap complete.
I20240503 08:13:31.129873 3579 ts_tablet_manager.cc:1367] T 4a23c8186c4d4b23b62d7474088b45c1 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.007s
I20240503 08:13:31.130628 3579 raft_consensus.cc:348] T 4a23c8186c4d4b23b62d7474088b45c1 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:31.130906 3579 raft_consensus.cc:729] T 4a23c8186c4d4b23b62d7474088b45c1 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:13:31.131153 3579 consensus_queue.cc:260] T 4a23c8186c4d4b23b62d7474088b45c1 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:31.131778 3579 ts_tablet_manager.cc:1398] T 4a23c8186c4d4b23b62d7474088b45c1 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:13:31.132472 31181 tablet_copy_service.cc:342] P b00f244c69df401997ea99d492218a98: Request end of tablet copy session ce1496a47d084365ad786cd09e472a2c-4a23c8186c4d4b23b62d7474088b45c1 received from {username='slave'} at 127.20.230.70:40787
I20240503 08:13:31.132588 31181 tablet_copy_service.cc:433] P b00f244c69df401997ea99d492218a98: ending tablet copy session ce1496a47d084365ad786cd09e472a2c-4a23c8186c4d4b23b62d7474088b45c1 on tablet 4a23c8186c4d4b23b62d7474088b45c1 with peer ce1496a47d084365ad786cd09e472a2c
I20240503 08:13:31.134912 3561 tablet_copy_client.cc:308] T df31bd9e544a44f280db2b0d021000c2 P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.65:39081
I20240503 08:13:31.135509 28023 tablet_copy_service.cc:140] P 833582ff68c94269be364ebd187a75db: Received BeginTabletCopySession request for tablet df31bd9e544a44f280db2b0d021000c2 from peer d09a11a03989421b9f28019dc1c14cc2 ({username='slave'} at 127.20.230.68:42231)
I20240503 08:13:31.135622 28023 tablet_copy_service.cc:161] P 833582ff68c94269be364ebd187a75db: Beginning new tablet copy session on tablet df31bd9e544a44f280db2b0d021000c2 from peer d09a11a03989421b9f28019dc1c14cc2 at {username='slave'} at 127.20.230.68:42231: session id = d09a11a03989421b9f28019dc1c14cc2-df31bd9e544a44f280db2b0d021000c2
I20240503 08:13:31.138254 28023 tablet_copy_source_session.cc:205] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:31.141373 3561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df31bd9e544a44f280db2b0d021000c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:31.149583 3561 tablet_copy_client.cc:783] T df31bd9e544a44f280db2b0d021000c2 P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:31.149823 3561 tablet_copy_client.cc:647] T df31bd9e544a44f280db2b0d021000c2 P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:31.154891 3561 tablet_copy_client.cc:523] T df31bd9e544a44f280db2b0d021000c2 P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:31.158300 3561 tablet_bootstrap.cc:492] T df31bd9e544a44f280db2b0d021000c2 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:13:31.164692 3561 tablet_bootstrap.cc:492] T df31bd9e544a44f280db2b0d021000c2 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:31.165014 3561 tablet_bootstrap.cc:492] T df31bd9e544a44f280db2b0d021000c2 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap complete.
I20240503 08:13:31.165210 3561 ts_tablet_manager.cc:1367] T df31bd9e544a44f280db2b0d021000c2 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20240503 08:13:31.165793 3561 raft_consensus.cc:348] T df31bd9e544a44f280db2b0d021000c2 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:31.165978 3561 raft_consensus.cc:729] T df31bd9e544a44f280db2b0d021000c2 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:13:31.166158 3561 consensus_queue.cc:260] T df31bd9e544a44f280db2b0d021000c2 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:31.166644 3561 ts_tablet_manager.cc:1398] T df31bd9e544a44f280db2b0d021000c2 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 08:13:31.167205 28023 tablet_copy_service.cc:342] P 833582ff68c94269be364ebd187a75db: Request end of tablet copy session d09a11a03989421b9f28019dc1c14cc2-df31bd9e544a44f280db2b0d021000c2 received from {username='slave'} at 127.20.230.68:42231
I20240503 08:13:31.167301 28023 tablet_copy_service.cc:433] P 833582ff68c94269be364ebd187a75db: ending tablet copy session d09a11a03989421b9f28019dc1c14cc2-df31bd9e544a44f280db2b0d021000c2 on tablet df31bd9e544a44f280db2b0d021000c2 with peer d09a11a03989421b9f28019dc1c14cc2
I20240503 08:13:31.184684 28390 consensus_queue.cc:237] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
W20240503 08:13:31.186900 28326 consensus_peers.cc:485] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2 -> Peer b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): Couldn't send request to peer b00f244c69df401997ea99d492218a98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 599cb18c58a74e93a3e0334cc43a73d4. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:31.186929 28131 raft_consensus.cc:1261] T 599cb18c58a74e93a3e0334cc43a73d4 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:31.187063 28262 raft_consensus.cc:1261] T 599cb18c58a74e93a3e0334cc43a73d4 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:31.187378 3437 consensus_queue.cc:1035] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:31.187638 3003 consensus_queue.cc:1035] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:31.190541 3003 raft_consensus.cc:2932] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:13:31.191283 28262 raft_consensus.cc:2932] T 599cb18c58a74e93a3e0334cc43a73d4 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:13:31.200037 28131 raft_consensus.cc:2932] T 599cb18c58a74e93a3e0334cc43a73d4 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:13:31.201349 27862 catalog_manager.cc:5484] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: config changed from index -1 to 4, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) added. New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:13:31.235666 31159 consensus_queue.cc:237] T b40a43ad7ed44c30a070189ebfe5aab6 P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:31.237999 31027 raft_consensus.cc:1261] T b40a43ad7ed44c30a070189ebfe5aab6 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:13:31.238442 31096 consensus_peers.cc:485] T b40a43ad7ed44c30a070189ebfe5aab6 P b00f244c69df401997ea99d492218a98 -> Peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): Couldn't send request to peer 6d1d7de19459435fbde1999e3dfd226b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b40a43ad7ed44c30a070189ebfe5aab6. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:31.238523 2924 consensus_queue.cc:1035] T b40a43ad7ed44c30a070189ebfe5aab6 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:31.238973 28003 raft_consensus.cc:1261] T b40a43ad7ed44c30a070189ebfe5aab6 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:31.240016 3316 consensus_queue.cc:1035] T b40a43ad7ed44c30a070189ebfe5aab6 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:31.242626 3316 raft_consensus.cc:2932] T b40a43ad7ed44c30a070189ebfe5aab6 P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:31.242970 28001 raft_consensus.cc:2932] T b40a43ad7ed44c30a070189ebfe5aab6 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:31.249614 27857 catalog_manager.cc:5484] T b40a43ad7ed44c30a070189ebfe5aab6 P 833582ff68c94269be364ebd187a75db reported cstate change: config changed from index -1 to 2, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) added. New cstate: current_term: 1 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:31.250984 31026 raft_consensus.cc:2932] T b40a43ad7ed44c30a070189ebfe5aab6 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:31.255034 3536 ts_tablet_manager.cc:902] T 08fb64977309479b831635344969c662 P 6d1d7de19459435fbde1999e3dfd226b: Initiating tablet copy from peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:13:31.262117 31159 raft_consensus.cc:1203] T ca38671a760d4534ab903579e0487772 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:31.263769 3630 ts_tablet_manager.cc:902] T 9fc39354ba064392be1af1769513f738 P ced21ca983b749d0b4d4f6a0e2426bf8: Initiating tablet copy from peer b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:13:31.270381 3630 tablet_copy_client.cc:308] T 9fc39354ba064392be1af1769513f738 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.72:40893
I20240503 08:13:31.270998 31181 tablet_copy_service.cc:140] P b00f244c69df401997ea99d492218a98: Received BeginTabletCopySession request for tablet 9fc39354ba064392be1af1769513f738 from peer ced21ca983b749d0b4d4f6a0e2426bf8 ({username='slave'} at 127.20.230.71:59573)
I20240503 08:13:31.271117 31181 tablet_copy_service.cc:161] P b00f244c69df401997ea99d492218a98: Beginning new tablet copy session on tablet 9fc39354ba064392be1af1769513f738 from peer ced21ca983b749d0b4d4f6a0e2426bf8 at {username='slave'} at 127.20.230.71:59573: session id = ced21ca983b749d0b4d4f6a0e2426bf8-9fc39354ba064392be1af1769513f738
I20240503 08:13:31.271268 3536 tablet_copy_client.cc:308] T 08fb64977309479b831635344969c662 P 6d1d7de19459435fbde1999e3dfd226b: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.67:39919
I20240503 08:13:31.271837 28283 tablet_copy_service.cc:140] P aae860d7ae2b4002bf7794c278f9cc56: Received BeginTabletCopySession request for tablet 08fb64977309479b831635344969c662 from peer 6d1d7de19459435fbde1999e3dfd226b ({username='slave'} at 127.20.230.69:54309)
I20240503 08:13:31.271940 28283 tablet_copy_service.cc:161] P aae860d7ae2b4002bf7794c278f9cc56: Beginning new tablet copy session on tablet 08fb64977309479b831635344969c662 from peer 6d1d7de19459435fbde1999e3dfd226b at {username='slave'} at 127.20.230.69:54309: session id = 6d1d7de19459435fbde1999e3dfd226b-08fb64977309479b831635344969c662
I20240503 08:13:31.273595 31181 tablet_copy_source_session.cc:205] T 9fc39354ba064392be1af1769513f738 P b00f244c69df401997ea99d492218a98: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:31.273885 28283 tablet_copy_source_session.cc:205] T 08fb64977309479b831635344969c662 P aae860d7ae2b4002bf7794c278f9cc56: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:31.274812 3630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fc39354ba064392be1af1769513f738. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:31.274961 3536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08fb64977309479b831635344969c662. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:31.279029 3536 tablet_copy_client.cc:783] T 08fb64977309479b831635344969c662 P 6d1d7de19459435fbde1999e3dfd226b: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:31.279196 3536 tablet_copy_client.cc:647] T 08fb64977309479b831635344969c662 P 6d1d7de19459435fbde1999e3dfd226b: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:31.281117 3630 tablet_copy_client.cc:783] T 9fc39354ba064392be1af1769513f738 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:31.281327 3630 tablet_copy_client.cc:647] T 9fc39354ba064392be1af1769513f738 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:31.283174 3630 tablet_copy_client.cc:523] T 9fc39354ba064392be1af1769513f738 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:31.283550 3536 tablet_copy_client.cc:523] T 08fb64977309479b831635344969c662 P 6d1d7de19459435fbde1999e3dfd226b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:31.286628 3630 tablet_bootstrap.cc:492] T 9fc39354ba064392be1af1769513f738 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:13:31.286664 3536 tablet_bootstrap.cc:492] T 08fb64977309479b831635344969c662 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:13:31.302057 3630 tablet_bootstrap.cc:492] T 9fc39354ba064392be1af1769513f738 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:31.302417 3630 tablet_bootstrap.cc:492] T 9fc39354ba064392be1af1769513f738 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap complete.
I20240503 08:13:31.302624 3630 ts_tablet_manager.cc:1367] T 9fc39354ba064392be1af1769513f738 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.005s
I20240503 08:13:31.303227 3630 raft_consensus.cc:348] T 9fc39354ba064392be1af1769513f738 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:31.303426 3630 raft_consensus.cc:729] T 9fc39354ba064392be1af1769513f738 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:13:31.303614 3630 consensus_queue.cc:260] T 9fc39354ba064392be1af1769513f738 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:31.304116 3630 ts_tablet_manager.cc:1398] T 9fc39354ba064392be1af1769513f738 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:13:31.304741 31181 tablet_copy_service.cc:342] P b00f244c69df401997ea99d492218a98: Request end of tablet copy session ced21ca983b749d0b4d4f6a0e2426bf8-9fc39354ba064392be1af1769513f738 received from {username='slave'} at 127.20.230.71:59573
I20240503 08:13:31.304826 31181 tablet_copy_service.cc:433] P b00f244c69df401997ea99d492218a98: ending tablet copy session ced21ca983b749d0b4d4f6a0e2426bf8-9fc39354ba064392be1af1769513f738 on tablet 9fc39354ba064392be1af1769513f738 with peer ced21ca983b749d0b4d4f6a0e2426bf8
I20240503 08:13:31.306123 3549 ts_tablet_manager.cc:902] T 17c6e3293aad4591a9c1395ebffff2e6 P aae860d7ae2b4002bf7794c278f9cc56: Initiating tablet copy from peer b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:13:31.325243 3549 tablet_copy_client.cc:308] T 17c6e3293aad4591a9c1395ebffff2e6 P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.72:40893
I20240503 08:13:31.325948 31181 tablet_copy_service.cc:140] P b00f244c69df401997ea99d492218a98: Received BeginTabletCopySession request for tablet 17c6e3293aad4591a9c1395ebffff2e6 from peer aae860d7ae2b4002bf7794c278f9cc56 ({username='slave'} at 127.20.230.67:45737)
I20240503 08:13:31.326071 31181 tablet_copy_service.cc:161] P b00f244c69df401997ea99d492218a98: Beginning new tablet copy session on tablet 17c6e3293aad4591a9c1395ebffff2e6 from peer aae860d7ae2b4002bf7794c278f9cc56 at {username='slave'} at 127.20.230.67:45737: session id = aae860d7ae2b4002bf7794c278f9cc56-17c6e3293aad4591a9c1395ebffff2e6
I20240503 08:13:31.328408 31181 tablet_copy_source_session.cc:205] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:31.330133 3549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17c6e3293aad4591a9c1395ebffff2e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:31.336357 3549 tablet_copy_client.cc:783] T 17c6e3293aad4591a9c1395ebffff2e6 P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:31.336701 3549 tablet_copy_client.cc:647] T 17c6e3293aad4591a9c1395ebffff2e6 P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:31.338145 3549 tablet_copy_client.cc:523] T 17c6e3293aad4591a9c1395ebffff2e6 P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:31.342353 3549 tablet_bootstrap.cc:492] T 17c6e3293aad4591a9c1395ebffff2e6 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:13:31.342262 30894 consensus_queue.cc:237] T 62ca502a089f4a37999d2252776b741b P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:31.344870 28001 raft_consensus.cc:1261] T 62ca502a089f4a37999d2252776b741b P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:31.344882 28261 raft_consensus.cc:1261] T 62ca502a089f4a37999d2252776b741b P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:31.346315 3544 consensus_queue.cc:1035] T 62ca502a089f4a37999d2252776b741b P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 08:13:31.346992 30829 consensus_peers.cc:485] T 62ca502a089f4a37999d2252776b741b P ce1496a47d084365ad786cd09e472a2c -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62ca502a089f4a37999d2252776b741b. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:31.347249 3164 consensus_queue.cc:1035] T 62ca502a089f4a37999d2252776b741b P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:31.353607 3536 tablet_bootstrap.cc:492] T 08fb64977309479b831635344969c662 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:31.354008 3536 tablet_bootstrap.cc:492] T 08fb64977309479b831635344969c662 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap complete.
I20240503 08:13:31.354239 3536 ts_tablet_manager.cc:1367] T 08fb64977309479b831635344969c662 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.068s user 0.041s sys 0.007s
I20240503 08:13:31.354349 28391 raft_consensus.cc:1203] T b12ceaf7f8d04fad8b07b36459da7116 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:31.356297 3536 raft_consensus.cc:348] T 08fb64977309479b831635344969c662 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:31.356541 3536 raft_consensus.cc:729] T 08fb64977309479b831635344969c662 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:13:31.356866 3549 tablet_bootstrap.cc:492] T 17c6e3293aad4591a9c1395ebffff2e6 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:31.356775 3536 consensus_queue.cc:260] T 08fb64977309479b831635344969c662 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:31.357162 3549 tablet_bootstrap.cc:492] T 17c6e3293aad4591a9c1395ebffff2e6 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap complete.
I20240503 08:13:31.357364 3549 ts_tablet_manager.cc:1367] T 17c6e3293aad4591a9c1395ebffff2e6 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20240503 08:13:31.358074 3549 raft_consensus.cc:348] T 17c6e3293aad4591a9c1395ebffff2e6 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:31.358251 3549 raft_consensus.cc:729] T 17c6e3293aad4591a9c1395ebffff2e6 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:13:31.358466 3549 consensus_queue.cc:260] T 17c6e3293aad4591a9c1395ebffff2e6 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:31.359110 3549 ts_tablet_manager.cc:1398] T 17c6e3293aad4591a9c1395ebffff2e6 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:31.359920 3536 ts_tablet_manager.cc:1398] T 08fb64977309479b831635344969c662 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240503 08:13:31.360436 28283 tablet_copy_service.cc:342] P aae860d7ae2b4002bf7794c278f9cc56: Request end of tablet copy session 6d1d7de19459435fbde1999e3dfd226b-08fb64977309479b831635344969c662 received from {username='slave'} at 127.20.230.69:54309
I20240503 08:13:31.360553 28283 tablet_copy_service.cc:433] P aae860d7ae2b4002bf7794c278f9cc56: ending tablet copy session 6d1d7de19459435fbde1999e3dfd226b-08fb64977309479b831635344969c662 on tablet 08fb64977309479b831635344969c662 with peer 6d1d7de19459435fbde1999e3dfd226b
I20240503 08:13:31.360579 31181 tablet_copy_service.cc:342] P b00f244c69df401997ea99d492218a98: Request end of tablet copy session aae860d7ae2b4002bf7794c278f9cc56-17c6e3293aad4591a9c1395ebffff2e6 received from {username='slave'} at 127.20.230.67:45737
I20240503 08:13:31.360654 31181 tablet_copy_service.cc:433] P b00f244c69df401997ea99d492218a98: ending tablet copy session aae860d7ae2b4002bf7794c278f9cc56-17c6e3293aad4591a9c1395ebffff2e6 on tablet 17c6e3293aad4591a9c1395ebffff2e6 with peer aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:13:31.363070 28263 raft_consensus.cc:1203] T f53acf725b62447382e9bae95e6e74d5 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:31.366966 28263 raft_consensus.cc:1203] T 8425cfe5c6134a60852e12f9af55227b P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:31.372499 3544 raft_consensus.cc:2932] T 62ca502a089f4a37999d2252776b741b P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:31.373018 28262 raft_consensus.cc:2932] T 62ca502a089f4a37999d2252776b741b P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:31.376192 28001 raft_consensus.cc:2932] T 62ca502a089f4a37999d2252776b741b P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:31.379544 27862 catalog_manager.cc:5484] T 62ca502a089f4a37999d2252776b741b P ce1496a47d084365ad786cd09e472a2c reported cstate change: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New cstate: current_term: 1 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:31.406749 30762 raft_consensus.cc:1203] T cdad6ef53b6f413ebfda67f9b42c9c23 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:31.417348 30894 consensus_queue.cc:237] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:31.419245 30762 raft_consensus.cc:1261] T f9746769e2514c65ac8d48ca7df8bca7 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:13:31.419709 30830 consensus_peers.cc:485] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c -> Peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Couldn't send request to peer 833582ff68c94269be364ebd187a75db. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f9746769e2514c65ac8d48ca7df8bca7. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:31.419803 3164 consensus_queue.cc:1035] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:31.419880 31159 raft_consensus.cc:1261] T f9746769e2514c65ac8d48ca7df8bca7 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:31.420406 2909 consensus_queue.cc:1035] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:31.422606 2909 raft_consensus.cc:2932] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:31.423151 31159 raft_consensus.cc:2932] T f9746769e2514c65ac8d48ca7df8bca7 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:31.427235 30762 raft_consensus.cc:2932] T f9746769e2514c65ac8d48ca7df8bca7 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:31.429047 27854 catalog_manager.cc:5484] T f9746769e2514c65ac8d48ca7df8bca7 P b00f244c69df401997ea99d492218a98 reported cstate change: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New cstate: current_term: 1 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:31.464658 28262 consensus_queue.cc:237] T 46d27fa12990405cae5f94976ac7a014 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:31.467422 28001 raft_consensus.cc:1261] T 46d27fa12990405cae5f94976ac7a014 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:13:31.467545 28196 consensus_peers.cc:485] T 46d27fa12990405cae5f94976ac7a014 P aae860d7ae2b4002bf7794c278f9cc56 -> Peer b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): Couldn't send request to peer b00f244c69df401997ea99d492218a98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 46d27fa12990405cae5f94976ac7a014. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:31.467525 28391 raft_consensus.cc:1261] T 46d27fa12990405cae5f94976ac7a014 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:31.467990 2492 consensus_queue.cc:1035] T 46d27fa12990405cae5f94976ac7a014 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:31.468215 3007 consensus_queue.cc:1035] T 46d27fa12990405cae5f94976ac7a014 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:31.471207 2492 raft_consensus.cc:2932] T 46d27fa12990405cae5f94976ac7a014 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:13:31.474099 28001 raft_consensus.cc:2932] T 46d27fa12990405cae5f94976ac7a014 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:13:31.474587 28391 raft_consensus.cc:2932] T 46d27fa12990405cae5f94976ac7a014 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:13:31.476585 27862 catalog_manager.cc:5484] T 46d27fa12990405cae5f94976ac7a014 P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: config changed from index -1 to 4, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) added. New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:13:31.578130 3561 ts_tablet_manager.cc:902] T 4ed8026daee840c6b31ca7d989ed25f0 P d09a11a03989421b9f28019dc1c14cc2: Initiating tablet copy from peer ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:13:31.579865 3561 tablet_copy_client.cc:308] T 4ed8026daee840c6b31ca7d989ed25f0 P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.71:41915
I20240503 08:13:31.580592 31048 tablet_copy_service.cc:140] P ced21ca983b749d0b4d4f6a0e2426bf8: Received BeginTabletCopySession request for tablet 4ed8026daee840c6b31ca7d989ed25f0 from peer d09a11a03989421b9f28019dc1c14cc2 ({username='slave'} at 127.20.230.68:37753)
I20240503 08:13:31.580729 31048 tablet_copy_service.cc:161] P ced21ca983b749d0b4d4f6a0e2426bf8: Beginning new tablet copy session on tablet 4ed8026daee840c6b31ca7d989ed25f0 from peer d09a11a03989421b9f28019dc1c14cc2 at {username='slave'} at 127.20.230.68:37753: session id = d09a11a03989421b9f28019dc1c14cc2-4ed8026daee840c6b31ca7d989ed25f0
I20240503 08:13:31.583454 31048 tablet_copy_source_session.cc:205] T 4ed8026daee840c6b31ca7d989ed25f0 P ced21ca983b749d0b4d4f6a0e2426bf8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:31.584921 3561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ed8026daee840c6b31ca7d989ed25f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:31.590401 3561 tablet_copy_client.cc:783] T 4ed8026daee840c6b31ca7d989ed25f0 P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:31.590651 3561 tablet_copy_client.cc:647] T 4ed8026daee840c6b31ca7d989ed25f0 P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:31.592070 3561 tablet_copy_client.cc:523] T 4ed8026daee840c6b31ca7d989ed25f0 P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:31.594619 3561 tablet_bootstrap.cc:492] T 4ed8026daee840c6b31ca7d989ed25f0 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:13:31.599503 3561 tablet_bootstrap.cc:492] T 4ed8026daee840c6b31ca7d989ed25f0 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:31.599824 3561 tablet_bootstrap.cc:492] T 4ed8026daee840c6b31ca7d989ed25f0 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap complete.
I20240503 08:13:31.600553 3561 ts_tablet_manager.cc:1367] T 4ed8026daee840c6b31ca7d989ed25f0 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20240503 08:13:31.601085 3561 raft_consensus.cc:348] T 4ed8026daee840c6b31ca7d989ed25f0 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:31.601260 3561 raft_consensus.cc:729] T 4ed8026daee840c6b31ca7d989ed25f0 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:13:31.601441 3561 consensus_queue.cc:260] T 4ed8026daee840c6b31ca7d989ed25f0 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:31.601992 3561 ts_tablet_manager.cc:1398] T 4ed8026daee840c6b31ca7d989ed25f0 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 08:13:31.603056 31048 tablet_copy_service.cc:342] P ced21ca983b749d0b4d4f6a0e2426bf8: Request end of tablet copy session d09a11a03989421b9f28019dc1c14cc2-4ed8026daee840c6b31ca7d989ed25f0 received from {username='slave'} at 127.20.230.68:37753
I20240503 08:13:31.603156 31048 tablet_copy_service.cc:433] P ced21ca983b749d0b4d4f6a0e2426bf8: ending tablet copy session d09a11a03989421b9f28019dc1c14cc2-4ed8026daee840c6b31ca7d989ed25f0 on tablet 4ed8026daee840c6b31ca7d989ed25f0 with peer d09a11a03989421b9f28019dc1c14cc2
I20240503 08:13:31.615917 30894 raft_consensus.cc:1203] T 4a23c8186c4d4b23b62d7474088b45c1 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:31.639561 30761 raft_consensus.cc:1203] T 08fb64977309479b831635344969c662 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:31.660364 3737 ts_tablet_manager.cc:902] T 599cb18c58a74e93a3e0334cc43a73d4 P b00f244c69df401997ea99d492218a98: Initiating tablet copy from peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:13:31.665315 3536 ts_tablet_manager.cc:902] T b40a43ad7ed44c30a070189ebfe5aab6 P 6d1d7de19459435fbde1999e3dfd226b: Initiating tablet copy from peer b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:13:31.683099 28390 raft_consensus.cc:1203] T df31bd9e544a44f280db2b0d021000c2 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:31.697335 28262 raft_consensus.cc:1203] T 17c6e3293aad4591a9c1395ebffff2e6 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:31.747843 3737 tablet_copy_client.cc:308] T 599cb18c58a74e93a3e0334cc43a73d4 P b00f244c69df401997ea99d492218a98: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.68:46211
I20240503 08:13:31.749251 28413 tablet_copy_service.cc:140] P d09a11a03989421b9f28019dc1c14cc2: Received BeginTabletCopySession request for tablet 599cb18c58a74e93a3e0334cc43a73d4 from peer b00f244c69df401997ea99d492218a98 ({username='slave'} at 127.20.230.72:51109)
I20240503 08:13:31.751696 28413 tablet_copy_service.cc:161] P d09a11a03989421b9f28019dc1c14cc2: Beginning new tablet copy session on tablet 599cb18c58a74e93a3e0334cc43a73d4 from peer b00f244c69df401997ea99d492218a98 at {username='slave'} at 127.20.230.72:51109: session id = b00f244c69df401997ea99d492218a98-599cb18c58a74e93a3e0334cc43a73d4
I20240503 08:13:31.754376 28413 tablet_copy_source_session.cc:205] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:31.756134 3737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 599cb18c58a74e93a3e0334cc43a73d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:31.759142 3536 tablet_copy_client.cc:308] T b40a43ad7ed44c30a070189ebfe5aab6 P 6d1d7de19459435fbde1999e3dfd226b: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.72:40893
I20240503 08:13:31.760743 31181 tablet_copy_service.cc:140] P b00f244c69df401997ea99d492218a98: Received BeginTabletCopySession request for tablet b40a43ad7ed44c30a070189ebfe5aab6 from peer 6d1d7de19459435fbde1999e3dfd226b ({username='slave'} at 127.20.230.69:43821)
I20240503 08:13:31.760876 31181 tablet_copy_service.cc:161] P b00f244c69df401997ea99d492218a98: Beginning new tablet copy session on tablet b40a43ad7ed44c30a070189ebfe5aab6 from peer 6d1d7de19459435fbde1999e3dfd226b at {username='slave'} at 127.20.230.69:43821: session id = 6d1d7de19459435fbde1999e3dfd226b-b40a43ad7ed44c30a070189ebfe5aab6
I20240503 08:13:31.763406 31181 tablet_copy_source_session.cc:205] T b40a43ad7ed44c30a070189ebfe5aab6 P b00f244c69df401997ea99d492218a98: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:31.764190 3737 tablet_copy_client.cc:783] T 599cb18c58a74e93a3e0334cc43a73d4 P b00f244c69df401997ea99d492218a98: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:31.764461 3536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b40a43ad7ed44c30a070189ebfe5aab6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:31.764865 3737 tablet_copy_client.cc:647] T 599cb18c58a74e93a3e0334cc43a73d4 P b00f244c69df401997ea99d492218a98: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:31.766170 3737 tablet_copy_client.cc:523] T 599cb18c58a74e93a3e0334cc43a73d4 P b00f244c69df401997ea99d492218a98: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:31.772716 3737 tablet_bootstrap.cc:492] T 599cb18c58a74e93a3e0334cc43a73d4 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:13:31.776232 31026 raft_consensus.cc:1203] T 9fc39354ba064392be1af1769513f738 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:31.810925 3536 tablet_copy_client.cc:783] T b40a43ad7ed44c30a070189ebfe5aab6 P 6d1d7de19459435fbde1999e3dfd226b: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:31.811177 3536 tablet_copy_client.cc:647] T b40a43ad7ed44c30a070189ebfe5aab6 P 6d1d7de19459435fbde1999e3dfd226b: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:31.817636 3536 tablet_copy_client.cc:523] T b40a43ad7ed44c30a070189ebfe5aab6 P 6d1d7de19459435fbde1999e3dfd226b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:31.827082 3536 tablet_bootstrap.cc:492] T b40a43ad7ed44c30a070189ebfe5aab6 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:13:31.864141 3536 tablet_bootstrap.cc:492] T b40a43ad7ed44c30a070189ebfe5aab6 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:31.864516 3536 tablet_bootstrap.cc:492] T b40a43ad7ed44c30a070189ebfe5aab6 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap complete.
I20240503 08:13:31.864763 3536 ts_tablet_manager.cc:1367] T b40a43ad7ed44c30a070189ebfe5aab6 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.038s user 0.006s sys 0.000s
I20240503 08:13:31.865434 3536 raft_consensus.cc:348] T b40a43ad7ed44c30a070189ebfe5aab6 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:31.865662 3536 raft_consensus.cc:729] T b40a43ad7ed44c30a070189ebfe5aab6 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:13:31.867424 3737 tablet_bootstrap.cc:492] T 599cb18c58a74e93a3e0334cc43a73d4 P b00f244c69df401997ea99d492218a98: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:31.867798 3737 tablet_bootstrap.cc:492] T 599cb18c58a74e93a3e0334cc43a73d4 P b00f244c69df401997ea99d492218a98: Bootstrap complete.
I20240503 08:13:31.867764 3536 consensus_queue.cc:260] T b40a43ad7ed44c30a070189ebfe5aab6 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:31.868155 3737 ts_tablet_manager.cc:1367] T 599cb18c58a74e93a3e0334cc43a73d4 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.096s user 0.046s sys 0.015s
I20240503 08:13:31.868849 3737 raft_consensus.cc:348] T 599cb18c58a74e93a3e0334cc43a73d4 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:31.869050 3737 raft_consensus.cc:729] T 599cb18c58a74e93a3e0334cc43a73d4 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:13:31.869282 3737 consensus_queue.cc:260] T 599cb18c58a74e93a3e0334cc43a73d4 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:31.869911 3737 ts_tablet_manager.cc:1398] T 599cb18c58a74e93a3e0334cc43a73d4 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:13:31.870774 28413 tablet_copy_service.cc:342] P d09a11a03989421b9f28019dc1c14cc2: Request end of tablet copy session b00f244c69df401997ea99d492218a98-599cb18c58a74e93a3e0334cc43a73d4 received from {username='slave'} at 127.20.230.72:51109
I20240503 08:13:31.870869 28413 tablet_copy_service.cc:433] P d09a11a03989421b9f28019dc1c14cc2: ending tablet copy session b00f244c69df401997ea99d492218a98-599cb18c58a74e93a3e0334cc43a73d4 on tablet 599cb18c58a74e93a3e0334cc43a73d4 with peer b00f244c69df401997ea99d492218a98
I20240503 08:13:31.877710 3536 ts_tablet_manager.cc:1398] T b40a43ad7ed44c30a070189ebfe5aab6 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.013s user 0.000s sys 0.003s
I20240503 08:13:31.882325 3769 ts_tablet_manager.cc:902] T 46d27fa12990405cae5f94976ac7a014 P b00f244c69df401997ea99d492218a98: Initiating tablet copy from peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:13:31.888339 31181 tablet_copy_service.cc:342] P b00f244c69df401997ea99d492218a98: Request end of tablet copy session 6d1d7de19459435fbde1999e3dfd226b-b40a43ad7ed44c30a070189ebfe5aab6 received from {username='slave'} at 127.20.230.69:43821
I20240503 08:13:31.888440 31181 tablet_copy_service.cc:433] P b00f244c69df401997ea99d492218a98: ending tablet copy session 6d1d7de19459435fbde1999e3dfd226b-b40a43ad7ed44c30a070189ebfe5aab6 on tablet b40a43ad7ed44c30a070189ebfe5aab6 with peer 6d1d7de19459435fbde1999e3dfd226b
I20240503 08:13:31.905606 3765 ts_tablet_manager.cc:902] T f9746769e2514c65ac8d48ca7df8bca7 P 833582ff68c94269be364ebd187a75db: Initiating tablet copy from peer ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:13:31.914760 3769 tablet_copy_client.cc:308] T 46d27fa12990405cae5f94976ac7a014 P b00f244c69df401997ea99d492218a98: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.67:39919
I20240503 08:13:31.914959 3772 ts_tablet_manager.cc:902] T 62ca502a089f4a37999d2252776b741b P 1b411c6bb25549368041773d7b710d48: Initiating tablet copy from peer ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:13:31.915560 28283 tablet_copy_service.cc:140] P aae860d7ae2b4002bf7794c278f9cc56: Received BeginTabletCopySession request for tablet 46d27fa12990405cae5f94976ac7a014 from peer b00f244c69df401997ea99d492218a98 ({username='slave'} at 127.20.230.72:55725)
I20240503 08:13:31.915688 28283 tablet_copy_service.cc:161] P aae860d7ae2b4002bf7794c278f9cc56: Beginning new tablet copy session on tablet 46d27fa12990405cae5f94976ac7a014 from peer b00f244c69df401997ea99d492218a98 at {username='slave'} at 127.20.230.72:55725: session id = b00f244c69df401997ea99d492218a98-46d27fa12990405cae5f94976ac7a014
I20240503 08:13:31.917894 28283 tablet_copy_source_session.cc:205] T 46d27fa12990405cae5f94976ac7a014 P aae860d7ae2b4002bf7794c278f9cc56: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:31.924368 3769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d27fa12990405cae5f94976ac7a014. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:31.943470 3765 tablet_copy_client.cc:308] T f9746769e2514c65ac8d48ca7df8bca7 P 833582ff68c94269be364ebd187a75db: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.70:33747
I20240503 08:13:31.944128 30915 tablet_copy_service.cc:140] P ce1496a47d084365ad786cd09e472a2c: Received BeginTabletCopySession request for tablet f9746769e2514c65ac8d48ca7df8bca7 from peer 833582ff68c94269be364ebd187a75db ({username='slave'} at 127.20.230.65:60309)
I20240503 08:13:31.944268 30915 tablet_copy_service.cc:161] P ce1496a47d084365ad786cd09e472a2c: Beginning new tablet copy session on tablet f9746769e2514c65ac8d48ca7df8bca7 from peer 833582ff68c94269be364ebd187a75db at {username='slave'} at 127.20.230.65:60309: session id = 833582ff68c94269be364ebd187a75db-f9746769e2514c65ac8d48ca7df8bca7
I20240503 08:13:31.944591 3769 tablet_copy_client.cc:783] T 46d27fa12990405cae5f94976ac7a014 P b00f244c69df401997ea99d492218a98: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:31.944792 3769 tablet_copy_client.cc:647] T 46d27fa12990405cae5f94976ac7a014 P b00f244c69df401997ea99d492218a98: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:31.946219 3769 tablet_copy_client.cc:523] T 46d27fa12990405cae5f94976ac7a014 P b00f244c69df401997ea99d492218a98: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:31.949380 30915 tablet_copy_source_session.cc:205] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:31.950306 3769 tablet_bootstrap.cc:492] T 46d27fa12990405cae5f94976ac7a014 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:13:31.952297 3765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9746769e2514c65ac8d48ca7df8bca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:31.959911 3772 tablet_copy_client.cc:308] T 62ca502a089f4a37999d2252776b741b P 1b411c6bb25549368041773d7b710d48: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.70:33747
I20240503 08:13:31.960505 30915 tablet_copy_service.cc:140] P ce1496a47d084365ad786cd09e472a2c: Received BeginTabletCopySession request for tablet 62ca502a089f4a37999d2252776b741b from peer 1b411c6bb25549368041773d7b710d48 ({username='slave'} at 127.20.230.66:47541)
I20240503 08:13:31.960626 30915 tablet_copy_service.cc:161] P ce1496a47d084365ad786cd09e472a2c: Beginning new tablet copy session on tablet 62ca502a089f4a37999d2252776b741b from peer 1b411c6bb25549368041773d7b710d48 at {username='slave'} at 127.20.230.66:47541: session id = 1b411c6bb25549368041773d7b710d48-62ca502a089f4a37999d2252776b741b
I20240503 08:13:31.963518 30915 tablet_copy_source_session.cc:205] T 62ca502a089f4a37999d2252776b741b P ce1496a47d084365ad786cd09e472a2c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:31.965016 3765 tablet_copy_client.cc:783] T f9746769e2514c65ac8d48ca7df8bca7 P 833582ff68c94269be364ebd187a75db: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:31.965262 3765 tablet_copy_client.cc:647] T f9746769e2514c65ac8d48ca7df8bca7 P 833582ff68c94269be364ebd187a75db: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:31.966080 3772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62ca502a089f4a37999d2252776b741b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:31.966532 3765 tablet_copy_client.cc:523] T f9746769e2514c65ac8d48ca7df8bca7 P 833582ff68c94269be364ebd187a75db: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:31.969681 3765 tablet_bootstrap.cc:492] T f9746769e2514c65ac8d48ca7df8bca7 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:13:31.974877 3772 tablet_copy_client.cc:783] T 62ca502a089f4a37999d2252776b741b P 1b411c6bb25549368041773d7b710d48: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:31.975416 3772 tablet_copy_client.cc:647] T 62ca502a089f4a37999d2252776b741b P 1b411c6bb25549368041773d7b710d48: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:31.976971 3772 tablet_copy_client.cc:523] T 62ca502a089f4a37999d2252776b741b P 1b411c6bb25549368041773d7b710d48: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:31.980082 3772 tablet_bootstrap.cc:492] T 62ca502a089f4a37999d2252776b741b P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:13:31.982865 3765 tablet_bootstrap.cc:492] T f9746769e2514c65ac8d48ca7df8bca7 P 833582ff68c94269be364ebd187a75db: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:31.983250 3765 tablet_bootstrap.cc:492] T f9746769e2514c65ac8d48ca7df8bca7 P 833582ff68c94269be364ebd187a75db: Bootstrap complete.
I20240503 08:13:31.983508 3765 ts_tablet_manager.cc:1367] T f9746769e2514c65ac8d48ca7df8bca7 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20240503 08:13:31.984258 3765 raft_consensus.cc:348] T f9746769e2514c65ac8d48ca7df8bca7 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:31.984532 3765 raft_consensus.cc:729] T f9746769e2514c65ac8d48ca7df8bca7 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:13:31.984858 3765 consensus_queue.cc:260] T f9746769e2514c65ac8d48ca7df8bca7 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:31.985563 3765 ts_tablet_manager.cc:1398] T f9746769e2514c65ac8d48ca7df8bca7 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:31.986719 30915 tablet_copy_service.cc:342] P ce1496a47d084365ad786cd09e472a2c: Request end of tablet copy session 833582ff68c94269be364ebd187a75db-f9746769e2514c65ac8d48ca7df8bca7 received from {username='slave'} at 127.20.230.65:60309
I20240503 08:13:31.986835 30915 tablet_copy_service.cc:433] P ce1496a47d084365ad786cd09e472a2c: ending tablet copy session 833582ff68c94269be364ebd187a75db-f9746769e2514c65ac8d48ca7df8bca7 on tablet f9746769e2514c65ac8d48ca7df8bca7 with peer 833582ff68c94269be364ebd187a75db
I20240503 08:13:31.997983 3772 tablet_bootstrap.cc:492] T 62ca502a089f4a37999d2252776b741b P 1b411c6bb25549368041773d7b710d48: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:31.998355 3772 tablet_bootstrap.cc:492] T 62ca502a089f4a37999d2252776b741b P 1b411c6bb25549368041773d7b710d48: Bootstrap complete.
I20240503 08:13:31.998634 3772 ts_tablet_manager.cc:1367] T 62ca502a089f4a37999d2252776b741b P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.007s
I20240503 08:13:31.999289 3772 raft_consensus.cc:348] T 62ca502a089f4a37999d2252776b741b P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:31.999606 3772 raft_consensus.cc:729] T 62ca502a089f4a37999d2252776b741b P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:13:31.999908 3772 consensus_queue.cc:260] T 62ca502a089f4a37999d2252776b741b P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:32.000595 3772 ts_tablet_manager.cc:1398] T 62ca502a089f4a37999d2252776b741b P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:32.001348 30915 tablet_copy_service.cc:342] P ce1496a47d084365ad786cd09e472a2c: Request end of tablet copy session 1b411c6bb25549368041773d7b710d48-62ca502a089f4a37999d2252776b741b received from {username='slave'} at 127.20.230.66:47541
I20240503 08:13:32.001437 30915 tablet_copy_service.cc:433] P ce1496a47d084365ad786cd09e472a2c: ending tablet copy session 1b411c6bb25549368041773d7b710d48-62ca502a089f4a37999d2252776b741b on tablet 62ca502a089f4a37999d2252776b741b with peer 1b411c6bb25549368041773d7b710d48
I20240503 08:13:32.023993 30742 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 08:13:32.025317 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:32.051589 27983 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 08:13:32.085757 3769 tablet_bootstrap.cc:492] T 46d27fa12990405cae5f94976ac7a014 P b00f244c69df401997ea99d492218a98: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:32.087728 3769 tablet_bootstrap.cc:492] T 46d27fa12990405cae5f94976ac7a014 P b00f244c69df401997ea99d492218a98: Bootstrap complete.
I20240503 08:13:32.087982 3769 ts_tablet_manager.cc:1367] T 46d27fa12990405cae5f94976ac7a014 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.138s user 0.044s sys 0.005s
I20240503 08:13:32.088728 3769 raft_consensus.cc:348] T 46d27fa12990405cae5f94976ac7a014 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:32.088932 3769 raft_consensus.cc:729] T 46d27fa12990405cae5f94976ac7a014 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:13:32.089116 3769 consensus_queue.cc:260] T 46d27fa12990405cae5f94976ac7a014 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:32.096482 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:32.096268 3769 ts_tablet_manager.cc:1398] T 46d27fa12990405cae5f94976ac7a014 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240503 08:13:32.097282 28283 tablet_copy_service.cc:342] P aae860d7ae2b4002bf7794c278f9cc56: Request end of tablet copy session b00f244c69df401997ea99d492218a98-46d27fa12990405cae5f94976ac7a014 received from {username='slave'} at 127.20.230.72:55725
I20240503 08:13:32.097394 28283 tablet_copy_service.cc:433] P aae860d7ae2b4002bf7794c278f9cc56: ending tablet copy session b00f244c69df401997ea99d492218a98-46d27fa12990405cae5f94976ac7a014 on tablet 46d27fa12990405cae5f94976ac7a014 with peer b00f244c69df401997ea99d492218a98
I20240503 08:13:32.113519 28243 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:32.164979 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:32.192063 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:32.193123 28391 raft_consensus.cc:1203] T 4ed8026daee840c6b31ca7d989ed25f0 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:32.220925 30760 raft_consensus.cc:1203] T b40a43ad7ed44c30a070189ebfe5aab6 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:32.221441 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:32.224967 31159 raft_consensus.cc:1203] T 599cb18c58a74e93a3e0334cc43a73d4 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:32.270905 28001 raft_consensus.cc:1203] T f9746769e2514c65ac8d48ca7df8bca7 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:32.434881 31160 raft_consensus.cc:1203] T 46d27fa12990405cae5f94976ac7a014 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:32.441670 28133 raft_consensus.cc:1203] T 62ca502a089f4a37999d2252776b741b P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:32.565860 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:32.585584 30742 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 08:13:32.609826 28243 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:32.633363 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:32.684111 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:32.695290 27983 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 08:13:32.726884 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:32.786856 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:32.848513 30895 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f9746769e2514c65ac8d48ca7df8bca7"
dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46522
I20240503 08:13:32.848666 30895 raft_consensus.cc:595] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Received request to transfer leadership
I20240503 08:13:32.886883 2909 raft_consensus.cc:979] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c: : Instructing follower 6d1d7de19459435fbde1999e3dfd226b to start an election
I20240503 08:13:32.887056 3544 raft_consensus.cc:1067] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Signalling peer 6d1d7de19459435fbde1999e3dfd226b to start an election
I20240503 08:13:32.889632 30761 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f9746769e2514c65ac8d48ca7df8bca7"
dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
from {username='slave'} at 127.20.230.70:41691
I20240503 08:13:32.889904 30761 raft_consensus.cc:482] T f9746769e2514c65ac8d48ca7df8bca7 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:32.890017 30761 raft_consensus.cc:3037] T f9746769e2514c65ac8d48ca7df8bca7 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:32.892628 30761 raft_consensus.cc:504] T f9746769e2514c65ac8d48ca7df8bca7 P 6d1d7de19459435fbde1999e3dfd226b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:32.894551 31159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9746769e2514c65ac8d48ca7df8bca7" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:13:32.894752 31159 raft_consensus.cc:3037] T f9746769e2514c65ac8d48ca7df8bca7 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:32.894990 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9746769e2514c65ac8d48ca7df8bca7" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:13:32.895164 30895 raft_consensus.cc:3032] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:32.895229 30895 raft_consensus.cc:729] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Running, Role: LEADER
I20240503 08:13:32.895447 30895 consensus_queue.cc:260] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:32.895838 30895 raft_consensus.cc:3037] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:32.896838 31159 raft_consensus.cc:2454] T f9746769e2514c65ac8d48ca7df8bca7 P b00f244c69df401997ea99d492218a98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 2.
I20240503 08:13:32.899333 30895 raft_consensus.cc:2454] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 2.
I20240503 08:13:32.899755 30694 leader_election.cc:302] T f9746769e2514c65ac8d48ca7df8bca7 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, b00f244c69df401997ea99d492218a98, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:13:32.900092 2812 raft_consensus.cc:2781] T f9746769e2514c65ac8d48ca7df8bca7 P 6d1d7de19459435fbde1999e3dfd226b [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:32.900203 2812 raft_consensus.cc:686] T f9746769e2514c65ac8d48ca7df8bca7 P 6d1d7de19459435fbde1999e3dfd226b [term 2 LEADER]: Becoming Leader. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Running, Role: LEADER
I20240503 08:13:32.900295 28001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9746769e2514c65ac8d48ca7df8bca7" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:13:32.900481 28001 raft_consensus.cc:3037] T f9746769e2514c65ac8d48ca7df8bca7 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:32.900550 30761 leader_election.cc:288] T f9746769e2514c65ac8d48ca7df8bca7 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 2 election: Requested vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:13:32.900457 2812 consensus_queue.cc:237] T f9746769e2514c65ac8d48ca7df8bca7 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:32.903069 28001 raft_consensus.cc:2454] T f9746769e2514c65ac8d48ca7df8bca7 P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 2.
I20240503 08:13:32.906611 27858 catalog_manager.cc:5484] T f9746769e2514c65ac8d48ca7df8bca7 P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: term changed from 1 to 2, leader changed from ce1496a47d084365ad786cd09e472a2c (127.20.230.70) to 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69). New cstate: current_term: 2 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:33.105548 27983 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 08:13:33.112462 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:33.114012 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:33.137667 28243 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:33.164094 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:33.166575 30874 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:13:33.218827 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:33.283632 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:33.315959 28000 raft_consensus.cc:1261] T f9746769e2514c65ac8d48ca7df8bca7 P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:13:33.316679 2812 consensus_queue.cc:1035] T f9746769e2514c65ac8d48ca7df8bca7 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:33.328267 31160 raft_consensus.cc:1261] T f9746769e2514c65ac8d48ca7df8bca7 P b00f244c69df401997ea99d492218a98 [term 2 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:13:33.328787 2812 consensus_queue.cc:1035] T f9746769e2514c65ac8d48ca7df8bca7 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:33.340538 30894 raft_consensus.cc:1261] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:13:33.341660 3312 consensus_queue.cc:1035] T f9746769e2514c65ac8d48ca7df8bca7 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:33.704325 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:33.744872 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:33.787606 27983 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 08:13:33.794066 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:33.801313 28243 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:33.827750 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:33.844574 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:33.854844 30874 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:13:34.021809 31160 consensus_queue.cc:237] T b40a43ad7ed44c30a070189ebfe5aab6 P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:34.024086 28002 raft_consensus.cc:1261] T b40a43ad7ed44c30a070189ebfe5aab6 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:34.024755 30761 raft_consensus.cc:1261] T b40a43ad7ed44c30a070189ebfe5aab6 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:34.025442 3860 consensus_queue.cc:1035] T b40a43ad7ed44c30a070189ebfe5aab6 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:34.025795 3926 consensus_queue.cc:1035] T b40a43ad7ed44c30a070189ebfe5aab6 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:34.028681 4021 raft_consensus.cc:2932] T b40a43ad7ed44c30a070189ebfe5aab6 P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:34.029619 30761 raft_consensus.cc:2932] T b40a43ad7ed44c30a070189ebfe5aab6 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:34.029486 28000 raft_consensus.cc:2932] T b40a43ad7ed44c30a070189ebfe5aab6 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:34.035665 27854 catalog_manager.cc:5484] T b40a43ad7ed44c30a070189ebfe5aab6 P b00f244c69df401997ea99d492218a98 reported cstate change: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New cstate: current_term: 1 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:34.048408 31006 tablet_service.cc:1514] Processing DeleteTablet for tablet b40a43ad7ed44c30a070189ebfe5aab6 with delete_type TABLET_DATA_TOMBSTONED (TS ced21ca983b749d0b4d4f6a0e2426bf8 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47020
I20240503 08:13:34.061661 4028 tablet_replica.cc:330] T b40a43ad7ed44c30a070189ebfe5aab6 P ced21ca983b749d0b4d4f6a0e2426bf8: stopping tablet replica
I20240503 08:13:34.062144 4028 raft_consensus.cc:2229] T b40a43ad7ed44c30a070189ebfe5aab6 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:34.062520 4028 raft_consensus.cc:2258] T b40a43ad7ed44c30a070189ebfe5aab6 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:34.063352 4028 ts_tablet_manager.cc:1875] T b40a43ad7ed44c30a070189ebfe5aab6 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:34.070304 4028 ts_tablet_manager.cc:1888] T b40a43ad7ed44c30a070189ebfe5aab6 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:13:34.070425 4028 log.cc:1196] T b40a43ad7ed44c30a070189ebfe5aab6 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal/wals/b40a43ad7ed44c30a070189ebfe5aab6
I20240503 08:13:34.071244 27847 catalog_manager.cc:4830] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): tablet b40a43ad7ed44c30a070189ebfe5aab6 (table rebalancer_extra_table_40 [id=3aef15bedeb544e9a15460f0b11c368a]) successfully deleted
I20240503 08:13:34.353382 27983 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 08:13:34.387496 28243 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:34.400735 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:34.401506 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:34.407765 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:34.449097 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:34.455641 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:34.487345 30874 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:13:34.620983 31027 consensus_queue.cc:237] T 4ed8026daee840c6b31ca7d989ed25f0 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:34.623425 28133 raft_consensus.cc:1261] T 4ed8026daee840c6b31ca7d989ed25f0 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer ced21ca983b749d0b4d4f6a0e2426bf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:34.623440 28389 raft_consensus.cc:1261] T 4ed8026daee840c6b31ca7d989ed25f0 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer ced21ca983b749d0b4d4f6a0e2426bf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:34.624032 3738 consensus_queue.cc:1035] T 4ed8026daee840c6b31ca7d989ed25f0 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:34.624336 3981 consensus_queue.cc:1035] T 4ed8026daee840c6b31ca7d989ed25f0 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:34.628099 3738 raft_consensus.cc:2932] T 4ed8026daee840c6b31ca7d989ed25f0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:34.628459 28133 raft_consensus.cc:2932] T 4ed8026daee840c6b31ca7d989ed25f0 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:34.641119 27858 catalog_manager.cc:5484] T 4ed8026daee840c6b31ca7d989ed25f0 P ced21ca983b749d0b4d4f6a0e2426bf8 reported cstate change: config changed from index 2 to 3, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) evicted. New cstate: current_term: 1 leader_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:34.648875 28390 raft_consensus.cc:2932] T 4ed8026daee840c6b31ca7d989ed25f0 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:34.675981 27983 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ed8026daee840c6b31ca7d989ed25f0 with delete_type TABLET_DATA_TOMBSTONED (TS 833582ff68c94269be364ebd187a75db not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:55610
I20240503 08:13:34.677937 4121 tablet_replica.cc:330] T 4ed8026daee840c6b31ca7d989ed25f0 P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:13:34.678198 4121 raft_consensus.cc:2229] T 4ed8026daee840c6b31ca7d989ed25f0 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:34.679993 4121 raft_consensus.cc:2258] T 4ed8026daee840c6b31ca7d989ed25f0 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:34.680658 4121 ts_tablet_manager.cc:1875] T 4ed8026daee840c6b31ca7d989ed25f0 P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:34.687413 4121 ts_tablet_manager.cc:1888] T 4ed8026daee840c6b31ca7d989ed25f0 P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:13:34.687583 4121 log.cc:1196] T 4ed8026daee840c6b31ca7d989ed25f0 P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/4ed8026daee840c6b31ca7d989ed25f0
I20240503 08:13:34.689672 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet 4ed8026daee840c6b31ca7d989ed25f0 (table rebalancer_extra_table_6 [id=a80c6942ff454bcfb6a142a36fb80741]) successfully deleted
I20240503 08:13:34.953933 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:35.006608 27983 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 08:13:35.016875 30874 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:13:35.027611 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:35.036345 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:35.066172 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:35.066613 28243 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:35.080018 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:35.265911 31161 consensus_queue.cc:237] T 9fc39354ba064392be1af1769513f738 P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:35.268321 30761 raft_consensus.cc:1261] T 9fc39354ba064392be1af1769513f738 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:35.268993 31026 raft_consensus.cc:1261] T 9fc39354ba064392be1af1769513f738 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:35.269076 4179 consensus_queue.cc:1035] T 9fc39354ba064392be1af1769513f738 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:35.269503 4021 consensus_queue.cc:1035] T 9fc39354ba064392be1af1769513f738 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:35.271986 3860 raft_consensus.cc:2932] T 9fc39354ba064392be1af1769513f738 P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:13:35.273274 30761 raft_consensus.cc:2932] T 9fc39354ba064392be1af1769513f738 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:13:35.274057 31026 raft_consensus.cc:2932] T 9fc39354ba064392be1af1769513f738 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:13:35.278712 27861 catalog_manager.cc:5484] T 9fc39354ba064392be1af1769513f738 P b00f244c69df401997ea99d492218a98 reported cstate change: config changed from index 2 to 3, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New cstate: current_term: 1 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:13:35.290881 30874 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fc39354ba064392be1af1769513f738 with delete_type TABLET_DATA_TOMBSTONED (TS ce1496a47d084365ad786cd09e472a2c not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:34426
I20240503 08:13:35.294498 4201 tablet_replica.cc:330] T 9fc39354ba064392be1af1769513f738 P ce1496a47d084365ad786cd09e472a2c: stopping tablet replica
I20240503 08:13:35.294768 4201 raft_consensus.cc:2229] T 9fc39354ba064392be1af1769513f738 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:35.295001 4201 raft_consensus.cc:2258] T 9fc39354ba064392be1af1769513f738 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:35.295616 4201 ts_tablet_manager.cc:1875] T 9fc39354ba064392be1af1769513f738 P ce1496a47d084365ad786cd09e472a2c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:35.300283 4201 ts_tablet_manager.cc:1888] T 9fc39354ba064392be1af1769513f738 P ce1496a47d084365ad786cd09e472a2c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:13:35.300390 4201 log.cc:1196] T 9fc39354ba064392be1af1769513f738 P ce1496a47d084365ad786cd09e472a2c: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal/wals/9fc39354ba064392be1af1769513f738
I20240503 08:13:35.301088 27846 catalog_manager.cc:4830] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): tablet 9fc39354ba064392be1af1769513f738 (table rebalancer_extra_table_26 [id=1b577ea991674f799fea7ba7f25e80ad]) successfully deleted
I20240503 08:13:35.499837 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:35.506398 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:35.507529 27983 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 08:13:35.552182 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:35.610707 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:35.621747 28243 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:35.667250 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:35.707208 30874 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:13:35.816402 31161 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "17c6e3293aad4591a9c1395ebffff2e6"
dest_uuid: "b00f244c69df401997ea99d492218a98"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56894
I20240503 08:13:35.816646 31161 raft_consensus.cc:595] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:13:35.930279 4021 raft_consensus.cc:979] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98: : Instructing follower aae860d7ae2b4002bf7794c278f9cc56 to start an election
I20240503 08:13:35.930424 4021 raft_consensus.cc:1067] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Signalling peer aae860d7ae2b4002bf7794c278f9cc56 to start an election
I20240503 08:13:35.933048 28262 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "17c6e3293aad4591a9c1395ebffff2e6"
dest_uuid: "aae860d7ae2b4002bf7794c278f9cc56"
from {username='slave'} at 127.20.230.72:55725
I20240503 08:13:35.933230 28262 raft_consensus.cc:482] T 17c6e3293aad4591a9c1395ebffff2e6 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:35.933315 28262 raft_consensus.cc:3037] T 17c6e3293aad4591a9c1395ebffff2e6 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:35.937516 28262 raft_consensus.cc:504] T 17c6e3293aad4591a9c1395ebffff2e6 P aae860d7ae2b4002bf7794c278f9cc56 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:35.938992 31161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17c6e3293aad4591a9c1395ebffff2e6" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:13:35.939167 31161 raft_consensus.cc:3032] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:35.939230 31161 raft_consensus.cc:729] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Running, Role: LEADER
I20240503 08:13:35.939239 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17c6e3293aad4591a9c1395ebffff2e6" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:13:35.939400 31027 raft_consensus.cc:3037] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:35.939482 31161 consensus_queue.cc:260] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:35.939868 31161 raft_consensus.cc:3037] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:35.942039 31027 raft_consensus.cc:2454] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 2.
I20240503 08:13:35.942263 31161 raft_consensus.cc:2454] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 2.
I20240503 08:13:35.942909 28133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17c6e3293aad4591a9c1395ebffff2e6" candidate_uuid: "aae860d7ae2b4002bf7794c278f9cc56" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:13:35.943073 28133 raft_consensus.cc:3037] T 17c6e3293aad4591a9c1395ebffff2e6 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:35.943929 28196 leader_election.cc:302] T 17c6e3293aad4591a9c1395ebffff2e6 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: aae860d7ae2b4002bf7794c278f9cc56, b00f244c69df401997ea99d492218a98, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:13:35.944187 3983 raft_consensus.cc:2781] T 17c6e3293aad4591a9c1395ebffff2e6 P aae860d7ae2b4002bf7794c278f9cc56 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:35.944307 3983 raft_consensus.cc:686] T 17c6e3293aad4591a9c1395ebffff2e6 P aae860d7ae2b4002bf7794c278f9cc56 [term 2 LEADER]: Becoming Leader. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Running, Role: LEADER
I20240503 08:13:35.944554 3983 consensus_queue.cc:237] T 17c6e3293aad4591a9c1395ebffff2e6 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:35.945091 28133 raft_consensus.cc:2454] T 17c6e3293aad4591a9c1395ebffff2e6 P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae860d7ae2b4002bf7794c278f9cc56 in term 2.
I20240503 08:13:35.951507 28262 leader_election.cc:288] T 17c6e3293aad4591a9c1395ebffff2e6 P aae860d7ae2b4002bf7794c278f9cc56 [CANDIDATE]: Term 2 election: Requested vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:13:35.954811 27856 catalog_manager.cc:5484] T 17c6e3293aad4591a9c1395ebffff2e6 P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: term changed from 1 to 2, leader changed from b00f244c69df401997ea99d492218a98 (127.20.230.72) to aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67). New cstate: current_term: 2 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:36.035951 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:36.057586 27983 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 08:13:36.064658 31141 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:36.069625 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:36.082247 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:36.114856 30874 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:13:36.140761 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:36.184331 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:36.442940 31159 raft_consensus.cc:1261] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98 [term 2 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:13:36.443442 4178 consensus_queue.cc:1035] T 17c6e3293aad4591a9c1395ebffff2e6 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:36.487005 28132 raft_consensus.cc:1261] T 17c6e3293aad4591a9c1395ebffff2e6 P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:13:36.487581 4178 consensus_queue.cc:1035] T 17c6e3293aad4591a9c1395ebffff2e6 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:36.537894 31028 raft_consensus.cc:1261] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:13:36.538439 3983 consensus_queue.cc:1035] T 17c6e3293aad4591a9c1395ebffff2e6 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:36.605254 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:36.629798 31141 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:36.641798 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:36.653973 30874 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:13:36.660853 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:36.667639 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:36.680658 27983 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 08:13:36.733551 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:36.875411 28263 consensus_queue.cc:237] T 08fb64977309479b831635344969c662 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:36.878726 30761 raft_consensus.cc:1261] T 08fb64977309479b831635344969c662 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:36.878855 28002 raft_consensus.cc:1261] T 08fb64977309479b831635344969c662 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:36.879242 3983 consensus_queue.cc:1035] T 08fb64977309479b831635344969c662 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:36.879583 3978 consensus_queue.cc:1035] T 08fb64977309479b831635344969c662 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:36.882843 3983 raft_consensus.cc:2932] T 08fb64977309479b831635344969c662 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:36.883487 30761 raft_consensus.cc:2932] T 08fb64977309479b831635344969c662 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:36.883834 28002 raft_consensus.cc:2932] T 08fb64977309479b831635344969c662 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:36.891002 27862 catalog_manager.cc:5484] T 08fb64977309479b831635344969c662 P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: config changed from index 4 to 5, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:36.896348 28111 tablet_service.cc:1514] Processing DeleteTablet for tablet 08fb64977309479b831635344969c662 with delete_type TABLET_DATA_TOMBSTONED (TS 1b411c6bb25549368041773d7b710d48 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57912
I20240503 08:13:36.907476 4411 tablet_replica.cc:330] T 08fb64977309479b831635344969c662 P 1b411c6bb25549368041773d7b710d48: stopping tablet replica
I20240503 08:13:36.907689 4411 raft_consensus.cc:2229] T 08fb64977309479b831635344969c662 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:36.907868 4411 raft_consensus.cc:2258] T 08fb64977309479b831635344969c662 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:36.908432 4411 ts_tablet_manager.cc:1875] T 08fb64977309479b831635344969c662 P 1b411c6bb25549368041773d7b710d48: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:36.916074 4411 ts_tablet_manager.cc:1888] T 08fb64977309479b831635344969c662 P 1b411c6bb25549368041773d7b710d48: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:13:36.916234 4411 log.cc:1196] T 08fb64977309479b831635344969c662 P 1b411c6bb25549368041773d7b710d48: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/08fb64977309479b831635344969c662
I20240503 08:13:36.916847 27848 catalog_manager.cc:4830] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): tablet 08fb64977309479b831635344969c662 (table rebalance_test_table_19 [id=18d795e9c19046fdba033edcea369490]) successfully deleted
I20240503 08:13:37.241060 27983 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 08:13:37.290571 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:37.309933 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:37.324040 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:37.339624 31141 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:37.365360 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:37.376030 30874 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:13:37.401713 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:37.495775 28002 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "df31bd9e544a44f280db2b0d021000c2"
dest_uuid: "833582ff68c94269be364ebd187a75db"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57934
I20240503 08:13:37.496026 28002 raft_consensus.cc:595] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Received request to transfer leadership
I20240503 08:13:37.520239 3911 raft_consensus.cc:979] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db: : Instructing follower ce1496a47d084365ad786cd09e472a2c to start an election
I20240503 08:13:37.520401 3911 raft_consensus.cc:1067] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Signalling peer ce1496a47d084365ad786cd09e472a2c to start an election
I20240503 08:13:37.521253 30894 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "df31bd9e544a44f280db2b0d021000c2"
dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
from {username='slave'} at 127.20.230.65:60309
I20240503 08:13:37.521452 30894 raft_consensus.cc:482] T df31bd9e544a44f280db2b0d021000c2 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:37.521541 30894 raft_consensus.cc:3037] T df31bd9e544a44f280db2b0d021000c2 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:37.524262 30894 raft_consensus.cc:504] T df31bd9e544a44f280db2b0d021000c2 P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:37.536185 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df31bd9e544a44f280db2b0d021000c2" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:13:37.536371 28002 raft_consensus.cc:3032] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:37.536435 28002 raft_consensus.cc:729] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Running, Role: LEADER
I20240503 08:13:37.536749 28002 consensus_queue.cc:260] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:37.537168 28002 raft_consensus.cc:3037] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:37.539880 28002 raft_consensus.cc:2454] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 2.
I20240503 08:13:37.540313 30894 leader_election.cc:288] T df31bd9e544a44f280db2b0d021000c2 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 election: Requested vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:13:37.540588 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df31bd9e544a44f280db2b0d021000c2" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:13:37.540771 30761 raft_consensus.cc:3037] T df31bd9e544a44f280db2b0d021000c2 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:37.540966 28390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df31bd9e544a44f280db2b0d021000c2" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:13:37.541131 28390 raft_consensus.cc:3037] T df31bd9e544a44f280db2b0d021000c2 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:37.542882 30761 raft_consensus.cc:2454] T df31bd9e544a44f280db2b0d021000c2 P 6d1d7de19459435fbde1999e3dfd226b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 2.
I20240503 08:13:37.543263 30829 leader_election.cc:302] T df31bd9e544a44f280db2b0d021000c2 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, 833582ff68c94269be364ebd187a75db, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:13:37.543726 2909 raft_consensus.cc:2781] T df31bd9e544a44f280db2b0d021000c2 P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:37.543725 28390 raft_consensus.cc:2454] T df31bd9e544a44f280db2b0d021000c2 P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 2.
I20240503 08:13:37.543851 2909 raft_consensus.cc:686] T df31bd9e544a44f280db2b0d021000c2 P ce1496a47d084365ad786cd09e472a2c [term 2 LEADER]: Becoming Leader. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Running, Role: LEADER
I20240503 08:13:37.544077 2909 consensus_queue.cc:237] T df31bd9e544a44f280db2b0d021000c2 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:37.548772 27853 catalog_manager.cc:5484] T df31bd9e544a44f280db2b0d021000c2 P ce1496a47d084365ad786cd09e472a2c reported cstate change: term changed from 1 to 2, leader changed from 833582ff68c94269be364ebd187a75db (127.20.230.65) to ce1496a47d084365ad786cd09e472a2c (127.20.230.70). New cstate: current_term: 2 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:37.842227 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:37.851204 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:37.869345 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:37.869499 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:37.880023 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:37.942924 31141 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:37.944608 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:38.025986 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:38.095393 28390 raft_consensus.cc:1261] T df31bd9e544a44f280db2b0d021000c2 P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:13:38.095960 4480 consensus_queue.cc:1035] T df31bd9e544a44f280db2b0d021000c2 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:38.104888 30760 raft_consensus.cc:1261] T df31bd9e544a44f280db2b0d021000c2 P 6d1d7de19459435fbde1999e3dfd226b [term 2 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:13:38.105906 3544 consensus_queue.cc:1035] T df31bd9e544a44f280db2b0d021000c2 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:38.113621 28002 raft_consensus.cc:1261] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:13:38.114248 4480 consensus_queue.cc:1035] T df31bd9e544a44f280db2b0d021000c2 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:38.414889 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:38.416045 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:38.425364 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:38.438956 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:38.447542 31141 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:38.483573 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:38.515071 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:38.522475 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:38.740003 31159 consensus_queue.cc:237] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:38.741938 30893 raft_consensus.cc:1261] T 4a23c8186c4d4b23b62d7474088b45c1 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:38.742202 28263 raft_consensus.cc:1261] T 4a23c8186c4d4b23b62d7474088b45c1 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:38.742617 4179 consensus_queue.cc:1035] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:38.742841 4456 consensus_queue.cc:1035] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:38.745265 4179 raft_consensus.cc:2932] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:38.747102 30893 raft_consensus.cc:2932] T 4a23c8186c4d4b23b62d7474088b45c1 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:38.749475 28261 raft_consensus.cc:2932] T 4a23c8186c4d4b23b62d7474088b45c1 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:38.755615 27860 catalog_manager.cc:5484] T 4a23c8186c4d4b23b62d7474088b45c1 P ce1496a47d084365ad786cd09e472a2c reported cstate change: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New cstate: current_term: 1 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:38.762030 31006 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a23c8186c4d4b23b62d7474088b45c1 with delete_type TABLET_DATA_TOMBSTONED (TS ced21ca983b749d0b4d4f6a0e2426bf8 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47020
I20240503 08:13:38.762609 4645 tablet_replica.cc:330] T 4a23c8186c4d4b23b62d7474088b45c1 P ced21ca983b749d0b4d4f6a0e2426bf8: stopping tablet replica
I20240503 08:13:38.762928 4645 raft_consensus.cc:2229] T 4a23c8186c4d4b23b62d7474088b45c1 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:38.763183 4645 raft_consensus.cc:2258] T 4a23c8186c4d4b23b62d7474088b45c1 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:38.763769 4645 ts_tablet_manager.cc:1875] T 4a23c8186c4d4b23b62d7474088b45c1 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:38.769395 4645 ts_tablet_manager.cc:1888] T 4a23c8186c4d4b23b62d7474088b45c1 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:13:38.769549 4645 log.cc:1196] T 4a23c8186c4d4b23b62d7474088b45c1 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal/wals/4a23c8186c4d4b23b62d7474088b45c1
I20240503 08:13:38.771090 27847 catalog_manager.cc:4830] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): tablet 4a23c8186c4d4b23b62d7474088b45c1 (table rebalancer_extra_table_10 [id=9ee47b3ce8574777976ff3cb1ea03e9c]) successfully deleted
I20240503 08:13:39.000727 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:39.024658 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:39.073004 31141 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:39.077575 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:39.077980 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:39.096892 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:39.117130 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:39.120797 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:39.304847 28390 consensus_queue.cc:237] T 8425cfe5c6134a60852e12f9af55227b P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:39.306623 28002 raft_consensus.cc:1261] T 8425cfe5c6134a60852e12f9af55227b P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:39.306855 28261 raft_consensus.cc:1261] T 8425cfe5c6134a60852e12f9af55227b P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:39.307145 4317 consensus_queue.cc:1035] T 8425cfe5c6134a60852e12f9af55227b P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:39.307389 4089 consensus_queue.cc:1035] T 8425cfe5c6134a60852e12f9af55227b P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:39.309656 4089 raft_consensus.cc:2932] T 8425cfe5c6134a60852e12f9af55227b P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:39.310254 28261 raft_consensus.cc:2932] T 8425cfe5c6134a60852e12f9af55227b P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:39.313086 28002 raft_consensus.cc:2932] T 8425cfe5c6134a60852e12f9af55227b P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:39.316378 27862 catalog_manager.cc:5484] T 8425cfe5c6134a60852e12f9af55227b P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:39.323403 31141 tablet_service.cc:1514] Processing DeleteTablet for tablet 8425cfe5c6134a60852e12f9af55227b with delete_type TABLET_DATA_TOMBSTONED (TS b00f244c69df401997ea99d492218a98 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45814
I20240503 08:13:39.325585 4730 tablet_replica.cc:330] T 8425cfe5c6134a60852e12f9af55227b P b00f244c69df401997ea99d492218a98: stopping tablet replica
I20240503 08:13:39.325803 4730 raft_consensus.cc:2229] T 8425cfe5c6134a60852e12f9af55227b P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:39.326001 4730 raft_consensus.cc:2258] T 8425cfe5c6134a60852e12f9af55227b P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:39.326539 4730 ts_tablet_manager.cc:1875] T 8425cfe5c6134a60852e12f9af55227b P b00f244c69df401997ea99d492218a98: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:39.331120 4730 ts_tablet_manager.cc:1888] T 8425cfe5c6134a60852e12f9af55227b P b00f244c69df401997ea99d492218a98: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:13:39.331254 4730 log.cc:1196] T 8425cfe5c6134a60852e12f9af55227b P b00f244c69df401997ea99d492218a98: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal/wals/8425cfe5c6134a60852e12f9af55227b
I20240503 08:13:39.331854 27846 catalog_manager.cc:4830] TS b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): tablet 8425cfe5c6134a60852e12f9af55227b (table rebalancer_extra_table_14 [id=6db67fd7274948c0858a325398f6acbc]) successfully deleted
I20240503 08:13:39.583406 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:39.588986 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:39.613391 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:39.628911 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:39.701586 31141 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:39.734609 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:39.742305 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:39.742424 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:39.892220 28261 consensus_queue.cc:237] T cdad6ef53b6f413ebfda67f9b42c9c23 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:39.894037 28389 raft_consensus.cc:1261] T cdad6ef53b6f413ebfda67f9b42c9c23 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:39.894038 30761 raft_consensus.cc:1261] T cdad6ef53b6f413ebfda67f9b42c9c23 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:39.894485 4178 consensus_queue.cc:1035] T cdad6ef53b6f413ebfda67f9b42c9c23 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:39.894721 3983 consensus_queue.cc:1035] T cdad6ef53b6f413ebfda67f9b42c9c23 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:39.897133 3983 raft_consensus.cc:2932] T cdad6ef53b6f413ebfda67f9b42c9c23 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:39.897547 30761 raft_consensus.cc:2932] T cdad6ef53b6f413ebfda67f9b42c9c23 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:39.897702 28389 raft_consensus.cc:2932] T cdad6ef53b6f413ebfda67f9b42c9c23 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:39.904044 27862 catalog_manager.cc:5484] T cdad6ef53b6f413ebfda67f9b42c9c23 P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: config changed from index 4 to 5, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:39.914842 28111 tablet_service.cc:1514] Processing DeleteTablet for tablet cdad6ef53b6f413ebfda67f9b42c9c23 with delete_type TABLET_DATA_TOMBSTONED (TS 1b411c6bb25549368041773d7b710d48 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57912
I20240503 08:13:39.917476 4817 tablet_replica.cc:330] T cdad6ef53b6f413ebfda67f9b42c9c23 P 1b411c6bb25549368041773d7b710d48: stopping tablet replica
I20240503 08:13:39.917675 4817 raft_consensus.cc:2229] T cdad6ef53b6f413ebfda67f9b42c9c23 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:39.917966 4817 raft_consensus.cc:2258] T cdad6ef53b6f413ebfda67f9b42c9c23 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:39.918818 4817 ts_tablet_manager.cc:1875] T cdad6ef53b6f413ebfda67f9b42c9c23 P 1b411c6bb25549368041773d7b710d48: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:39.923935 4817 ts_tablet_manager.cc:1888] T cdad6ef53b6f413ebfda67f9b42c9c23 P 1b411c6bb25549368041773d7b710d48: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:13:39.924093 4817 log.cc:1196] T cdad6ef53b6f413ebfda67f9b42c9c23 P 1b411c6bb25549368041773d7b710d48: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/cdad6ef53b6f413ebfda67f9b42c9c23
I20240503 08:13:39.925676 27848 catalog_manager.cc:4830] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): tablet cdad6ef53b6f413ebfda67f9b42c9c23 (table rebalance_test_table_17 [id=7e5325b51bd7410e93410447bfd72f9c]) successfully deleted
I20240503 08:13:40.209359 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:40.226662 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:40.232159 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:40.243162 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:40.289314 31141 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:40.360852 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:40.361701 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:40.389681 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240503 08:13:40.398355 28327 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.67:39919, user_credentials={real_user=slave}} blocked reactor thread for 47884us
I20240503 08:13:40.507032 31027 consensus_queue.cc:237] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:40.516608 28261 raft_consensus.cc:1261] T b12ceaf7f8d04fad8b07b36459da7116 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer ced21ca983b749d0b4d4f6a0e2426bf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:40.516635 28389 raft_consensus.cc:1261] T b12ceaf7f8d04fad8b07b36459da7116 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer ced21ca983b749d0b4d4f6a0e2426bf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:40.517045 4854 consensus_queue.cc:1035] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:40.517575 3738 consensus_queue.cc:1035] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:40.530277 3738 raft_consensus.cc:2932] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:40.537425 27860 catalog_manager.cc:5484] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8 reported cstate change: config changed from index 2 to 3, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) evicted. New cstate: current_term: 1 leader_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:40.544713 28261 raft_consensus.cc:2932] T b12ceaf7f8d04fad8b07b36459da7116 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:40.558405 28390 raft_consensus.cc:2932] T b12ceaf7f8d04fad8b07b36459da7116 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:40.572026 27983 tablet_service.cc:1514] Processing DeleteTablet for tablet b12ceaf7f8d04fad8b07b36459da7116 with delete_type TABLET_DATA_TOMBSTONED (TS 833582ff68c94269be364ebd187a75db not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:55610
I20240503 08:13:40.580145 4909 tablet_replica.cc:330] T b12ceaf7f8d04fad8b07b36459da7116 P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:13:40.580361 4909 raft_consensus.cc:2229] T b12ceaf7f8d04fad8b07b36459da7116 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:40.580544 4909 raft_consensus.cc:2258] T b12ceaf7f8d04fad8b07b36459da7116 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:40.581115 4909 ts_tablet_manager.cc:1875] T b12ceaf7f8d04fad8b07b36459da7116 P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:40.590505 4909 ts_tablet_manager.cc:1888] T b12ceaf7f8d04fad8b07b36459da7116 P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:13:40.590682 4909 log.cc:1196] T b12ceaf7f8d04fad8b07b36459da7116 P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/b12ceaf7f8d04fad8b07b36459da7116
I20240503 08:13:40.591410 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet b12ceaf7f8d04fad8b07b36459da7116 (table rebalancer_extra_table_16 [id=c740702474bd4c328043a94e39c2db1b]) successfully deleted
I20240503 08:13:40.856071 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:40.867544 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:40.879557 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:40.891459 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:40.910169 31141 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:40.936158 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:40.973729 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:41.038939 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:41.126776 30894 consensus_queue.cc:237] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:41.128639 28261 raft_consensus.cc:1261] T f53acf725b62447382e9bae95e6e74d5 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:41.128764 30761 raft_consensus.cc:1261] T f53acf725b62447382e9bae95e6e74d5 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:41.129179 4480 consensus_queue.cc:1035] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:41.129851 3544 consensus_queue.cc:1035] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:41.131937 3544 raft_consensus.cc:2932] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:41.139145 27854 catalog_manager.cc:5484] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c reported cstate change: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New cstate: current_term: 1 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:41.140612 28261 raft_consensus.cc:2932] T f53acf725b62447382e9bae95e6e74d5 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:41.146975 30761 raft_consensus.cc:2932] T f53acf725b62447382e9bae95e6e74d5 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:41.158118 31006 tablet_service.cc:1514] Processing DeleteTablet for tablet f53acf725b62447382e9bae95e6e74d5 with delete_type TABLET_DATA_TOMBSTONED (TS ced21ca983b749d0b4d4f6a0e2426bf8 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47020
I20240503 08:13:41.159037 5003 tablet_replica.cc:330] T f53acf725b62447382e9bae95e6e74d5 P ced21ca983b749d0b4d4f6a0e2426bf8: stopping tablet replica
I20240503 08:13:41.159253 5003 raft_consensus.cc:2229] T f53acf725b62447382e9bae95e6e74d5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:41.159505 5003 raft_consensus.cc:2258] T f53acf725b62447382e9bae95e6e74d5 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:41.160298 5003 ts_tablet_manager.cc:1875] T f53acf725b62447382e9bae95e6e74d5 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:41.167318 5003 ts_tablet_manager.cc:1888] T f53acf725b62447382e9bae95e6e74d5 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:13:41.167440 5003 log.cc:1196] T f53acf725b62447382e9bae95e6e74d5 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal/wals/f53acf725b62447382e9bae95e6e74d5
I20240503 08:13:41.168542 27847 catalog_manager.cc:4830] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): tablet f53acf725b62447382e9bae95e6e74d5 (table rebalancer_extra_table_18 [id=e4d6c0dac75042f984dc1aa8fa801cee]) successfully deleted
I20240503 08:13:41.430307 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:41.433053 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:41.444798 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:41.452788 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:41.467736 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:41.482093 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:41.489738 31141 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:41.512156 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240503 08:13:41.541802 31096 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.69:34725, user_credentials={real_user=slave}} blocked reactor thread for 114572us
I20240503 08:13:41.683804 28390 consensus_queue.cc:237] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:41.685570 28262 raft_consensus.cc:1261] T 599cb18c58a74e93a3e0334cc43a73d4 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:41.685657 31159 raft_consensus.cc:1261] T 599cb18c58a74e93a3e0334cc43a73d4 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:41.686136 4972 consensus_queue.cc:1035] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:41.686399 4089 consensus_queue.cc:1035] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:41.688882 4972 raft_consensus.cc:2932] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:13:41.689913 28262 raft_consensus.cc:2932] T 599cb18c58a74e93a3e0334cc43a73d4 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:13:41.691555 31159 raft_consensus.cc:2932] T 599cb18c58a74e93a3e0334cc43a73d4 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:13:41.694447 27853 catalog_manager.cc:5484] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: config changed from index 4 to 5, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:13:41.698789 28111 tablet_service.cc:1514] Processing DeleteTablet for tablet 599cb18c58a74e93a3e0334cc43a73d4 with delete_type TABLET_DATA_TOMBSTONED (TS 1b411c6bb25549368041773d7b710d48 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57912
I20240503 08:13:41.699798 5080 tablet_replica.cc:330] T 599cb18c58a74e93a3e0334cc43a73d4 P 1b411c6bb25549368041773d7b710d48: stopping tablet replica
I20240503 08:13:41.700006 5080 raft_consensus.cc:2229] T 599cb18c58a74e93a3e0334cc43a73d4 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:41.700428 5080 raft_consensus.cc:2258] T 599cb18c58a74e93a3e0334cc43a73d4 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:41.701023 5080 ts_tablet_manager.cc:1875] T 599cb18c58a74e93a3e0334cc43a73d4 P 1b411c6bb25549368041773d7b710d48: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:41.707187 5080 ts_tablet_manager.cc:1888] T 599cb18c58a74e93a3e0334cc43a73d4 P 1b411c6bb25549368041773d7b710d48: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:13:41.707271 5080 log.cc:1196] T 599cb18c58a74e93a3e0334cc43a73d4 P 1b411c6bb25549368041773d7b710d48: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/599cb18c58a74e93a3e0334cc43a73d4
I20240503 08:13:41.707847 27848 catalog_manager.cc:4830] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): tablet 599cb18c58a74e93a3e0334cc43a73d4 (table rebalance_test_table_9 [id=eeeb4ab7272f4686b013f5d8f74fc917]) successfully deleted
I20240503 08:13:41.977921 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:41.982132 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:41.996655 31141 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:42.008716 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:42.017922 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:42.048801 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:42.068837 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:42.088784 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:42.257700 28262 consensus_queue.cc:237] T ca38671a760d4534ab903579e0487772 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:42.259527 28001 raft_consensus.cc:1261] T ca38671a760d4534ab903579e0487772 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:42.259546 31159 raft_consensus.cc:1261] T ca38671a760d4534ab903579e0487772 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:42.259992 5111 consensus_queue.cc:1035] T ca38671a760d4534ab903579e0487772 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:42.260262 4178 consensus_queue.cc:1035] T ca38671a760d4534ab903579e0487772 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:42.263404 5111 raft_consensus.cc:2932] T ca38671a760d4534ab903579e0487772 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:13:42.263731 28001 raft_consensus.cc:2932] T ca38671a760d4534ab903579e0487772 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:13:42.269275 27859 catalog_manager.cc:5484] T ca38671a760d4534ab903579e0487772 P 833582ff68c94269be364ebd187a75db reported cstate change: config changed from index 4 to 5, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:13:42.277935 28111 tablet_service.cc:1514] Processing DeleteTablet for tablet ca38671a760d4534ab903579e0487772 with delete_type TABLET_DATA_TOMBSTONED (TS 1b411c6bb25549368041773d7b710d48 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57912
I20240503 08:13:42.280984 5172 tablet_replica.cc:330] T ca38671a760d4534ab903579e0487772 P 1b411c6bb25549368041773d7b710d48: stopping tablet replica
I20240503 08:13:42.280923 31159 raft_consensus.cc:2932] T ca38671a760d4534ab903579e0487772 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:13:42.281445 5172 raft_consensus.cc:2229] T ca38671a760d4534ab903579e0487772 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:42.283088 5172 raft_consensus.cc:2258] T ca38671a760d4534ab903579e0487772 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:42.283812 5172 ts_tablet_manager.cc:1875] T ca38671a760d4534ab903579e0487772 P 1b411c6bb25549368041773d7b710d48: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:42.289899 5172 ts_tablet_manager.cc:1888] T ca38671a760d4534ab903579e0487772 P 1b411c6bb25549368041773d7b710d48: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:13:42.290668 5172 log.cc:1196] T ca38671a760d4534ab903579e0487772 P 1b411c6bb25549368041773d7b710d48: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/ca38671a760d4534ab903579e0487772
I20240503 08:13:42.291464 27848 catalog_manager.cc:4830] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): tablet ca38671a760d4534ab903579e0487772 (table rebalance_test_table_13 [id=6e8538506685489f8a1e931a2a781746]) successfully deleted
I20240503 08:13:42.481447 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:42.515592 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:42.515615 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:42.516086 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:42.519845 31141 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:42.595661 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:42.596223 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:42.657368 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:42.794905 30894 consensus_queue.cc:237] T 62ca502a089f4a37999d2252776b741b P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:42.797073 28002 raft_consensus.cc:1261] T 62ca502a089f4a37999d2252776b741b P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:42.797073 28131 raft_consensus.cc:1261] T 62ca502a089f4a37999d2252776b741b P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:42.797537 5228 consensus_queue.cc:1035] T 62ca502a089f4a37999d2252776b741b P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:42.798559 5228 consensus_queue.cc:1035] T 62ca502a089f4a37999d2252776b741b P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:42.800869 5228 raft_consensus.cc:2932] T 62ca502a089f4a37999d2252776b741b P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:42.801266 28131 raft_consensus.cc:2932] T 62ca502a089f4a37999d2252776b741b P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:42.807125 27857 catalog_manager.cc:5484] T 62ca502a089f4a37999d2252776b741b P ce1496a47d084365ad786cd09e472a2c reported cstate change: config changed from index 2 to 3, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) evicted. New cstate: current_term: 1 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:42.811281 28002 raft_consensus.cc:2932] T 62ca502a089f4a37999d2252776b741b P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:42.820188 28243 tablet_service.cc:1514] Processing DeleteTablet for tablet 62ca502a089f4a37999d2252776b741b with delete_type TABLET_DATA_TOMBSTONED (TS aae860d7ae2b4002bf7794c278f9cc56 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:41960
I20240503 08:13:42.820755 5257 tablet_replica.cc:330] T 62ca502a089f4a37999d2252776b741b P aae860d7ae2b4002bf7794c278f9cc56: stopping tablet replica
I20240503 08:13:42.821007 5257 raft_consensus.cc:2229] T 62ca502a089f4a37999d2252776b741b P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:42.821211 5257 raft_consensus.cc:2258] T 62ca502a089f4a37999d2252776b741b P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:42.821696 5257 ts_tablet_manager.cc:1875] T 62ca502a089f4a37999d2252776b741b P aae860d7ae2b4002bf7794c278f9cc56: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:42.826217 5257 ts_tablet_manager.cc:1888] T 62ca502a089f4a37999d2252776b741b P aae860d7ae2b4002bf7794c278f9cc56: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:13:42.826310 5257 log.cc:1196] T 62ca502a089f4a37999d2252776b741b P aae860d7ae2b4002bf7794c278f9cc56: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal/wals/62ca502a089f4a37999d2252776b741b
I20240503 08:13:42.826879 27847 catalog_manager.cc:4830] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): tablet 62ca502a089f4a37999d2252776b741b (table rebalancer_extra_table_54 [id=a50fec2484c34f48867b5f6f7ca7f1e0]) successfully deleted
I20240503 08:13:42.876322 31027 consensus_queue.cc:237] T 48075c4a45d24e84bb2b9760710de3a0 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:42.878870 30760 raft_consensus.cc:1261] T 48075c4a45d24e84bb2b9760710de3a0 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer ced21ca983b749d0b4d4f6a0e2426bf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:13:42.879283 30959 consensus_peers.cc:485] T 48075c4a45d24e84bb2b9760710de3a0 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): Couldn't send request to peer ce1496a47d084365ad786cd09e472a2c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48075c4a45d24e84bb2b9760710de3a0. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:42.879325 31161 raft_consensus.cc:1261] T 48075c4a45d24e84bb2b9760710de3a0 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer ced21ca983b749d0b4d4f6a0e2426bf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:42.879796 4854 consensus_queue.cc:1035] T 48075c4a45d24e84bb2b9760710de3a0 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:42.880117 5225 consensus_queue.cc:1035] T 48075c4a45d24e84bb2b9760710de3a0 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:42.887271 4854 raft_consensus.cc:2932] T 48075c4a45d24e84bb2b9760710de3a0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:42.887867 30760 raft_consensus.cc:2932] T 48075c4a45d24e84bb2b9760710de3a0 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:42.894146 27859 catalog_manager.cc:5484] T 48075c4a45d24e84bb2b9760710de3a0 P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: config changed from index -1 to 2, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New cstate: current_term: 1 leader_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:42.894213 31161 raft_consensus.cc:2932] T 48075c4a45d24e84bb2b9760710de3a0 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:42.926891 31027 consensus_queue.cc:237] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
W20240503 08:13:42.929107 30962 consensus_peers.cc:485] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): Couldn't send request to peer d09a11a03989421b9f28019dc1c14cc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 200387e6edce4ee9bf11331380923005. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:42.930238 30894 raft_consensus.cc:1261] T 200387e6edce4ee9bf11331380923005 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer ced21ca983b749d0b4d4f6a0e2426bf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:42.930325 31161 raft_consensus.cc:1261] T 200387e6edce4ee9bf11331380923005 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer ced21ca983b749d0b4d4f6a0e2426bf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:42.930855 5288 consensus_queue.cc:1035] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:42.931128 4854 consensus_queue.cc:1035] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:42.934785 4854 raft_consensus.cc:2932] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:42.935257 30894 raft_consensus.cc:2932] T 200387e6edce4ee9bf11331380923005 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:42.937810 31161 raft_consensus.cc:2932] T 200387e6edce4ee9bf11331380923005 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:42.939769 27857 catalog_manager.cc:5484] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8 reported cstate change: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New cstate: current_term: 1 leader_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:42.979321 28389 consensus_queue.cc:237] T 58611f4ce8b04ac39064a51580d73f31 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:42.981705 30760 raft_consensus.cc:1261] T 58611f4ce8b04ac39064a51580d73f31 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:13:42.982036 28329 consensus_peers.cc:485] T 58611f4ce8b04ac39064a51580d73f31 P d09a11a03989421b9f28019dc1c14cc2 -> Peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Couldn't send request to peer 833582ff68c94269be364ebd187a75db. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 58611f4ce8b04ac39064a51580d73f31. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:42.982304 4972 consensus_queue.cc:1035] T 58611f4ce8b04ac39064a51580d73f31 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:42.986220 30894 raft_consensus.cc:1261] T 58611f4ce8b04ac39064a51580d73f31 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:42.986730 5324 consensus_queue.cc:1035] T 58611f4ce8b04ac39064a51580d73f31 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:42.990341 5324 raft_consensus.cc:2932] T 58611f4ce8b04ac39064a51580d73f31 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:42.991009 30894 raft_consensus.cc:2932] T 58611f4ce8b04ac39064a51580d73f31 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:42.997822 27858 catalog_manager.cc:5484] T 58611f4ce8b04ac39064a51580d73f31 P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:42.999028 30762 raft_consensus.cc:2932] T 58611f4ce8b04ac39064a51580d73f31 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:43.339098 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:43.346729 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:43.347659 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:43.355221 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:43.365094 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:43.399160 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:43.445302 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:43.456470 5391 ts_tablet_manager.cc:902] T 58611f4ce8b04ac39064a51580d73f31 P 833582ff68c94269be364ebd187a75db: Initiating tablet copy from peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:13:43.457547 31141 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:43.468479 5389 ts_tablet_manager.cc:902] T 48075c4a45d24e84bb2b9760710de3a0 P ce1496a47d084365ad786cd09e472a2c: Initiating tablet copy from peer ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:13:43.497112 5391 tablet_copy_client.cc:308] T 58611f4ce8b04ac39064a51580d73f31 P 833582ff68c94269be364ebd187a75db: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.68:46211
I20240503 08:13:43.497831 28413 tablet_copy_service.cc:140] P d09a11a03989421b9f28019dc1c14cc2: Received BeginTabletCopySession request for tablet 58611f4ce8b04ac39064a51580d73f31 from peer 833582ff68c94269be364ebd187a75db ({username='slave'} at 127.20.230.65:36529)
I20240503 08:13:43.497973 28413 tablet_copy_service.cc:161] P d09a11a03989421b9f28019dc1c14cc2: Beginning new tablet copy session on tablet 58611f4ce8b04ac39064a51580d73f31 from peer 833582ff68c94269be364ebd187a75db at {username='slave'} at 127.20.230.65:36529: session id = 833582ff68c94269be364ebd187a75db-58611f4ce8b04ac39064a51580d73f31
I20240503 08:13:43.500846 28413 tablet_copy_source_session.cc:205] T 58611f4ce8b04ac39064a51580d73f31 P d09a11a03989421b9f28019dc1c14cc2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:43.502336 5391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58611f4ce8b04ac39064a51580d73f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:43.510828 5391 tablet_copy_client.cc:783] T 58611f4ce8b04ac39064a51580d73f31 P 833582ff68c94269be364ebd187a75db: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:43.511976 5391 tablet_copy_client.cc:647] T 58611f4ce8b04ac39064a51580d73f31 P 833582ff68c94269be364ebd187a75db: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:43.513990 5391 tablet_copy_client.cc:523] T 58611f4ce8b04ac39064a51580d73f31 P 833582ff68c94269be364ebd187a75db: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:43.518072 5391 tablet_bootstrap.cc:492] T 58611f4ce8b04ac39064a51580d73f31 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:13:43.518559 5393 ts_tablet_manager.cc:902] T 200387e6edce4ee9bf11331380923005 P d09a11a03989421b9f28019dc1c14cc2: Initiating tablet copy from peer ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:13:43.577531 5393 tablet_copy_client.cc:308] T 200387e6edce4ee9bf11331380923005 P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.71:41915
I20240503 08:13:43.578137 31048 tablet_copy_service.cc:140] P ced21ca983b749d0b4d4f6a0e2426bf8: Received BeginTabletCopySession request for tablet 200387e6edce4ee9bf11331380923005 from peer d09a11a03989421b9f28019dc1c14cc2 ({username='slave'} at 127.20.230.68:37753)
I20240503 08:13:43.578294 31048 tablet_copy_service.cc:161] P ced21ca983b749d0b4d4f6a0e2426bf8: Beginning new tablet copy session on tablet 200387e6edce4ee9bf11331380923005 from peer d09a11a03989421b9f28019dc1c14cc2 at {username='slave'} at 127.20.230.68:37753: session id = d09a11a03989421b9f28019dc1c14cc2-200387e6edce4ee9bf11331380923005
I20240503 08:13:43.581168 31048 tablet_copy_source_session.cc:205] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:43.581475 5391 tablet_bootstrap.cc:492] T 58611f4ce8b04ac39064a51580d73f31 P 833582ff68c94269be364ebd187a75db: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:43.581845 5391 tablet_bootstrap.cc:492] T 58611f4ce8b04ac39064a51580d73f31 P 833582ff68c94269be364ebd187a75db: Bootstrap complete.
I20240503 08:13:43.582100 5391 ts_tablet_manager.cc:1367] T 58611f4ce8b04ac39064a51580d73f31 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.064s user 0.027s sys 0.021s
I20240503 08:13:43.582911 5391 raft_consensus.cc:348] T 58611f4ce8b04ac39064a51580d73f31 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:43.583108 5391 raft_consensus.cc:729] T 58611f4ce8b04ac39064a51580d73f31 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:13:43.583310 5391 consensus_queue.cc:260] T 58611f4ce8b04ac39064a51580d73f31 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:43.583873 5391 ts_tablet_manager.cc:1398] T 58611f4ce8b04ac39064a51580d73f31 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:43.585287 5393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 200387e6edce4ee9bf11331380923005. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:43.585539 28413 tablet_copy_service.cc:342] P d09a11a03989421b9f28019dc1c14cc2: Request end of tablet copy session 833582ff68c94269be364ebd187a75db-58611f4ce8b04ac39064a51580d73f31 received from {username='slave'} at 127.20.230.65:36529
I20240503 08:13:43.585640 28413 tablet_copy_service.cc:433] P d09a11a03989421b9f28019dc1c14cc2: ending tablet copy session 833582ff68c94269be364ebd187a75db-58611f4ce8b04ac39064a51580d73f31 on tablet 58611f4ce8b04ac39064a51580d73f31 with peer 833582ff68c94269be364ebd187a75db
I20240503 08:13:43.592093 5389 tablet_copy_client.cc:308] T 48075c4a45d24e84bb2b9760710de3a0 P ce1496a47d084365ad786cd09e472a2c: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.71:41915
I20240503 08:13:43.592751 31048 tablet_copy_service.cc:140] P ced21ca983b749d0b4d4f6a0e2426bf8: Received BeginTabletCopySession request for tablet 48075c4a45d24e84bb2b9760710de3a0 from peer ce1496a47d084365ad786cd09e472a2c ({username='slave'} at 127.20.230.70:57987)
I20240503 08:13:43.592902 31048 tablet_copy_service.cc:161] P ced21ca983b749d0b4d4f6a0e2426bf8: Beginning new tablet copy session on tablet 48075c4a45d24e84bb2b9760710de3a0 from peer ce1496a47d084365ad786cd09e472a2c at {username='slave'} at 127.20.230.70:57987: session id = ce1496a47d084365ad786cd09e472a2c-48075c4a45d24e84bb2b9760710de3a0
I20240503 08:13:43.595710 31048 tablet_copy_source_session.cc:205] T 48075c4a45d24e84bb2b9760710de3a0 P ced21ca983b749d0b4d4f6a0e2426bf8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:43.597924 5389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48075c4a45d24e84bb2b9760710de3a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:43.601084 5393 tablet_copy_client.cc:783] T 200387e6edce4ee9bf11331380923005 P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:43.601325 5393 tablet_copy_client.cc:647] T 200387e6edce4ee9bf11331380923005 P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:43.602731 5393 tablet_copy_client.cc:523] T 200387e6edce4ee9bf11331380923005 P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:43.606276 5393 tablet_bootstrap.cc:492] T 200387e6edce4ee9bf11331380923005 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:13:43.614746 5389 tablet_copy_client.cc:783] T 48075c4a45d24e84bb2b9760710de3a0 P ce1496a47d084365ad786cd09e472a2c: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:43.614984 5389 tablet_copy_client.cc:647] T 48075c4a45d24e84bb2b9760710de3a0 P ce1496a47d084365ad786cd09e472a2c: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:43.619680 5393 tablet_bootstrap.cc:492] T 200387e6edce4ee9bf11331380923005 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:43.620069 5393 tablet_bootstrap.cc:492] T 200387e6edce4ee9bf11331380923005 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap complete.
I20240503 08:13:43.620286 5393 ts_tablet_manager.cc:1367] T 200387e6edce4ee9bf11331380923005 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.001s
I20240503 08:13:43.620981 5393 raft_consensus.cc:348] T 200387e6edce4ee9bf11331380923005 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:43.621191 5393 raft_consensus.cc:729] T 200387e6edce4ee9bf11331380923005 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:13:43.621402 5393 consensus_queue.cc:260] T 200387e6edce4ee9bf11331380923005 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:43.621922 5393 ts_tablet_manager.cc:1398] T 200387e6edce4ee9bf11331380923005 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:13:43.622583 31048 tablet_copy_service.cc:342] P ced21ca983b749d0b4d4f6a0e2426bf8: Request end of tablet copy session d09a11a03989421b9f28019dc1c14cc2-200387e6edce4ee9bf11331380923005 received from {username='slave'} at 127.20.230.68:37753
I20240503 08:13:43.622725 31048 tablet_copy_service.cc:433] P ced21ca983b749d0b4d4f6a0e2426bf8: ending tablet copy session d09a11a03989421b9f28019dc1c14cc2-200387e6edce4ee9bf11331380923005 on tablet 200387e6edce4ee9bf11331380923005 with peer d09a11a03989421b9f28019dc1c14cc2
I20240503 08:13:43.639195 5389 tablet_copy_client.cc:523] T 48075c4a45d24e84bb2b9760710de3a0 P ce1496a47d084365ad786cd09e472a2c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:43.655167 5389 tablet_bootstrap.cc:492] T 48075c4a45d24e84bb2b9760710de3a0 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:13:43.682011 5389 tablet_bootstrap.cc:492] T 48075c4a45d24e84bb2b9760710de3a0 P ce1496a47d084365ad786cd09e472a2c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:43.682443 5389 tablet_bootstrap.cc:492] T 48075c4a45d24e84bb2b9760710de3a0 P ce1496a47d084365ad786cd09e472a2c: Bootstrap complete.
I20240503 08:13:43.682663 5389 ts_tablet_manager.cc:1367] T 48075c4a45d24e84bb2b9760710de3a0 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.028s user 0.016s sys 0.000s
I20240503 08:13:43.683252 5389 raft_consensus.cc:348] T 48075c4a45d24e84bb2b9760710de3a0 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:43.683457 5389 raft_consensus.cc:729] T 48075c4a45d24e84bb2b9760710de3a0 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:13:43.684051 5389 consensus_queue.cc:260] T 48075c4a45d24e84bb2b9760710de3a0 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:43.684587 5389 ts_tablet_manager.cc:1398] T 48075c4a45d24e84bb2b9760710de3a0 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:43.685441 31048 tablet_copy_service.cc:342] P ced21ca983b749d0b4d4f6a0e2426bf8: Request end of tablet copy session ce1496a47d084365ad786cd09e472a2c-48075c4a45d24e84bb2b9760710de3a0 received from {username='slave'} at 127.20.230.70:57987
I20240503 08:13:43.685571 31048 tablet_copy_service.cc:433] P ced21ca983b749d0b4d4f6a0e2426bf8: ending tablet copy session ce1496a47d084365ad786cd09e472a2c-48075c4a45d24e84bb2b9760710de3a0 on tablet 48075c4a45d24e84bb2b9760710de3a0 with peer ce1496a47d084365ad786cd09e472a2c
I20240503 08:13:43.839658 30895 raft_consensus.cc:1203] T 48075c4a45d24e84bb2b9760710de3a0 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:43.914888 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:43.919927 28002 raft_consensus.cc:1203] T 58611f4ce8b04ac39064a51580d73f31 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:43.949383 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:43.967880 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:43.975358 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:44.032366 31141 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:44.038894 28389 raft_consensus.cc:1203] T 200387e6edce4ee9bf11331380923005 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:44.065658 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:44.072975 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:44.104974 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:44.238090 28262 consensus_queue.cc:237] T 46d27fa12990405cae5f94976ac7a014 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:13:44.240557 28002 raft_consensus.cc:1261] T 46d27fa12990405cae5f94976ac7a014 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:44.240782 31159 raft_consensus.cc:1261] T 46d27fa12990405cae5f94976ac7a014 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:44.241178 4810 consensus_queue.cc:1035] T 46d27fa12990405cae5f94976ac7a014 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:44.241389 5111 consensus_queue.cc:1035] T 46d27fa12990405cae5f94976ac7a014 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:44.245035 5111 raft_consensus.cc:2932] T 46d27fa12990405cae5f94976ac7a014 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:13:44.246637 28002 raft_consensus.cc:2932] T 46d27fa12990405cae5f94976ac7a014 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:13:44.248728 31159 raft_consensus.cc:2932] T 46d27fa12990405cae5f94976ac7a014 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:13:44.252470 27854 catalog_manager.cc:5484] T 46d27fa12990405cae5f94976ac7a014 P 833582ff68c94269be364ebd187a75db reported cstate change: config changed from index 4 to 5, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) evicted. New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:13:44.267364 28373 tablet_service.cc:1514] Processing DeleteTablet for tablet 46d27fa12990405cae5f94976ac7a014 with delete_type TABLET_DATA_TOMBSTONED (TS d09a11a03989421b9f28019dc1c14cc2 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34786
I20240503 08:13:44.267889 5493 tablet_replica.cc:330] T 46d27fa12990405cae5f94976ac7a014 P d09a11a03989421b9f28019dc1c14cc2: stopping tablet replica
I20240503 08:13:44.268214 5493 raft_consensus.cc:2229] T 46d27fa12990405cae5f94976ac7a014 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:44.268484 5493 raft_consensus.cc:2258] T 46d27fa12990405cae5f94976ac7a014 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:44.269150 5493 ts_tablet_manager.cc:1875] T 46d27fa12990405cae5f94976ac7a014 P d09a11a03989421b9f28019dc1c14cc2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:44.273944 5493 ts_tablet_manager.cc:1888] T 46d27fa12990405cae5f94976ac7a014 P d09a11a03989421b9f28019dc1c14cc2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:13:44.274039 5493 log.cc:1196] T 46d27fa12990405cae5f94976ac7a014 P d09a11a03989421b9f28019dc1c14cc2: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/wals/46d27fa12990405cae5f94976ac7a014
I20240503 08:13:44.274652 27849 catalog_manager.cc:4830] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): tablet 46d27fa12990405cae5f94976ac7a014 (table rebalance_test_table_15 [id=261c96b04fc141ddb92650c24ce76fa8]) successfully deleted
I20240503 08:13:44.518539 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:44.521823 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:44.544148 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:44.545116 31141 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:44.575340 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:44.612113 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:44.677654 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240503 08:13:44.714977 27937 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.71:41915, user_credentials={real_user=slave}} blocked reactor thread for 182455us
I20240503 08:13:44.751657 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:44.893471 30762 consensus_queue.cc:237] T f9746769e2514c65ac8d48ca7df8bca7 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:44.895586 28000 raft_consensus.cc:1261] T f9746769e2514c65ac8d48ca7df8bca7 P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:44.895656 31159 raft_consensus.cc:1261] T f9746769e2514c65ac8d48ca7df8bca7 P b00f244c69df401997ea99d492218a98 [term 2 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:44.896112 5388 consensus_queue.cc:1035] T f9746769e2514c65ac8d48ca7df8bca7 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:44.896569 5388 consensus_queue.cc:1035] T f9746769e2514c65ac8d48ca7df8bca7 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:44.902033 5388 raft_consensus.cc:2932] T f9746769e2514c65ac8d48ca7df8bca7 P 6d1d7de19459435fbde1999e3dfd226b [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:44.902549 31159 raft_consensus.cc:2932] T f9746769e2514c65ac8d48ca7df8bca7 P b00f244c69df401997ea99d492218a98 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:44.908829 27858 catalog_manager.cc:5484] T f9746769e2514c65ac8d48ca7df8bca7 P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: config changed from index 2 to 4, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New cstate: current_term: 2 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:44.913980 30874 tablet_service.cc:1514] Processing DeleteTablet for tablet f9746769e2514c65ac8d48ca7df8bca7 with delete_type TABLET_DATA_TOMBSTONED (TS ce1496a47d084365ad786cd09e472a2c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34426
I20240503 08:13:44.916891 28000 raft_consensus.cc:2932] T f9746769e2514c65ac8d48ca7df8bca7 P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:44.926296 5585 tablet_replica.cc:330] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c: stopping tablet replica
I20240503 08:13:44.926604 5585 raft_consensus.cc:2229] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:44.926791 5585 raft_consensus.cc:2258] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:44.927671 5585 ts_tablet_manager.cc:1875] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:44.933187 5585 ts_tablet_manager.cc:1888] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240503 08:13:44.933297 5585 log.cc:1196] T f9746769e2514c65ac8d48ca7df8bca7 P ce1496a47d084365ad786cd09e472a2c: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal/wals/f9746769e2514c65ac8d48ca7df8bca7
I20240503 08:13:44.933954 27846 catalog_manager.cc:4830] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): tablet f9746769e2514c65ac8d48ca7df8bca7 (table rebalancer_extra_table_36 [id=ae8d4f3f001646efbeb5154b3ce01aac]) successfully deleted
I20240503 08:13:45.262276 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:45.276183 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:45.322336 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:45.338024 31141 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:45.372834 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:45.386003 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:45.431245 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:45.508816 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:45.574124 31028 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "200387e6edce4ee9bf11331380923005"
dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40452
I20240503 08:13:45.574333 31028 raft_consensus.cc:595] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:13:45.646589 5558 raft_consensus.cc:979] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8: : Instructing follower b00f244c69df401997ea99d492218a98 to start an election
I20240503 08:13:45.646735 5558 raft_consensus.cc:1067] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 LEADER]: Signalling peer b00f244c69df401997ea99d492218a98 to start an election
I20240503 08:13:45.647614 31159 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "200387e6edce4ee9bf11331380923005"
dest_uuid: "b00f244c69df401997ea99d492218a98"
from {username='slave'} at 127.20.230.71:59573
I20240503 08:13:45.647856 31159 raft_consensus.cc:482] T 200387e6edce4ee9bf11331380923005 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:45.647948 31159 raft_consensus.cc:3037] T 200387e6edce4ee9bf11331380923005 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:45.650775 31159 raft_consensus.cc:504] T 200387e6edce4ee9bf11331380923005 P b00f244c69df401997ea99d492218a98 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:45.652243 30894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "200387e6edce4ee9bf11331380923005" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:13:45.652412 30894 raft_consensus.cc:3037] T 200387e6edce4ee9bf11331380923005 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:45.654955 30894 raft_consensus.cc:2454] T 200387e6edce4ee9bf11331380923005 P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 2.
I20240503 08:13:45.656632 31028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "200387e6edce4ee9bf11331380923005" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:13:45.656816 31028 raft_consensus.cc:3032] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:45.656883 31028 raft_consensus.cc:729] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Running, Role: LEADER
I20240503 08:13:45.657164 31028 consensus_queue.cc:260] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:45.657619 31028 raft_consensus.cc:3037] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:45.659408 31159 leader_election.cc:288] T 200387e6edce4ee9bf11331380923005 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 election: Requested vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:13:45.660143 31028 raft_consensus.cc:2454] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 2.
I20240503 08:13:45.660405 28389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "200387e6edce4ee9bf11331380923005" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:13:45.660564 31095 leader_election.cc:302] T 200387e6edce4ee9bf11331380923005 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: b00f244c69df401997ea99d492218a98, ce1496a47d084365ad786cd09e472a2c, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:13:45.660598 28389 raft_consensus.cc:3037] T 200387e6edce4ee9bf11331380923005 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:45.660845 5555 raft_consensus.cc:2781] T 200387e6edce4ee9bf11331380923005 P b00f244c69df401997ea99d492218a98 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:45.660955 5555 raft_consensus.cc:686] T 200387e6edce4ee9bf11331380923005 P b00f244c69df401997ea99d492218a98 [term 2 LEADER]: Becoming Leader. State: Replica: b00f244c69df401997ea99d492218a98, State: Running, Role: LEADER
I20240503 08:13:45.661209 5555 consensus_queue.cc:237] T 200387e6edce4ee9bf11331380923005 P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:45.663287 28389 raft_consensus.cc:2454] T 200387e6edce4ee9bf11331380923005 P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 2.
I20240503 08:13:45.676494 27860 catalog_manager.cc:5484] T 200387e6edce4ee9bf11331380923005 P b00f244c69df401997ea99d492218a98 reported cstate change: term changed from 1 to 2, leader changed from ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) to b00f244c69df401997ea99d492218a98 (127.20.230.72). New cstate: current_term: 2 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:45.951540 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:45.960858 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:46.006371 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:46.045325 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:13:46.080317 31028 raft_consensus.cc:1261] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:13:46.080982 5468 consensus_queue.cc:1035] T 200387e6edce4ee9bf11331380923005 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:46.082648 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:46.085122 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:46.127825 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:46.128931 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:46.244395 30895 raft_consensus.cc:1261] T 200387e6edce4ee9bf11331380923005 P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:13:46.244910 5555 consensus_queue.cc:1035] T 200387e6edce4ee9bf11331380923005 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:46.250320 28389 raft_consensus.cc:1261] T 200387e6edce4ee9bf11331380923005 P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:13:46.254127 5468 consensus_queue.cc:1035] T 200387e6edce4ee9bf11331380923005 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:46.538324 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:46.549327 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:46.580338 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:46.618170 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:13:46.630239 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:46.636932 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240503 08:13:46.643499 30830 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.68:46211, user_credentials={real_user=slave}} blocked reactor thread for 114301us
I20240503 08:13:46.675980 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:46.757934 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:46.862450 28262 consensus_queue.cc:237] T 17c6e3293aad4591a9c1395ebffff2e6 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:46.864291 28131 raft_consensus.cc:1261] T 17c6e3293aad4591a9c1395ebffff2e6 P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:46.864369 31027 raft_consensus.cc:1261] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:46.864787 5787 consensus_queue.cc:1035] T 17c6e3293aad4591a9c1395ebffff2e6 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:46.865033 5560 consensus_queue.cc:1035] T 17c6e3293aad4591a9c1395ebffff2e6 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:46.867864 5787 raft_consensus.cc:2932] T 17c6e3293aad4591a9c1395ebffff2e6 P aae860d7ae2b4002bf7794c278f9cc56 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:46.870200 28131 raft_consensus.cc:2932] T 17c6e3293aad4591a9c1395ebffff2e6 P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:46.875277 27857 catalog_manager.cc:5484] T 17c6e3293aad4591a9c1395ebffff2e6 P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: config changed from index 2 to 4, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New cstate: current_term: 2 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:46.888684 31027 raft_consensus.cc:2932] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:46.905289 31141 tablet_service.cc:1514] Processing DeleteTablet for tablet 17c6e3293aad4591a9c1395ebffff2e6 with delete_type TABLET_DATA_TOMBSTONED (TS b00f244c69df401997ea99d492218a98 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45814
I20240503 08:13:46.910722 5842 tablet_replica.cc:330] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98: stopping tablet replica
I20240503 08:13:46.910933 5842 raft_consensus.cc:2229] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:46.911098 5842 raft_consensus.cc:2258] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:46.911659 5842 ts_tablet_manager.cc:1875] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:46.918973 5842 ts_tablet_manager.cc:1888] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240503 08:13:46.919085 5842 log.cc:1196] T 17c6e3293aad4591a9c1395ebffff2e6 P b00f244c69df401997ea99d492218a98: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal/wals/17c6e3293aad4591a9c1395ebffff2e6
I20240503 08:13:46.919876 27846 catalog_manager.cc:4830] TS b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): tablet 17c6e3293aad4591a9c1395ebffff2e6 (table rebalancer_extra_table_52 [id=40dc8c7f01804911a18e5c62a4b6be48]) successfully deleted
I20240503 08:13:47.248037 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:47.248589 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:47.252950 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:47.280815 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:47.299147 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:47.300195 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 08:13:47.302428 27938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.69:34725, user_credentials={real_user=slave}} blocked reactor thread for 48329.5us
W20240503 08:13:47.303196 27936 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.72:40893, user_credentials={real_user=slave}} blocked reactor thread for 50420.5us
I20240503 08:13:47.355859 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:47.375347 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 08:13:47.390206 30829 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.69:34725, user_credentials={real_user=slave}} blocked reactor thread for 64207.7us
I20240503 08:13:47.518654 30895 consensus_queue.cc:237] T df31bd9e544a44f280db2b0d021000c2 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:47.526301 28389 raft_consensus.cc:1261] T df31bd9e544a44f280db2b0d021000c2 P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:47.526675 30761 raft_consensus.cc:1261] T df31bd9e544a44f280db2b0d021000c2 P 6d1d7de19459435fbde1999e3dfd226b [term 2 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:47.526837 5672 consensus_queue.cc:1035] T df31bd9e544a44f280db2b0d021000c2 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:47.527097 5915 consensus_queue.cc:1035] T df31bd9e544a44f280db2b0d021000c2 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:47.529748 5672 raft_consensus.cc:2932] T df31bd9e544a44f280db2b0d021000c2 P ce1496a47d084365ad786cd09e472a2c [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:47.530360 28389 raft_consensus.cc:2932] T df31bd9e544a44f280db2b0d021000c2 P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:47.535153 27857 catalog_manager.cc:5484] T df31bd9e544a44f280db2b0d021000c2 P ce1496a47d084365ad786cd09e472a2c reported cstate change: config changed from index 2 to 4, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) evicted. New cstate: current_term: 2 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:47.562569 27983 tablet_service.cc:1514] Processing DeleteTablet for tablet df31bd9e544a44f280db2b0d021000c2 with delete_type TABLET_DATA_TOMBSTONED (TS 833582ff68c94269be364ebd187a75db not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55610
I20240503 08:13:47.563069 5942 tablet_replica.cc:330] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:13:47.563254 5942 raft_consensus.cc:2229] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:47.563447 5942 raft_consensus.cc:2258] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:47.563975 5942 ts_tablet_manager.cc:1875] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:47.570854 30761 raft_consensus.cc:2932] T df31bd9e544a44f280db2b0d021000c2 P 6d1d7de19459435fbde1999e3dfd226b [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:47.573585 5942 ts_tablet_manager.cc:1888] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240503 08:13:47.573678 5942 log.cc:1196] T df31bd9e544a44f280db2b0d021000c2 P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/df31bd9e544a44f280db2b0d021000c2
I20240503 08:13:47.576475 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet df31bd9e544a44f280db2b0d021000c2 (table rebalancer_extra_table_12 [id=7f3bcd2f061e424892943977bcde8c80]) successfully deleted
I20240503 08:13:47.854852 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:47.878777 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:47.881402 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:47.882356 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:47.895977 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:47.909018 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:47.974440 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:13:48.011675 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:48.156237 28389 consensus_queue.cc:237] T 58611f4ce8b04ac39064a51580d73f31 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:48.158172 28000 raft_consensus.cc:1261] T 58611f4ce8b04ac39064a51580d73f31 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:48.158305 30893 raft_consensus.cc:1261] T 58611f4ce8b04ac39064a51580d73f31 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:48.158629 5913 consensus_queue.cc:1035] T 58611f4ce8b04ac39064a51580d73f31 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:48.159125 5913 consensus_queue.cc:1035] T 58611f4ce8b04ac39064a51580d73f31 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:48.160959 5409 raft_consensus.cc:2932] T 58611f4ce8b04ac39064a51580d73f31 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:48.161510 28000 raft_consensus.cc:2932] T 58611f4ce8b04ac39064a51580d73f31 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:48.163847 30895 raft_consensus.cc:2932] T 58611f4ce8b04ac39064a51580d73f31 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:48.167984 27854 catalog_manager.cc:5484] T 58611f4ce8b04ac39064a51580d73f31 P 833582ff68c94269be364ebd187a75db reported cstate change: config changed from index 2 to 3, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:48.180701 30742 tablet_service.cc:1514] Processing DeleteTablet for tablet 58611f4ce8b04ac39064a51580d73f31 with delete_type TABLET_DATA_TOMBSTONED (TS 6d1d7de19459435fbde1999e3dfd226b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46794
I20240503 08:13:48.190136 6027 tablet_replica.cc:330] T 58611f4ce8b04ac39064a51580d73f31 P 6d1d7de19459435fbde1999e3dfd226b: stopping tablet replica
I20240503 08:13:48.190313 6027 raft_consensus.cc:2229] T 58611f4ce8b04ac39064a51580d73f31 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:48.190481 6027 raft_consensus.cc:2258] T 58611f4ce8b04ac39064a51580d73f31 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:48.191085 6027 ts_tablet_manager.cc:1875] T 58611f4ce8b04ac39064a51580d73f31 P 6d1d7de19459435fbde1999e3dfd226b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:48.196442 6027 ts_tablet_manager.cc:1888] T 58611f4ce8b04ac39064a51580d73f31 P 6d1d7de19459435fbde1999e3dfd226b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:13:48.196545 6027 log.cc:1196] T 58611f4ce8b04ac39064a51580d73f31 P 6d1d7de19459435fbde1999e3dfd226b: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal/wals/58611f4ce8b04ac39064a51580d73f31
I20240503 08:13:48.197206 27848 catalog_manager.cc:4830] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): tablet 58611f4ce8b04ac39064a51580d73f31 (table rebalancer_extra_table_44 [id=2af0a6bdbc404add8684cd11068a06f9]) successfully deleted
I20240503 08:13:48.321286 28261 consensus_queue.cc:237] T 41aa0e288ca4469ba34a07ff159ace91 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:48.323365 28390 raft_consensus.cc:1261] T 41aa0e288ca4469ba34a07ff159ace91 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:13:48.323577 28197 consensus_peers.cc:485] T 41aa0e288ca4469ba34a07ff159ace91 P aae860d7ae2b4002bf7794c278f9cc56 -> Peer ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): Couldn't send request to peer ced21ca983b749d0b4d4f6a0e2426bf8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41aa0e288ca4469ba34a07ff159ace91. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:48.323807 28133 raft_consensus.cc:1261] T 41aa0e288ca4469ba34a07ff159ace91 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:48.324304 6004 consensus_queue.cc:1035] T 41aa0e288ca4469ba34a07ff159ace91 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:48.324563 5559 consensus_queue.cc:1035] T 41aa0e288ca4469ba34a07ff159ace91 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:48.326938 5559 raft_consensus.cc:2932] T 41aa0e288ca4469ba34a07ff159ace91 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:13:48.327481 28133 raft_consensus.cc:2932] T 41aa0e288ca4469ba34a07ff159ace91 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:13:48.330904 28390 raft_consensus.cc:2932] T 41aa0e288ca4469ba34a07ff159ace91 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:13:48.333989 27862 catalog_manager.cc:5484] T 41aa0e288ca4469ba34a07ff159ace91 P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: config changed from index -1 to 4, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) added. New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:13:48.364662 28390 consensus_queue.cc:237] T 1fca3d7f4b734eceb210e1186478eb72 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:48.366602 28261 raft_consensus.cc:1261] T 1fca3d7f4b734eceb210e1186478eb72 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:48.366845 30895 raft_consensus.cc:1261] T 1fca3d7f4b734eceb210e1186478eb72 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:13:48.366899 28328 consensus_peers.cc:485] T 1fca3d7f4b734eceb210e1186478eb72 P d09a11a03989421b9f28019dc1c14cc2 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1fca3d7f4b734eceb210e1186478eb72. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:48.367331 6098 consensus_queue.cc:1035] T 1fca3d7f4b734eceb210e1186478eb72 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:48.367574 5409 consensus_queue.cc:1035] T 1fca3d7f4b734eceb210e1186478eb72 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:48.370172 5409 raft_consensus.cc:2932] T 1fca3d7f4b734eceb210e1186478eb72 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:48.370812 28261 raft_consensus.cc:2932] T 1fca3d7f4b734eceb210e1186478eb72 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:48.374074 30895 raft_consensus.cc:2932] T 1fca3d7f4b734eceb210e1186478eb72 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:48.379196 27854 catalog_manager.cc:5484] T 1fca3d7f4b734eceb210e1186478eb72 P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:48.417721 31161 consensus_queue.cc:237] T 98b3c906cfe54f49bb88ce5e00e3cc04 P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:48.419812 31028 raft_consensus.cc:1261] T 98b3c906cfe54f49bb88ce5e00e3cc04 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:48.419937 30895 raft_consensus.cc:1261] T 98b3c906cfe54f49bb88ce5e00e3cc04 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:13:48.420466 31096 consensus_peers.cc:485] T 98b3c906cfe54f49bb88ce5e00e3cc04 P b00f244c69df401997ea99d492218a98 -> Peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): Couldn't send request to peer 6d1d7de19459435fbde1999e3dfd226b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 98b3c906cfe54f49bb88ce5e00e3cc04. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:48.420532 5743 consensus_queue.cc:1035] T 98b3c906cfe54f49bb88ce5e00e3cc04 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:48.420773 6119 consensus_queue.cc:1035] T 98b3c906cfe54f49bb88ce5e00e3cc04 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:48.423048 6119 raft_consensus.cc:2932] T 98b3c906cfe54f49bb88ce5e00e3cc04 P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:48.423642 31028 raft_consensus.cc:2932] T 98b3c906cfe54f49bb88ce5e00e3cc04 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:48.429845 27856 catalog_manager.cc:5484] T 98b3c906cfe54f49bb88ce5e00e3cc04 P b00f244c69df401997ea99d492218a98 reported cstate change: config changed from index -1 to 2, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) added. New cstate: current_term: 1 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:48.430018 30895 raft_consensus.cc:2932] T 98b3c906cfe54f49bb88ce5e00e3cc04 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:48.468071 30894 consensus_queue.cc:237] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:48.470279 30761 raft_consensus.cc:1261] T 3bc8f4968b9743baab5d25c8c7237c9c P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:48.470811 6138 consensus_queue.cc:1035] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:48.472642 31161 raft_consensus.cc:1261] T 3bc8f4968b9743baab5d25c8c7237c9c P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:13:48.473623 30828 consensus_peers.cc:485] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3bc8f4968b9743baab5d25c8c7237c9c. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:48.474676 6138 consensus_queue.cc:1035] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:48.477100 5672 raft_consensus.cc:2932] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:48.477512 31159 raft_consensus.cc:2932] T 3bc8f4968b9743baab5d25c8c7237c9c P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:48.480965 30761 raft_consensus.cc:2932] T 3bc8f4968b9743baab5d25c8c7237c9c P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:48.483002 27862 catalog_manager.cc:5484] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c reported cstate change: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New cstate: current_term: 1 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:48.525419 28390 consensus_queue.cc:237] T 76d79d8be9e94e99a7ba51d903ca8a91 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:48.527946 31028 raft_consensus.cc:1261] T 76d79d8be9e94e99a7ba51d903ca8a91 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:48.528450 6101 consensus_queue.cc:1035] T 76d79d8be9e94e99a7ba51d903ca8a91 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 08:13:48.529043 28328 consensus_peers.cc:485] T 76d79d8be9e94e99a7ba51d903ca8a91 P d09a11a03989421b9f28019dc1c14cc2 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 76d79d8be9e94e99a7ba51d903ca8a91. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:48.530220 30894 raft_consensus.cc:1261] T 76d79d8be9e94e99a7ba51d903ca8a91 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:48.531386 6098 consensus_queue.cc:1035] T 76d79d8be9e94e99a7ba51d903ca8a91 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:48.553845 5409 raft_consensus.cc:2932] T 76d79d8be9e94e99a7ba51d903ca8a91 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:48.558207 31028 raft_consensus.cc:2932] T 76d79d8be9e94e99a7ba51d903ca8a91 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:48.559110 30894 raft_consensus.cc:2932] T 76d79d8be9e94e99a7ba51d903ca8a91 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:48.560508 27854 catalog_manager.cc:5484] T 76d79d8be9e94e99a7ba51d903ca8a91 P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:48.610599 28390 consensus_queue.cc:237] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:48.613387 30894 raft_consensus.cc:1261] T 49b145f6946647248c967cf4badf8b40 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:48.613641 31028 raft_consensus.cc:1261] T 49b145f6946647248c967cf4badf8b40 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:13:48.613678 28329 consensus_peers.cc:485] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2 -> Peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Couldn't send request to peer 833582ff68c94269be364ebd187a75db. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49b145f6946647248c967cf4badf8b40. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:48.614204 5409 consensus_queue.cc:1035] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:48.614442 6098 consensus_queue.cc:1035] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:48.618055 6098 raft_consensus.cc:2932] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:48.618626 31028 raft_consensus.cc:2932] T 49b145f6946647248c967cf4badf8b40 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:48.627386 30894 raft_consensus.cc:2932] T 49b145f6946647248c967cf4badf8b40 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:48.629281 27862 catalog_manager.cc:5484] T 49b145f6946647248c967cf4badf8b40 P ced21ca983b749d0b4d4f6a0e2426bf8 reported cstate change: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:48.740149 30761 consensus_queue.cc:237] T 41e26e13355d4e0cb56bb526415a3998 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:48.742449 30894 raft_consensus.cc:1261] T 41e26e13355d4e0cb56bb526415a3998 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:13:48.742689 30696 consensus_peers.cc:485] T 41e26e13355d4e0cb56bb526415a3998 P 6d1d7de19459435fbde1999e3dfd226b -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41e26e13355d4e0cb56bb526415a3998. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:48.742722 31028 raft_consensus.cc:1261] T 41e26e13355d4e0cb56bb526415a3998 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:48.743162 5388 consensus_queue.cc:1035] T 41e26e13355d4e0cb56bb526415a3998 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:48.743399 5883 consensus_queue.cc:1035] T 41e26e13355d4e0cb56bb526415a3998 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:48.753526 5388 raft_consensus.cc:2932] T 41e26e13355d4e0cb56bb526415a3998 P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:48.755209 31028 raft_consensus.cc:2932] T 41e26e13355d4e0cb56bb526415a3998 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:48.758317 30894 raft_consensus.cc:2932] T 41e26e13355d4e0cb56bb526415a3998 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:48.759642 27856 catalog_manager.cc:5484] T 41e26e13355d4e0cb56bb526415a3998 P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:48.799094 27999 consensus_queue.cc:237] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:48.801240 30761 raft_consensus.cc:1261] T 9fbe2eaba5344dfaa29e78a0d73f224a P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:48.801602 31028 raft_consensus.cc:1261] T 9fbe2eaba5344dfaa29e78a0d73f224a P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:48.802772 6048 consensus_queue.cc:1035] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 08:13:48.803889 27937 consensus_peers.cc:485] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9fbe2eaba5344dfaa29e78a0d73f224a. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:48.803905 5907 consensus_queue.cc:1035] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:48.814564 5907 raft_consensus.cc:2932] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:48.815325 30761 raft_consensus.cc:2932] T 9fbe2eaba5344dfaa29e78a0d73f224a P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:48.817106 6217 ts_tablet_manager.cc:902] T 41aa0e288ca4469ba34a07ff159ace91 P ced21ca983b749d0b4d4f6a0e2426bf8: Initiating tablet copy from peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:13:48.823343 27862 catalog_manager.cc:5484] T 9fbe2eaba5344dfaa29e78a0d73f224a P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:48.828760 31028 raft_consensus.cc:2932] T 9fbe2eaba5344dfaa29e78a0d73f224a P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:48.840606 6225 ts_tablet_manager.cc:902] T 98b3c906cfe54f49bb88ce5e00e3cc04 P 6d1d7de19459435fbde1999e3dfd226b: Initiating tablet copy from peer b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:13:48.845233 6217 tablet_copy_client.cc:308] T 41aa0e288ca4469ba34a07ff159ace91 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.67:39919
I20240503 08:13:48.845976 28283 tablet_copy_service.cc:140] P aae860d7ae2b4002bf7794c278f9cc56: Received BeginTabletCopySession request for tablet 41aa0e288ca4469ba34a07ff159ace91 from peer ced21ca983b749d0b4d4f6a0e2426bf8 ({username='slave'} at 127.20.230.71:55281)
I20240503 08:13:48.846122 28283 tablet_copy_service.cc:161] P aae860d7ae2b4002bf7794c278f9cc56: Beginning new tablet copy session on tablet 41aa0e288ca4469ba34a07ff159ace91 from peer ced21ca983b749d0b4d4f6a0e2426bf8 at {username='slave'} at 127.20.230.71:55281: session id = ced21ca983b749d0b4d4f6a0e2426bf8-41aa0e288ca4469ba34a07ff159ace91
I20240503 08:13:48.849071 28283 tablet_copy_source_session.cc:205] T 41aa0e288ca4469ba34a07ff159ace91 P aae860d7ae2b4002bf7794c278f9cc56: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:48.851341 6217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41aa0e288ca4469ba34a07ff159ace91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:48.857163 6217 tablet_copy_client.cc:783] T 41aa0e288ca4469ba34a07ff159ace91 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:48.857481 6217 tablet_copy_client.cc:647] T 41aa0e288ca4469ba34a07ff159ace91 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:48.859243 6217 tablet_copy_client.cc:523] T 41aa0e288ca4469ba34a07ff159ace91 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:48.861917 6217 tablet_bootstrap.cc:492] T 41aa0e288ca4469ba34a07ff159ace91 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:13:48.866816 6225 tablet_copy_client.cc:308] T 98b3c906cfe54f49bb88ce5e00e3cc04 P 6d1d7de19459435fbde1999e3dfd226b: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.72:40893
I20240503 08:13:48.867751 31181 tablet_copy_service.cc:140] P b00f244c69df401997ea99d492218a98: Received BeginTabletCopySession request for tablet 98b3c906cfe54f49bb88ce5e00e3cc04 from peer 6d1d7de19459435fbde1999e3dfd226b ({username='slave'} at 127.20.230.69:43821)
I20240503 08:13:48.867887 31181 tablet_copy_service.cc:161] P b00f244c69df401997ea99d492218a98: Beginning new tablet copy session on tablet 98b3c906cfe54f49bb88ce5e00e3cc04 from peer 6d1d7de19459435fbde1999e3dfd226b at {username='slave'} at 127.20.230.69:43821: session id = 6d1d7de19459435fbde1999e3dfd226b-98b3c906cfe54f49bb88ce5e00e3cc04
I20240503 08:13:48.869941 31181 tablet_copy_source_session.cc:205] T 98b3c906cfe54f49bb88ce5e00e3cc04 P b00f244c69df401997ea99d492218a98: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:48.871111 6225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98b3c906cfe54f49bb88ce5e00e3cc04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:48.876442 6225 tablet_copy_client.cc:783] T 98b3c906cfe54f49bb88ce5e00e3cc04 P 6d1d7de19459435fbde1999e3dfd226b: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:48.876646 6225 tablet_copy_client.cc:647] T 98b3c906cfe54f49bb88ce5e00e3cc04 P 6d1d7de19459435fbde1999e3dfd226b: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:48.879794 6225 tablet_copy_client.cc:523] T 98b3c906cfe54f49bb88ce5e00e3cc04 P 6d1d7de19459435fbde1999e3dfd226b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:48.883234 6225 tablet_bootstrap.cc:492] T 98b3c906cfe54f49bb88ce5e00e3cc04 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:13:48.895499 6239 ts_tablet_manager.cc:902] T 3bc8f4968b9743baab5d25c8c7237c9c P aae860d7ae2b4002bf7794c278f9cc56: Initiating tablet copy from peer ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:13:48.902715 6225 tablet_bootstrap.cc:492] T 98b3c906cfe54f49bb88ce5e00e3cc04 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:48.903194 6225 tablet_bootstrap.cc:492] T 98b3c906cfe54f49bb88ce5e00e3cc04 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap complete.
I20240503 08:13:48.903488 6225 ts_tablet_manager.cc:1367] T 98b3c906cfe54f49bb88ce5e00e3cc04 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.009s
I20240503 08:13:48.904182 6225 raft_consensus.cc:348] T 98b3c906cfe54f49bb88ce5e00e3cc04 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:48.904496 6225 raft_consensus.cc:729] T 98b3c906cfe54f49bb88ce5e00e3cc04 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:13:48.904817 6225 consensus_queue.cc:260] T 98b3c906cfe54f49bb88ce5e00e3cc04 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:48.905493 6225 ts_tablet_manager.cc:1398] T 98b3c906cfe54f49bb88ce5e00e3cc04 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:48.913857 31181 tablet_copy_service.cc:342] P b00f244c69df401997ea99d492218a98: Request end of tablet copy session 6d1d7de19459435fbde1999e3dfd226b-98b3c906cfe54f49bb88ce5e00e3cc04 received from {username='slave'} at 127.20.230.69:43821
I20240503 08:13:48.913976 31181 tablet_copy_service.cc:433] P b00f244c69df401997ea99d492218a98: ending tablet copy session 6d1d7de19459435fbde1999e3dfd226b-98b3c906cfe54f49bb88ce5e00e3cc04 on tablet 98b3c906cfe54f49bb88ce5e00e3cc04 with peer 6d1d7de19459435fbde1999e3dfd226b
I20240503 08:13:48.920073 6243 ts_tablet_manager.cc:902] T 76d79d8be9e94e99a7ba51d903ca8a91 P 1b411c6bb25549368041773d7b710d48: Initiating tablet copy from peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:13:48.932400 6239 tablet_copy_client.cc:308] T 3bc8f4968b9743baab5d25c8c7237c9c P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.70:33747
I20240503 08:13:48.933051 30915 tablet_copy_service.cc:140] P ce1496a47d084365ad786cd09e472a2c: Received BeginTabletCopySession request for tablet 3bc8f4968b9743baab5d25c8c7237c9c from peer aae860d7ae2b4002bf7794c278f9cc56 ({username='slave'} at 127.20.230.67:50141)
I20240503 08:13:48.933207 30915 tablet_copy_service.cc:161] P ce1496a47d084365ad786cd09e472a2c: Beginning new tablet copy session on tablet 3bc8f4968b9743baab5d25c8c7237c9c from peer aae860d7ae2b4002bf7794c278f9cc56 at {username='slave'} at 127.20.230.67:50141: session id = aae860d7ae2b4002bf7794c278f9cc56-3bc8f4968b9743baab5d25c8c7237c9c
I20240503 08:13:48.934741 6217 tablet_bootstrap.cc:492] T 41aa0e288ca4469ba34a07ff159ace91 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:48.935178 6217 tablet_bootstrap.cc:492] T 41aa0e288ca4469ba34a07ff159ace91 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap complete.
I20240503 08:13:48.935462 6217 ts_tablet_manager.cc:1367] T 41aa0e288ca4469ba34a07ff159ace91 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.074s user 0.046s sys 0.010s
I20240503 08:13:48.935853 30915 tablet_copy_source_session.cc:205] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:48.936192 6217 raft_consensus.cc:348] T 41aa0e288ca4469ba34a07ff159ace91 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:48.936417 6217 raft_consensus.cc:729] T 41aa0e288ca4469ba34a07ff159ace91 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:13:48.937274 6239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bc8f4968b9743baab5d25c8c7237c9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:48.938227 6217 consensus_queue.cc:260] T 41aa0e288ca4469ba34a07ff159ace91 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:48.938652 6243 tablet_copy_client.cc:308] T 76d79d8be9e94e99a7ba51d903ca8a91 P 1b411c6bb25549368041773d7b710d48: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.68:46211
I20240503 08:13:48.938761 6217 ts_tablet_manager.cc:1398] T 41aa0e288ca4469ba34a07ff159ace91 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:13:48.939319 28283 tablet_copy_service.cc:342] P aae860d7ae2b4002bf7794c278f9cc56: Request end of tablet copy session ced21ca983b749d0b4d4f6a0e2426bf8-41aa0e288ca4469ba34a07ff159ace91 received from {username='slave'} at 127.20.230.71:55281
I20240503 08:13:48.939425 28283 tablet_copy_service.cc:433] P aae860d7ae2b4002bf7794c278f9cc56: ending tablet copy session ced21ca983b749d0b4d4f6a0e2426bf8-41aa0e288ca4469ba34a07ff159ace91 on tablet 41aa0e288ca4469ba34a07ff159ace91 with peer ced21ca983b749d0b4d4f6a0e2426bf8
I20240503 08:13:48.939903 28413 tablet_copy_service.cc:140] P d09a11a03989421b9f28019dc1c14cc2: Received BeginTabletCopySession request for tablet 76d79d8be9e94e99a7ba51d903ca8a91 from peer 1b411c6bb25549368041773d7b710d48 ({username='slave'} at 127.20.230.66:43721)
I20240503 08:13:48.940021 28413 tablet_copy_service.cc:161] P d09a11a03989421b9f28019dc1c14cc2: Beginning new tablet copy session on tablet 76d79d8be9e94e99a7ba51d903ca8a91 from peer 1b411c6bb25549368041773d7b710d48 at {username='slave'} at 127.20.230.66:43721: session id = 1b411c6bb25549368041773d7b710d48-76d79d8be9e94e99a7ba51d903ca8a91
I20240503 08:13:48.942865 28413 tablet_copy_source_session.cc:205] T 76d79d8be9e94e99a7ba51d903ca8a91 P d09a11a03989421b9f28019dc1c14cc2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:48.943923 6243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76d79d8be9e94e99a7ba51d903ca8a91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:48.951555 6243 tablet_copy_client.cc:783] T 76d79d8be9e94e99a7ba51d903ca8a91 P 1b411c6bb25549368041773d7b710d48: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:48.951581 30762 consensus_queue.cc:237] T 01c4229dfd25458296e85e412eae13bf P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:48.951889 6243 tablet_copy_client.cc:647] T 76d79d8be9e94e99a7ba51d903ca8a91 P 1b411c6bb25549368041773d7b710d48: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:48.953629 6243 tablet_copy_client.cc:523] T 76d79d8be9e94e99a7ba51d903ca8a91 P 1b411c6bb25549368041773d7b710d48: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:48.954598 6239 tablet_copy_client.cc:783] T 3bc8f4968b9743baab5d25c8c7237c9c P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:48.954814 6239 tablet_copy_client.cc:647] T 3bc8f4968b9743baab5d25c8c7237c9c P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Starting download of 1 WAL segments...
W20240503 08:13:48.954886 30697 consensus_peers.cc:485] T 01c4229dfd25458296e85e412eae13bf P 6d1d7de19459435fbde1999e3dfd226b -> Peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): Couldn't send request to peer d09a11a03989421b9f28019dc1c14cc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01c4229dfd25458296e85e412eae13bf. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:48.954923 27999 raft_consensus.cc:1261] T 01c4229dfd25458296e85e412eae13bf P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:48.955612 5388 consensus_queue.cc:1035] T 01c4229dfd25458296e85e412eae13bf P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:48.957609 6239 tablet_copy_client.cc:523] T 3bc8f4968b9743baab5d25c8c7237c9c P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:48.957787 31159 raft_consensus.cc:1261] T 01c4229dfd25458296e85e412eae13bf P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:48.958674 6243 tablet_bootstrap.cc:492] T 76d79d8be9e94e99a7ba51d903ca8a91 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:13:48.958654 5388 consensus_queue.cc:1035] T 01c4229dfd25458296e85e412eae13bf P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:48.960420 6239 tablet_bootstrap.cc:492] T 3bc8f4968b9743baab5d25c8c7237c9c P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:13:48.962574 5388 raft_consensus.cc:2932] T 01c4229dfd25458296e85e412eae13bf P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:48.964385 6243 tablet_bootstrap.cc:492] T 76d79d8be9e94e99a7ba51d903ca8a91 P 1b411c6bb25549368041773d7b710d48: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:48.964731 6243 tablet_bootstrap.cc:492] T 76d79d8be9e94e99a7ba51d903ca8a91 P 1b411c6bb25549368041773d7b710d48: Bootstrap complete.
I20240503 08:13:48.964938 6243 ts_tablet_manager.cc:1367] T 76d79d8be9e94e99a7ba51d903ca8a91 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20240503 08:13:48.966109 6243 raft_consensus.cc:348] T 76d79d8be9e94e99a7ba51d903ca8a91 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:48.966257 6243 raft_consensus.cc:729] T 76d79d8be9e94e99a7ba51d903ca8a91 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:13:48.966676 6239 tablet_bootstrap.cc:492] T 3bc8f4968b9743baab5d25c8c7237c9c P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:48.967283 6239 tablet_bootstrap.cc:492] T 3bc8f4968b9743baab5d25c8c7237c9c P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap complete.
I20240503 08:13:48.966892 6243 consensus_queue.cc:260] T 76d79d8be9e94e99a7ba51d903ca8a91 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:48.967504 6239 ts_tablet_manager.cc:1367] T 3bc8f4968b9743baab5d25c8c7237c9c P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.002s
I20240503 08:13:48.968031 6243 ts_tablet_manager.cc:1398] T 76d79d8be9e94e99a7ba51d903ca8a91 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.003s user 0.002s sys 0.002s
I20240503 08:13:48.968108 6239 raft_consensus.cc:348] T 3bc8f4968b9743baab5d25c8c7237c9c P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:48.968297 6239 raft_consensus.cc:729] T 3bc8f4968b9743baab5d25c8c7237c9c P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:13:48.968530 6239 consensus_queue.cc:260] T 3bc8f4968b9743baab5d25c8c7237c9c P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:48.968824 28413 tablet_copy_service.cc:342] P d09a11a03989421b9f28019dc1c14cc2: Request end of tablet copy session 1b411c6bb25549368041773d7b710d48-76d79d8be9e94e99a7ba51d903ca8a91 received from {username='slave'} at 127.20.230.66:43721
I20240503 08:13:48.968917 28413 tablet_copy_service.cc:433] P d09a11a03989421b9f28019dc1c14cc2: ending tablet copy session 1b411c6bb25549368041773d7b710d48-76d79d8be9e94e99a7ba51d903ca8a91 on tablet 76d79d8be9e94e99a7ba51d903ca8a91 with peer 1b411c6bb25549368041773d7b710d48
I20240503 08:13:48.970322 31159 raft_consensus.cc:2932] T 01c4229dfd25458296e85e412eae13bf P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:48.974485 6239 ts_tablet_manager.cc:1398] T 3bc8f4968b9743baab5d25c8c7237c9c P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:13:48.976811 30915 tablet_copy_service.cc:342] P ce1496a47d084365ad786cd09e472a2c: Request end of tablet copy session aae860d7ae2b4002bf7794c278f9cc56-3bc8f4968b9743baab5d25c8c7237c9c received from {username='slave'} at 127.20.230.67:50141
I20240503 08:13:48.976914 30915 tablet_copy_service.cc:433] P ce1496a47d084365ad786cd09e472a2c: ending tablet copy session aae860d7ae2b4002bf7794c278f9cc56-3bc8f4968b9743baab5d25c8c7237c9c on tablet 3bc8f4968b9743baab5d25c8c7237c9c with peer aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:13:48.979532 27999 raft_consensus.cc:2932] T 01c4229dfd25458296e85e412eae13bf P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:48.981026 27862 catalog_manager.cc:5484] T 01c4229dfd25458296e85e412eae13bf P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:48.987906 6243 ts_tablet_manager.cc:902] T 1fca3d7f4b734eceb210e1186478eb72 P 1b411c6bb25549368041773d7b710d48: Initiating tablet copy from peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:13:49.030797 6243 tablet_copy_client.cc:308] T 1fca3d7f4b734eceb210e1186478eb72 P 1b411c6bb25549368041773d7b710d48: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.68:46211
I20240503 08:13:49.031872 28413 tablet_copy_service.cc:140] P d09a11a03989421b9f28019dc1c14cc2: Received BeginTabletCopySession request for tablet 1fca3d7f4b734eceb210e1186478eb72 from peer 1b411c6bb25549368041773d7b710d48 ({username='slave'} at 127.20.230.66:43721)
I20240503 08:13:49.032001 28413 tablet_copy_service.cc:161] P d09a11a03989421b9f28019dc1c14cc2: Beginning new tablet copy session on tablet 1fca3d7f4b734eceb210e1186478eb72 from peer 1b411c6bb25549368041773d7b710d48 at {username='slave'} at 127.20.230.66:43721: session id = 1b411c6bb25549368041773d7b710d48-1fca3d7f4b734eceb210e1186478eb72
I20240503 08:13:49.034994 28413 tablet_copy_source_session.cc:205] T 1fca3d7f4b734eceb210e1186478eb72 P d09a11a03989421b9f28019dc1c14cc2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:49.036911 6243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fca3d7f4b734eceb210e1186478eb72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:49.042820 6243 tablet_copy_client.cc:783] T 1fca3d7f4b734eceb210e1186478eb72 P 1b411c6bb25549368041773d7b710d48: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:49.043015 6243 tablet_copy_client.cc:647] T 1fca3d7f4b734eceb210e1186478eb72 P 1b411c6bb25549368041773d7b710d48: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:49.044317 6243 tablet_copy_client.cc:523] T 1fca3d7f4b734eceb210e1186478eb72 P 1b411c6bb25549368041773d7b710d48: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:49.047956 6243 tablet_bootstrap.cc:492] T 1fca3d7f4b734eceb210e1186478eb72 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:13:49.060577 6243 tablet_bootstrap.cc:492] T 1fca3d7f4b734eceb210e1186478eb72 P 1b411c6bb25549368041773d7b710d48: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:49.060932 6243 tablet_bootstrap.cc:492] T 1fca3d7f4b734eceb210e1186478eb72 P 1b411c6bb25549368041773d7b710d48: Bootstrap complete.
I20240503 08:13:49.061156 6243 ts_tablet_manager.cc:1367] T 1fca3d7f4b734eceb210e1186478eb72 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.001s
I20240503 08:13:49.061826 6243 raft_consensus.cc:348] T 1fca3d7f4b734eceb210e1186478eb72 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:49.062026 6243 raft_consensus.cc:729] T 1fca3d7f4b734eceb210e1186478eb72 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:13:49.062258 6243 consensus_queue.cc:260] T 1fca3d7f4b734eceb210e1186478eb72 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:49.062919 6243 ts_tablet_manager.cc:1398] T 1fca3d7f4b734eceb210e1186478eb72 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 08:13:49.063489 28413 tablet_copy_service.cc:342] P d09a11a03989421b9f28019dc1c14cc2: Request end of tablet copy session 1b411c6bb25549368041773d7b710d48-1fca3d7f4b734eceb210e1186478eb72 received from {username='slave'} at 127.20.230.66:43721
I20240503 08:13:49.063584 28413 tablet_copy_service.cc:433] P d09a11a03989421b9f28019dc1c14cc2: ending tablet copy session 1b411c6bb25549368041773d7b710d48-1fca3d7f4b734eceb210e1186478eb72 on tablet 1fca3d7f4b734eceb210e1186478eb72 with peer 1b411c6bb25549368041773d7b710d48
I20240503 08:13:49.067541 28390 consensus_queue.cc:237] T ab7b01c2e05a430a9873398268b1e41f P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:49.073405 31028 raft_consensus.cc:1261] T ab7b01c2e05a430a9873398268b1e41f P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:49.074013 5913 consensus_queue.cc:1035] T ab7b01c2e05a430a9873398268b1e41f P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 08:13:49.074687 28328 consensus_peers.cc:485] T ab7b01c2e05a430a9873398268b1e41f P d09a11a03989421b9f28019dc1c14cc2 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ab7b01c2e05a430a9873398268b1e41f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:49.076071 30894 raft_consensus.cc:1261] T ab7b01c2e05a430a9873398268b1e41f P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:49.076588 6101 consensus_queue.cc:1035] T ab7b01c2e05a430a9873398268b1e41f P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:49.101927 6101 raft_consensus.cc:2932] T ab7b01c2e05a430a9873398268b1e41f P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:49.102541 31028 raft_consensus.cc:2932] T ab7b01c2e05a430a9873398268b1e41f P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:49.108246 30895 raft_consensus.cc:2932] T ab7b01c2e05a430a9873398268b1e41f P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:49.108518 27854 catalog_manager.cc:5484] T ab7b01c2e05a430a9873398268b1e41f P ced21ca983b749d0b4d4f6a0e2426bf8 reported cstate change: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:49.136106 6286 ts_tablet_manager.cc:902] T 49b145f6946647248c967cf4badf8b40 P 833582ff68c94269be364ebd187a75db: Initiating tablet copy from peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:13:49.140213 6243 ts_tablet_manager.cc:902] T 41e26e13355d4e0cb56bb526415a3998 P 1b411c6bb25549368041773d7b710d48: Initiating tablet copy from peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:13:49.146683 6243 tablet_copy_client.cc:308] T 41e26e13355d4e0cb56bb526415a3998 P 1b411c6bb25549368041773d7b710d48: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.69:34725
I20240503 08:13:49.147580 30783 tablet_copy_service.cc:140] P 6d1d7de19459435fbde1999e3dfd226b: Received BeginTabletCopySession request for tablet 41e26e13355d4e0cb56bb526415a3998 from peer 1b411c6bb25549368041773d7b710d48 ({username='slave'} at 127.20.230.66:42445)
I20240503 08:13:49.147716 30783 tablet_copy_service.cc:161] P 6d1d7de19459435fbde1999e3dfd226b: Beginning new tablet copy session on tablet 41e26e13355d4e0cb56bb526415a3998 from peer 1b411c6bb25549368041773d7b710d48 at {username='slave'} at 127.20.230.66:42445: session id = 1b411c6bb25549368041773d7b710d48-41e26e13355d4e0cb56bb526415a3998
I20240503 08:13:49.150665 30783 tablet_copy_source_session.cc:205] T 41e26e13355d4e0cb56bb526415a3998 P 6d1d7de19459435fbde1999e3dfd226b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:49.152758 6243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41e26e13355d4e0cb56bb526415a3998. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:49.159286 6243 tablet_copy_client.cc:783] T 41e26e13355d4e0cb56bb526415a3998 P 1b411c6bb25549368041773d7b710d48: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:49.159472 6243 tablet_copy_client.cc:647] T 41e26e13355d4e0cb56bb526415a3998 P 1b411c6bb25549368041773d7b710d48: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:49.162611 6243 tablet_copy_client.cc:523] T 41e26e13355d4e0cb56bb526415a3998 P 1b411c6bb25549368041773d7b710d48: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:49.166333 6243 tablet_bootstrap.cc:492] T 41e26e13355d4e0cb56bb526415a3998 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:13:49.175071 6243 tablet_bootstrap.cc:492] T 41e26e13355d4e0cb56bb526415a3998 P 1b411c6bb25549368041773d7b710d48: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:49.175967 6286 tablet_copy_client.cc:308] T 49b145f6946647248c967cf4badf8b40 P 833582ff68c94269be364ebd187a75db: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.68:46211
I20240503 08:13:49.177245 28413 tablet_copy_service.cc:140] P d09a11a03989421b9f28019dc1c14cc2: Received BeginTabletCopySession request for tablet 49b145f6946647248c967cf4badf8b40 from peer 833582ff68c94269be364ebd187a75db ({username='slave'} at 127.20.230.65:36529)
I20240503 08:13:49.177386 28413 tablet_copy_service.cc:161] P d09a11a03989421b9f28019dc1c14cc2: Beginning new tablet copy session on tablet 49b145f6946647248c967cf4badf8b40 from peer 833582ff68c94269be364ebd187a75db at {username='slave'} at 127.20.230.65:36529: session id = 833582ff68c94269be364ebd187a75db-49b145f6946647248c967cf4badf8b40
I20240503 08:13:49.178350 6243 tablet_bootstrap.cc:492] T 41e26e13355d4e0cb56bb526415a3998 P 1b411c6bb25549368041773d7b710d48: Bootstrap complete.
I20240503 08:13:49.178565 6243 ts_tablet_manager.cc:1367] T 41e26e13355d4e0cb56bb526415a3998 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20240503 08:13:49.179355 6243 raft_consensus.cc:348] T 41e26e13355d4e0cb56bb526415a3998 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:49.179545 6243 raft_consensus.cc:729] T 41e26e13355d4e0cb56bb526415a3998 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:13:49.179725 6243 consensus_queue.cc:260] T 41e26e13355d4e0cb56bb526415a3998 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:49.180132 28413 tablet_copy_source_session.cc:205] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:49.180370 6243 ts_tablet_manager.cc:1398] T 41e26e13355d4e0cb56bb526415a3998 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:49.181227 30783 tablet_copy_service.cc:342] P 6d1d7de19459435fbde1999e3dfd226b: Request end of tablet copy session 1b411c6bb25549368041773d7b710d48-41e26e13355d4e0cb56bb526415a3998 received from {username='slave'} at 127.20.230.66:42445
I20240503 08:13:49.181336 30783 tablet_copy_service.cc:433] P 6d1d7de19459435fbde1999e3dfd226b: ending tablet copy session 1b411c6bb25549368041773d7b710d48-41e26e13355d4e0cb56bb526415a3998 on tablet 41e26e13355d4e0cb56bb526415a3998 with peer 1b411c6bb25549368041773d7b710d48
I20240503 08:13:49.182159 6286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49b145f6946647248c967cf4badf8b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:49.188231 6286 tablet_copy_client.cc:783] T 49b145f6946647248c967cf4badf8b40 P 833582ff68c94269be364ebd187a75db: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:49.188457 6286 tablet_copy_client.cc:647] T 49b145f6946647248c967cf4badf8b40 P 833582ff68c94269be364ebd187a75db: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:49.193814 6286 tablet_copy_client.cc:523] T 49b145f6946647248c967cf4badf8b40 P 833582ff68c94269be364ebd187a75db: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:49.197358 6286 tablet_bootstrap.cc:492] T 49b145f6946647248c967cf4badf8b40 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:13:49.211758 6286 tablet_bootstrap.cc:492] T 49b145f6946647248c967cf4badf8b40 P 833582ff68c94269be364ebd187a75db: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:49.212101 6286 tablet_bootstrap.cc:492] T 49b145f6946647248c967cf4badf8b40 P 833582ff68c94269be364ebd187a75db: Bootstrap complete.
I20240503 08:13:49.212347 6286 ts_tablet_manager.cc:1367] T 49b145f6946647248c967cf4badf8b40 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.001s
I20240503 08:13:49.212939 6286 raft_consensus.cc:348] T 49b145f6946647248c967cf4badf8b40 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:49.213109 6286 raft_consensus.cc:729] T 49b145f6946647248c967cf4badf8b40 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:13:49.213300 6286 consensus_queue.cc:260] T 49b145f6946647248c967cf4badf8b40 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:49.213847 6286 ts_tablet_manager.cc:1398] T 49b145f6946647248c967cf4badf8b40 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 08:13:49.214442 28413 tablet_copy_service.cc:342] P d09a11a03989421b9f28019dc1c14cc2: Request end of tablet copy session 833582ff68c94269be364ebd187a75db-49b145f6946647248c967cf4badf8b40 received from {username='slave'} at 127.20.230.65:36529
I20240503 08:13:49.214560 28413 tablet_copy_service.cc:433] P d09a11a03989421b9f28019dc1c14cc2: ending tablet copy session 833582ff68c94269be364ebd187a75db-49b145f6946647248c967cf4badf8b40 on tablet 49b145f6946647248c967cf4badf8b40 with peer 833582ff68c94269be364ebd187a75db
I20240503 08:13:49.221354 28131 consensus_queue.cc:237] T 35c62760e1de4c2f88d06b97d0286c4c P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:49.224149 30762 raft_consensus.cc:1261] T 35c62760e1de4c2f88d06b97d0286c4c P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:13:49.224577 28069 consensus_peers.cc:485] T 35c62760e1de4c2f88d06b97d0286c4c P 1b411c6bb25549368041773d7b710d48 -> Peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Couldn't send request to peer 833582ff68c94269be364ebd187a75db. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 35c62760e1de4c2f88d06b97d0286c4c. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:49.224612 30895 raft_consensus.cc:1261] T 35c62760e1de4c2f88d06b97d0286c4c P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:49.225087 6282 consensus_queue.cc:1035] T 35c62760e1de4c2f88d06b97d0286c4c P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:49.225354 6005 consensus_queue.cc:1035] T 35c62760e1de4c2f88d06b97d0286c4c P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:49.226415 31028 raft_consensus.cc:1203] T 41aa0e288ca4469ba34a07ff159ace91 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:49.243867 6005 raft_consensus.cc:2932] T 35c62760e1de4c2f88d06b97d0286c4c P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:49.244484 30895 raft_consensus.cc:2932] T 35c62760e1de4c2f88d06b97d0286c4c P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:49.250413 30762 raft_consensus.cc:2932] T 35c62760e1de4c2f88d06b97d0286c4c P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:49.252303 27854 catalog_manager.cc:5484] T 35c62760e1de4c2f88d06b97d0286c4c P ce1496a47d084365ad786cd09e472a2c reported cstate change: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New cstate: current_term: 1 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:49.306079 28131 consensus_queue.cc:237] T fa7e18041c294a239eb7b8f735b66c53 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:49.308204 30762 raft_consensus.cc:1261] T fa7e18041c294a239eb7b8f735b66c53 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:49.308331 30894 raft_consensus.cc:1261] T fa7e18041c294a239eb7b8f735b66c53 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:13:49.308306 28067 consensus_peers.cc:485] T fa7e18041c294a239eb7b8f735b66c53 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fa7e18041c294a239eb7b8f735b66c53. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:49.308676 6282 consensus_queue.cc:1035] T fa7e18041c294a239eb7b8f735b66c53 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:49.308933 6283 consensus_queue.cc:1035] T fa7e18041c294a239eb7b8f735b66c53 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:49.317313 6239 ts_tablet_manager.cc:902] T 9fbe2eaba5344dfaa29e78a0d73f224a P aae860d7ae2b4002bf7794c278f9cc56: Initiating tablet copy from peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:13:49.325796 6283 raft_consensus.cc:2932] T fa7e18041c294a239eb7b8f735b66c53 P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:49.326206 30895 raft_consensus.cc:2932] T fa7e18041c294a239eb7b8f735b66c53 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:49.332269 6239 tablet_copy_client.cc:308] T 9fbe2eaba5344dfaa29e78a0d73f224a P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.65:39081
I20240503 08:13:49.332723 30762 raft_consensus.cc:2932] T fa7e18041c294a239eb7b8f735b66c53 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:49.333509 27862 catalog_manager.cc:5484] T fa7e18041c294a239eb7b8f735b66c53 P ce1496a47d084365ad786cd09e472a2c reported cstate change: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New cstate: current_term: 1 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:49.334443 28023 tablet_copy_service.cc:140] P 833582ff68c94269be364ebd187a75db: Received BeginTabletCopySession request for tablet 9fbe2eaba5344dfaa29e78a0d73f224a from peer aae860d7ae2b4002bf7794c278f9cc56 ({username='slave'} at 127.20.230.67:40275)
I20240503 08:13:49.334573 28023 tablet_copy_service.cc:161] P 833582ff68c94269be364ebd187a75db: Beginning new tablet copy session on tablet 9fbe2eaba5344dfaa29e78a0d73f224a from peer aae860d7ae2b4002bf7794c278f9cc56 at {username='slave'} at 127.20.230.67:40275: session id = aae860d7ae2b4002bf7794c278f9cc56-9fbe2eaba5344dfaa29e78a0d73f224a
I20240503 08:13:49.337141 28023 tablet_copy_source_session.cc:205] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:49.342484 6239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fbe2eaba5344dfaa29e78a0d73f224a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:49.347057 28131 raft_consensus.cc:1203] T 76d79d8be9e94e99a7ba51d903ca8a91 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:49.348151 6239 tablet_copy_client.cc:783] T 9fbe2eaba5344dfaa29e78a0d73f224a P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:49.348376 6239 tablet_copy_client.cc:647] T 9fbe2eaba5344dfaa29e78a0d73f224a P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:49.350252 6239 tablet_copy_client.cc:523] T 9fbe2eaba5344dfaa29e78a0d73f224a P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:49.354130 6239 tablet_bootstrap.cc:492] T 9fbe2eaba5344dfaa29e78a0d73f224a P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:13:49.358506 6239 tablet_bootstrap.cc:492] T 9fbe2eaba5344dfaa29e78a0d73f224a P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:49.358780 6239 tablet_bootstrap.cc:492] T 9fbe2eaba5344dfaa29e78a0d73f224a P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap complete.
I20240503 08:13:49.358999 6239 ts_tablet_manager.cc:1367] T 9fbe2eaba5344dfaa29e78a0d73f224a P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240503 08:13:49.359853 6239 raft_consensus.cc:348] T 9fbe2eaba5344dfaa29e78a0d73f224a P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:49.360072 6239 raft_consensus.cc:729] T 9fbe2eaba5344dfaa29e78a0d73f224a P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:13:49.360322 6239 consensus_queue.cc:260] T 9fbe2eaba5344dfaa29e78a0d73f224a P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:49.360879 6239 ts_tablet_manager.cc:1398] T 9fbe2eaba5344dfaa29e78a0d73f224a P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:49.361836 28023 tablet_copy_service.cc:342] P 833582ff68c94269be364ebd187a75db: Request end of tablet copy session aae860d7ae2b4002bf7794c278f9cc56-9fbe2eaba5344dfaa29e78a0d73f224a received from {username='slave'} at 127.20.230.67:40275
I20240503 08:13:49.361963 28023 tablet_copy_service.cc:433] P 833582ff68c94269be364ebd187a75db: ending tablet copy session aae860d7ae2b4002bf7794c278f9cc56-9fbe2eaba5344dfaa29e78a0d73f224a on tablet 9fbe2eaba5344dfaa29e78a0d73f224a with peer aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:13:49.367359 6347 ts_tablet_manager.cc:902] T 01c4229dfd25458296e85e412eae13bf P d09a11a03989421b9f28019dc1c14cc2: Initiating tablet copy from peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:13:49.380627 6347 tablet_copy_client.cc:308] T 01c4229dfd25458296e85e412eae13bf P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.69:34725
I20240503 08:13:49.381542 30783 tablet_copy_service.cc:140] P 6d1d7de19459435fbde1999e3dfd226b: Received BeginTabletCopySession request for tablet 01c4229dfd25458296e85e412eae13bf from peer d09a11a03989421b9f28019dc1c14cc2 ({username='slave'} at 127.20.230.68:50161)
I20240503 08:13:49.381711 30783 tablet_copy_service.cc:161] P 6d1d7de19459435fbde1999e3dfd226b: Beginning new tablet copy session on tablet 01c4229dfd25458296e85e412eae13bf from peer d09a11a03989421b9f28019dc1c14cc2 at {username='slave'} at 127.20.230.68:50161: session id = d09a11a03989421b9f28019dc1c14cc2-01c4229dfd25458296e85e412eae13bf
I20240503 08:13:49.384127 30783 tablet_copy_source_session.cc:205] T 01c4229dfd25458296e85e412eae13bf P 6d1d7de19459435fbde1999e3dfd226b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:49.386077 6347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01c4229dfd25458296e85e412eae13bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:49.392562 6347 tablet_copy_client.cc:783] T 01c4229dfd25458296e85e412eae13bf P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:49.393177 6347 tablet_copy_client.cc:647] T 01c4229dfd25458296e85e412eae13bf P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:49.397262 6347 tablet_copy_client.cc:523] T 01c4229dfd25458296e85e412eae13bf P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:49.400543 6347 tablet_bootstrap.cc:492] T 01c4229dfd25458296e85e412eae13bf P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:13:49.407989 6347 tablet_bootstrap.cc:492] T 01c4229dfd25458296e85e412eae13bf P d09a11a03989421b9f28019dc1c14cc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:49.408324 6347 tablet_bootstrap.cc:492] T 01c4229dfd25458296e85e412eae13bf P d09a11a03989421b9f28019dc1c14cc2: Bootstrap complete.
I20240503 08:13:49.408530 6347 ts_tablet_manager.cc:1367] T 01c4229dfd25458296e85e412eae13bf P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.005s
I20240503 08:13:49.409188 6347 raft_consensus.cc:348] T 01c4229dfd25458296e85e412eae13bf P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:49.409406 6347 raft_consensus.cc:729] T 01c4229dfd25458296e85e412eae13bf P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:13:49.409618 6347 consensus_queue.cc:260] T 01c4229dfd25458296e85e412eae13bf P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:49.410181 6347 ts_tablet_manager.cc:1398] T 01c4229dfd25458296e85e412eae13bf P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 08:13:49.410835 30783 tablet_copy_service.cc:342] P 6d1d7de19459435fbde1999e3dfd226b: Request end of tablet copy session d09a11a03989421b9f28019dc1c14cc2-01c4229dfd25458296e85e412eae13bf received from {username='slave'} at 127.20.230.68:50161
I20240503 08:13:49.411010 30783 tablet_copy_service.cc:433] P 6d1d7de19459435fbde1999e3dfd226b: ending tablet copy session d09a11a03989421b9f28019dc1c14cc2-01c4229dfd25458296e85e412eae13bf on tablet 01c4229dfd25458296e85e412eae13bf with peer d09a11a03989421b9f28019dc1c14cc2
I20240503 08:13:49.454667 30762 raft_consensus.cc:1203] T 98b3c906cfe54f49bb88ce5e00e3cc04 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:49.459170 28131 raft_consensus.cc:1203] T 1fca3d7f4b734eceb210e1186478eb72 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:49.514868 28262 raft_consensus.cc:1203] T 3bc8f4968b9743baab5d25c8c7237c9c P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:49.520910 6243 ts_tablet_manager.cc:902] T ab7b01c2e05a430a9873398268b1e41f P 1b411c6bb25549368041773d7b710d48: Initiating tablet copy from peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:13:49.541673 6243 tablet_copy_client.cc:308] T ab7b01c2e05a430a9873398268b1e41f P 1b411c6bb25549368041773d7b710d48: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.68:46211
I20240503 08:13:49.542279 28413 tablet_copy_service.cc:140] P d09a11a03989421b9f28019dc1c14cc2: Received BeginTabletCopySession request for tablet ab7b01c2e05a430a9873398268b1e41f from peer 1b411c6bb25549368041773d7b710d48 ({username='slave'} at 127.20.230.66:43721)
I20240503 08:13:49.542423 28413 tablet_copy_service.cc:161] P d09a11a03989421b9f28019dc1c14cc2: Beginning new tablet copy session on tablet ab7b01c2e05a430a9873398268b1e41f from peer 1b411c6bb25549368041773d7b710d48 at {username='slave'} at 127.20.230.66:43721: session id = 1b411c6bb25549368041773d7b710d48-ab7b01c2e05a430a9873398268b1e41f
I20240503 08:13:49.545050 28413 tablet_copy_source_session.cc:205] T ab7b01c2e05a430a9873398268b1e41f P d09a11a03989421b9f28019dc1c14cc2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:49.550092 6243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab7b01c2e05a430a9873398268b1e41f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:49.580443 28000 raft_consensus.cc:1203] T 49b145f6946647248c967cf4badf8b40 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:49.597972 6243 tablet_copy_client.cc:783] T ab7b01c2e05a430a9873398268b1e41f P 1b411c6bb25549368041773d7b710d48: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:49.598254 6243 tablet_copy_client.cc:647] T ab7b01c2e05a430a9873398268b1e41f P 1b411c6bb25549368041773d7b710d48: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:49.603768 6243 tablet_copy_client.cc:523] T ab7b01c2e05a430a9873398268b1e41f P 1b411c6bb25549368041773d7b710d48: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:49.607101 6243 tablet_bootstrap.cc:492] T ab7b01c2e05a430a9873398268b1e41f P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:13:49.621419 6243 tablet_bootstrap.cc:492] T ab7b01c2e05a430a9873398268b1e41f P 1b411c6bb25549368041773d7b710d48: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:49.621887 6243 tablet_bootstrap.cc:492] T ab7b01c2e05a430a9873398268b1e41f P 1b411c6bb25549368041773d7b710d48: Bootstrap complete.
I20240503 08:13:49.622184 6243 ts_tablet_manager.cc:1367] T ab7b01c2e05a430a9873398268b1e41f P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.006s
I20240503 08:13:49.622874 6243 raft_consensus.cc:348] T ab7b01c2e05a430a9873398268b1e41f P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:49.623222 6243 raft_consensus.cc:729] T ab7b01c2e05a430a9873398268b1e41f P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:13:49.623521 6243 consensus_queue.cc:260] T ab7b01c2e05a430a9873398268b1e41f P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:49.625670 6243 ts_tablet_manager.cc:1398] T ab7b01c2e05a430a9873398268b1e41f P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 08:13:49.626303 28413 tablet_copy_service.cc:342] P d09a11a03989421b9f28019dc1c14cc2: Request end of tablet copy session 1b411c6bb25549368041773d7b710d48-ab7b01c2e05a430a9873398268b1e41f received from {username='slave'} at 127.20.230.66:43721
I20240503 08:13:49.626399 28413 tablet_copy_service.cc:433] P d09a11a03989421b9f28019dc1c14cc2: ending tablet copy session 1b411c6bb25549368041773d7b710d48-ab7b01c2e05a430a9873398268b1e41f on tablet ab7b01c2e05a430a9873398268b1e41f with peer 1b411c6bb25549368041773d7b710d48
I20240503 08:13:49.661201 6286 ts_tablet_manager.cc:902] T 35c62760e1de4c2f88d06b97d0286c4c P 833582ff68c94269be364ebd187a75db: Initiating tablet copy from peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:13:49.707267 6286 tablet_copy_client.cc:308] T 35c62760e1de4c2f88d06b97d0286c4c P 833582ff68c94269be364ebd187a75db: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.66:37143
I20240503 08:13:49.707844 28153 tablet_copy_service.cc:140] P 1b411c6bb25549368041773d7b710d48: Received BeginTabletCopySession request for tablet 35c62760e1de4c2f88d06b97d0286c4c from peer 833582ff68c94269be364ebd187a75db ({username='slave'} at 127.20.230.65:36417)
I20240503 08:13:49.708005 28153 tablet_copy_service.cc:161] P 1b411c6bb25549368041773d7b710d48: Beginning new tablet copy session on tablet 35c62760e1de4c2f88d06b97d0286c4c from peer 833582ff68c94269be364ebd187a75db at {username='slave'} at 127.20.230.65:36417: session id = 833582ff68c94269be364ebd187a75db-35c62760e1de4c2f88d06b97d0286c4c
I20240503 08:13:49.710911 28153 tablet_copy_source_session.cc:205] T 35c62760e1de4c2f88d06b97d0286c4c P 1b411c6bb25549368041773d7b710d48: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:49.716879 6286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35c62760e1de4c2f88d06b97d0286c4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:49.727360 6286 tablet_copy_client.cc:783] T 35c62760e1de4c2f88d06b97d0286c4c P 833582ff68c94269be364ebd187a75db: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:49.728591 6286 tablet_copy_client.cc:647] T 35c62760e1de4c2f88d06b97d0286c4c P 833582ff68c94269be364ebd187a75db: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:49.730283 6286 tablet_copy_client.cc:523] T 35c62760e1de4c2f88d06b97d0286c4c P 833582ff68c94269be364ebd187a75db: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:49.730981 28131 raft_consensus.cc:1203] T 41e26e13355d4e0cb56bb526415a3998 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:49.735173 6286 tablet_bootstrap.cc:492] T 35c62760e1de4c2f88d06b97d0286c4c P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:13:49.750063 6286 tablet_bootstrap.cc:492] T 35c62760e1de4c2f88d06b97d0286c4c P 833582ff68c94269be364ebd187a75db: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:49.750521 6286 tablet_bootstrap.cc:492] T 35c62760e1de4c2f88d06b97d0286c4c P 833582ff68c94269be364ebd187a75db: Bootstrap complete.
I20240503 08:13:49.750871 6286 ts_tablet_manager.cc:1367] T 35c62760e1de4c2f88d06b97d0286c4c P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.003s
I20240503 08:13:49.751703 6286 raft_consensus.cc:348] T 35c62760e1de4c2f88d06b97d0286c4c P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:49.752192 6286 raft_consensus.cc:729] T 35c62760e1de4c2f88d06b97d0286c4c P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:13:49.752481 6286 consensus_queue.cc:260] T 35c62760e1de4c2f88d06b97d0286c4c P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:49.755123 6286 ts_tablet_manager.cc:1398] T 35c62760e1de4c2f88d06b97d0286c4c P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:13:49.755757 28153 tablet_copy_service.cc:342] P 1b411c6bb25549368041773d7b710d48: Request end of tablet copy session 833582ff68c94269be364ebd187a75db-35c62760e1de4c2f88d06b97d0286c4c received from {username='slave'} at 127.20.230.65:36417
I20240503 08:13:49.755861 28153 tablet_copy_service.cc:433] P 1b411c6bb25549368041773d7b710d48: ending tablet copy session 833582ff68c94269be364ebd187a75db-35c62760e1de4c2f88d06b97d0286c4c on tablet 35c62760e1de4c2f88d06b97d0286c4c with peer 833582ff68c94269be364ebd187a75db
I20240503 08:13:49.834714 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:49.858575 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:49.862725 28389 raft_consensus.cc:1203] T 01c4229dfd25458296e85e412eae13bf P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:49.878921 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:49.895390 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:13:49.909049 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:49.942618 28262 raft_consensus.cc:1203] T 9fbe2eaba5344dfaa29e78a0d73f224a P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:49.943807 6423 ts_tablet_manager.cc:902] T fa7e18041c294a239eb7b8f735b66c53 P aae860d7ae2b4002bf7794c278f9cc56: Initiating tablet copy from peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:13:49.954768 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:49.982565 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:49.988502 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:50.011667 6423 tablet_copy_client.cc:308] T fa7e18041c294a239eb7b8f735b66c53 P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.66:37143
I20240503 08:13:50.023126 28153 tablet_copy_service.cc:140] P 1b411c6bb25549368041773d7b710d48: Received BeginTabletCopySession request for tablet fa7e18041c294a239eb7b8f735b66c53 from peer aae860d7ae2b4002bf7794c278f9cc56 ({username='slave'} at 127.20.230.67:59481)
I20240503 08:13:50.023415 28153 tablet_copy_service.cc:161] P 1b411c6bb25549368041773d7b710d48: Beginning new tablet copy session on tablet fa7e18041c294a239eb7b8f735b66c53 from peer aae860d7ae2b4002bf7794c278f9cc56 at {username='slave'} at 127.20.230.67:59481: session id = aae860d7ae2b4002bf7794c278f9cc56-fa7e18041c294a239eb7b8f735b66c53
I20240503 08:13:50.027252 28153 tablet_copy_source_session.cc:205] T fa7e18041c294a239eb7b8f735b66c53 P 1b411c6bb25549368041773d7b710d48: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:50.028573 6423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa7e18041c294a239eb7b8f735b66c53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:50.035750 6423 tablet_copy_client.cc:783] T fa7e18041c294a239eb7b8f735b66c53 P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:50.036351 6423 tablet_copy_client.cc:647] T fa7e18041c294a239eb7b8f735b66c53 P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:50.038116 6423 tablet_copy_client.cc:523] T fa7e18041c294a239eb7b8f735b66c53 P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:50.041711 6423 tablet_bootstrap.cc:492] T fa7e18041c294a239eb7b8f735b66c53 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:13:50.085703 6423 tablet_bootstrap.cc:492] T fa7e18041c294a239eb7b8f735b66c53 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:50.086091 6423 tablet_bootstrap.cc:492] T fa7e18041c294a239eb7b8f735b66c53 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap complete.
I20240503 08:13:50.086325 6423 ts_tablet_manager.cc:1367] T fa7e18041c294a239eb7b8f735b66c53 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.045s user 0.013s sys 0.027s
I20240503 08:13:50.086978 6423 raft_consensus.cc:348] T fa7e18041c294a239eb7b8f735b66c53 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:50.087185 6423 raft_consensus.cc:729] T fa7e18041c294a239eb7b8f735b66c53 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:13:50.087383 6423 consensus_queue.cc:260] T fa7e18041c294a239eb7b8f735b66c53 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:50.088501 28133 raft_consensus.cc:1203] T ab7b01c2e05a430a9873398268b1e41f P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:50.089799 6423 ts_tablet_manager.cc:1398] T fa7e18041c294a239eb7b8f735b66c53 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:13:50.092594 28153 tablet_copy_service.cc:342] P 1b411c6bb25549368041773d7b710d48: Request end of tablet copy session aae860d7ae2b4002bf7794c278f9cc56-fa7e18041c294a239eb7b8f735b66c53 received from {username='slave'} at 127.20.230.67:59481
I20240503 08:13:50.092718 28153 tablet_copy_service.cc:433] P 1b411c6bb25549368041773d7b710d48: ending tablet copy session aae860d7ae2b4002bf7794c278f9cc56-fa7e18041c294a239eb7b8f735b66c53 on tablet fa7e18041c294a239eb7b8f735b66c53 with peer aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:13:50.170027 27999 raft_consensus.cc:1203] T 35c62760e1de4c2f88d06b97d0286c4c P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:50.354115 28263 raft_consensus.cc:1203] T fa7e18041c294a239eb7b8f735b66c53 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:50.410887 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:50.438374 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:50.452479 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:50.457837 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:50.475983 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:50.478610 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:50.512223 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:50.554786 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:13:50.735857 28389 consensus_queue.cc:237] T ab7b01c2e05a430a9873398268b1e41f P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:50.738019 28133 raft_consensus.cc:1261] T ab7b01c2e05a430a9873398268b1e41f P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:50.738570 30894 raft_consensus.cc:1261] T ab7b01c2e05a430a9873398268b1e41f P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:50.738508 6495 consensus_queue.cc:1035] T ab7b01c2e05a430a9873398268b1e41f P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:50.739064 6495 consensus_queue.cc:1035] T ab7b01c2e05a430a9873398268b1e41f P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:50.741539 6495 raft_consensus.cc:2932] T ab7b01c2e05a430a9873398268b1e41f P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:50.742463 30894 raft_consensus.cc:2932] T ab7b01c2e05a430a9873398268b1e41f P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:50.743511 28133 raft_consensus.cc:2932] T ab7b01c2e05a430a9873398268b1e41f P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:50.749814 27860 catalog_manager.cc:5484] T ab7b01c2e05a430a9873398268b1e41f P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:50.758658 31006 tablet_service.cc:1514] Processing DeleteTablet for tablet ab7b01c2e05a430a9873398268b1e41f with delete_type TABLET_DATA_TOMBSTONED (TS ced21ca983b749d0b4d4f6a0e2426bf8 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47020
I20240503 08:13:50.759232 6519 tablet_replica.cc:330] T ab7b01c2e05a430a9873398268b1e41f P ced21ca983b749d0b4d4f6a0e2426bf8: stopping tablet replica
I20240503 08:13:50.759490 6519 raft_consensus.cc:2229] T ab7b01c2e05a430a9873398268b1e41f P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:50.759732 6519 raft_consensus.cc:2258] T ab7b01c2e05a430a9873398268b1e41f P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:50.760351 6519 ts_tablet_manager.cc:1875] T ab7b01c2e05a430a9873398268b1e41f P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:50.766583 6519 ts_tablet_manager.cc:1888] T ab7b01c2e05a430a9873398268b1e41f P ced21ca983b749d0b4d4f6a0e2426bf8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:13:50.766738 6519 log.cc:1196] T ab7b01c2e05a430a9873398268b1e41f P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal/wals/ab7b01c2e05a430a9873398268b1e41f
I20240503 08:13:50.767544 27847 catalog_manager.cc:4830] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): tablet ab7b01c2e05a430a9873398268b1e41f (table rebalancer_extra_table_42 [id=3efe105f64254e5eb0be59c6b266462e]) successfully deleted
I20240503 08:13:51.021939 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:51.031268 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:51.043771 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:51.069253 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:51.086285 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:51.143913 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:13:51.163316 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:51.166571 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:51.291229 30761 consensus_queue.cc:237] T 01c4229dfd25458296e85e412eae13bf P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:51.292938 28390 raft_consensus.cc:1261] T 01c4229dfd25458296e85e412eae13bf P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:51.292980 27999 raft_consensus.cc:1261] T 01c4229dfd25458296e85e412eae13bf P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:51.293395 5388 consensus_queue.cc:1035] T 01c4229dfd25458296e85e412eae13bf P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:51.293687 6194 consensus_queue.cc:1035] T 01c4229dfd25458296e85e412eae13bf P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:51.296790 6194 raft_consensus.cc:2932] T 01c4229dfd25458296e85e412eae13bf P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:51.297166 27999 raft_consensus.cc:2932] T 01c4229dfd25458296e85e412eae13bf P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:51.302285 27857 catalog_manager.cc:5484] T 01c4229dfd25458296e85e412eae13bf P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:51.302938 28390 raft_consensus.cc:2932] T 01c4229dfd25458296e85e412eae13bf P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:51.311614 31141 tablet_service.cc:1514] Processing DeleteTablet for tablet 01c4229dfd25458296e85e412eae13bf with delete_type TABLET_DATA_TOMBSTONED (TS b00f244c69df401997ea99d492218a98 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45814
I20240503 08:13:51.312429 6606 tablet_replica.cc:330] T 01c4229dfd25458296e85e412eae13bf P b00f244c69df401997ea99d492218a98: stopping tablet replica
I20240503 08:13:51.312682 6606 raft_consensus.cc:2229] T 01c4229dfd25458296e85e412eae13bf P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:51.312908 6606 raft_consensus.cc:2258] T 01c4229dfd25458296e85e412eae13bf P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:51.315369 6606 ts_tablet_manager.cc:1875] T 01c4229dfd25458296e85e412eae13bf P b00f244c69df401997ea99d492218a98: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:51.320571 6606 ts_tablet_manager.cc:1888] T 01c4229dfd25458296e85e412eae13bf P b00f244c69df401997ea99d492218a98: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:13:51.320683 6606 log.cc:1196] T 01c4229dfd25458296e85e412eae13bf P b00f244c69df401997ea99d492218a98: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal/wals/01c4229dfd25458296e85e412eae13bf
I20240503 08:13:51.321331 27846 catalog_manager.cc:4830] TS b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): tablet 01c4229dfd25458296e85e412eae13bf (table rebalancer_extra_table_10 [id=9ee47b3ce8574777976ff3cb1ea03e9c]) successfully deleted
I20240503 08:13:51.582324 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:51.582368 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:51.604530 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:51.647897 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:51.681442 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:51.694655 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:51.727787 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:13:51.807664 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:51.918004 30761 consensus_queue.cc:237] T 41e26e13355d4e0cb56bb526415a3998 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:51.920007 28131 raft_consensus.cc:1261] T 41e26e13355d4e0cb56bb526415a3998 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:51.920154 31028 raft_consensus.cc:1261] T 41e26e13355d4e0cb56bb526415a3998 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:51.920430 6194 consensus_queue.cc:1035] T 41e26e13355d4e0cb56bb526415a3998 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:51.920670 5388 consensus_queue.cc:1035] T 41e26e13355d4e0cb56bb526415a3998 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:51.923082 6597 raft_consensus.cc:2932] T 41e26e13355d4e0cb56bb526415a3998 P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:51.923717 31028 raft_consensus.cc:2932] T 41e26e13355d4e0cb56bb526415a3998 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:51.926723 28131 raft_consensus.cc:2932] T 41e26e13355d4e0cb56bb526415a3998 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:51.930860 27858 catalog_manager.cc:5484] T 41e26e13355d4e0cb56bb526415a3998 P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: config changed from index 2 to 3, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:51.939427 30874 tablet_service.cc:1514] Processing DeleteTablet for tablet 41e26e13355d4e0cb56bb526415a3998 with delete_type TABLET_DATA_TOMBSTONED (TS ce1496a47d084365ad786cd09e472a2c not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:34426
I20240503 08:13:51.939950 6688 tablet_replica.cc:330] T 41e26e13355d4e0cb56bb526415a3998 P ce1496a47d084365ad786cd09e472a2c: stopping tablet replica
I20240503 08:13:51.940472 6688 raft_consensus.cc:2229] T 41e26e13355d4e0cb56bb526415a3998 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:51.940666 6688 raft_consensus.cc:2258] T 41e26e13355d4e0cb56bb526415a3998 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:51.941310 6688 ts_tablet_manager.cc:1875] T 41e26e13355d4e0cb56bb526415a3998 P ce1496a47d084365ad786cd09e472a2c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:51.947407 6688 ts_tablet_manager.cc:1888] T 41e26e13355d4e0cb56bb526415a3998 P ce1496a47d084365ad786cd09e472a2c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:13:51.947512 6688 log.cc:1196] T 41e26e13355d4e0cb56bb526415a3998 P ce1496a47d084365ad786cd09e472a2c: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal/wals/41e26e13355d4e0cb56bb526415a3998
I20240503 08:13:51.948295 27846 catalog_manager.cc:4830] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): tablet 41e26e13355d4e0cb56bb526415a3998 (table rebalancer_extra_table_22 [id=11f85d35fe5f401d990c665eefacf247]) successfully deleted
I20240503 08:13:52.286881 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:52.360548 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:52.364849 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:52.367400 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:52.378998 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:52.414566 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:13:52.428308 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:52.472769 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:52.590436 28391 consensus_queue.cc:237] T 76d79d8be9e94e99a7ba51d903ca8a91 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:52.592295 30895 raft_consensus.cc:1261] T 76d79d8be9e94e99a7ba51d903ca8a91 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:52.592418 28131 raft_consensus.cc:1261] T 76d79d8be9e94e99a7ba51d903ca8a91 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:52.592809 6700 consensus_queue.cc:1035] T 76d79d8be9e94e99a7ba51d903ca8a91 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:52.593034 6101 consensus_queue.cc:1035] T 76d79d8be9e94e99a7ba51d903ca8a91 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:52.595580 6101 raft_consensus.cc:2932] T 76d79d8be9e94e99a7ba51d903ca8a91 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:52.596257 30895 raft_consensus.cc:2932] T 76d79d8be9e94e99a7ba51d903ca8a91 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:52.596899 28131 raft_consensus.cc:2932] T 76d79d8be9e94e99a7ba51d903ca8a91 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:52.602346 27860 catalog_manager.cc:5484] T 76d79d8be9e94e99a7ba51d903ca8a91 P 1b411c6bb25549368041773d7b710d48 reported cstate change: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:52.608369 31006 tablet_service.cc:1514] Processing DeleteTablet for tablet 76d79d8be9e94e99a7ba51d903ca8a91 with delete_type TABLET_DATA_TOMBSTONED (TS ced21ca983b749d0b4d4f6a0e2426bf8 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47020
I20240503 08:13:52.608965 6788 tablet_replica.cc:330] T 76d79d8be9e94e99a7ba51d903ca8a91 P ced21ca983b749d0b4d4f6a0e2426bf8: stopping tablet replica
I20240503 08:13:52.609274 6788 raft_consensus.cc:2229] T 76d79d8be9e94e99a7ba51d903ca8a91 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:52.609470 6788 raft_consensus.cc:2258] T 76d79d8be9e94e99a7ba51d903ca8a91 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:52.610085 6788 ts_tablet_manager.cc:1875] T 76d79d8be9e94e99a7ba51d903ca8a91 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:52.616585 6788 ts_tablet_manager.cc:1888] T 76d79d8be9e94e99a7ba51d903ca8a91 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:13:52.616685 6788 log.cc:1196] T 76d79d8be9e94e99a7ba51d903ca8a91 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal/wals/76d79d8be9e94e99a7ba51d903ca8a91
I20240503 08:13:52.617493 27847 catalog_manager.cc:4830] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): tablet 76d79d8be9e94e99a7ba51d903ca8a91 (table rebalancer_extra_table_40 [id=3aef15bedeb544e9a15460f0b11c368a]) successfully deleted
I20240503 08:13:52.885713 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:52.902791 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:52.905975 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:52.948972 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:52.951699 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:52.961905 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:53.022264 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:13:53.024163 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:13:53.163736 28389 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "49b145f6946647248c967cf4badf8b40"
dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41830
I20240503 08:13:53.163980 28389 raft_consensus.cc:595] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:13:53.181847 6700 raft_consensus.cc:979] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2: : Instructing follower ce1496a47d084365ad786cd09e472a2c to start an election
I20240503 08:13:53.182001 6700 raft_consensus.cc:1067] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Signalling peer ce1496a47d084365ad786cd09e472a2c to start an election
I20240503 08:13:53.182747 30895 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "49b145f6946647248c967cf4badf8b40"
dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
from {username='slave'} at 127.20.230.68:45821
I20240503 08:13:53.182919 30895 raft_consensus.cc:482] T 49b145f6946647248c967cf4badf8b40 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:53.183003 30895 raft_consensus.cc:3037] T 49b145f6946647248c967cf4badf8b40 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:53.185501 30895 raft_consensus.cc:504] T 49b145f6946647248c967cf4badf8b40 P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:53.186543 28389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b145f6946647248c967cf4badf8b40" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:13:53.186717 28389 raft_consensus.cc:3032] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:53.186797 28389 raft_consensus.cc:729] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Running, Role: LEADER
I20240503 08:13:53.187063 28389 consensus_queue.cc:260] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:53.187469 28389 raft_consensus.cc:3037] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:53.191406 31028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b145f6946647248c967cf4badf8b40" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:13:53.191603 31028 raft_consensus.cc:3037] T 49b145f6946647248c967cf4badf8b40 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:53.191584 28389 raft_consensus.cc:2454] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 2.
I20240503 08:13:53.192031 30895 leader_election.cc:288] T 49b145f6946647248c967cf4badf8b40 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 election: Requested vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:13:53.193922 27999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b145f6946647248c967cf4badf8b40" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:13:53.194074 27999 raft_consensus.cc:3037] T 49b145f6946647248c967cf4badf8b40 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:53.194165 31028 raft_consensus.cc:2454] T 49b145f6946647248c967cf4badf8b40 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 2.
I20240503 08:13:53.195760 27999 raft_consensus.cc:2454] T 49b145f6946647248c967cf4badf8b40 P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 2.
I20240503 08:13:53.196195 30828 leader_election.cc:302] T 49b145f6946647248c967cf4badf8b40 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c, ced21ca983b749d0b4d4f6a0e2426bf8, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:13:53.196548 6764 raft_consensus.cc:2781] T 49b145f6946647248c967cf4badf8b40 P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:53.196673 6764 raft_consensus.cc:686] T 49b145f6946647248c967cf4badf8b40 P ce1496a47d084365ad786cd09e472a2c [term 2 LEADER]: Becoming Leader. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Running, Role: LEADER
I20240503 08:13:53.196921 6764 consensus_queue.cc:237] T 49b145f6946647248c967cf4badf8b40 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:53.202237 27854 catalog_manager.cc:5484] T 49b145f6946647248c967cf4badf8b40 P ce1496a47d084365ad786cd09e472a2c reported cstate change: term changed from 1 to 2, leader changed from d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) to ce1496a47d084365ad786cd09e472a2c (127.20.230.70). New cstate: current_term: 2 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:53.390808 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:53.395017 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:53.398077 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:53.400421 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:53.401372 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:53.442873 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:13:53.446272 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:53.507763 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:53.584617 31028 raft_consensus.cc:1261] T 49b145f6946647248c967cf4badf8b40 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:13:53.586072 6138 consensus_queue.cc:1035] T 49b145f6946647248c967cf4badf8b40 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:53.781265 28000 raft_consensus.cc:1261] T 49b145f6946647248c967cf4badf8b40 P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:13:53.781893 6138 consensus_queue.cc:1035] T 49b145f6946647248c967cf4badf8b40 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:53.818464 28389 raft_consensus.cc:1261] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:13:53.819078 6138 consensus_queue.cc:1035] T 49b145f6946647248c967cf4badf8b40 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:53.895942 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:53.920859 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:53.934165 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:53.939570 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:53.973052 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:53.994801 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:54.013435 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:54.098335 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:13:54.180658 30894 consensus_queue.cc:237] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:54.182413 30761 raft_consensus.cc:1261] T 3bc8f4968b9743baab5d25c8c7237c9c P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:54.182440 28262 raft_consensus.cc:1261] T 3bc8f4968b9743baab5d25c8c7237c9c P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:54.182839 6956 consensus_queue.cc:1035] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:54.183178 6764 consensus_queue.cc:1035] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:54.185389 6956 raft_consensus.cc:2932] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:54.185930 30761 raft_consensus.cc:2932] T 3bc8f4968b9743baab5d25c8c7237c9c P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:54.187040 28262 raft_consensus.cc:2932] T 3bc8f4968b9743baab5d25c8c7237c9c P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:54.191764 27855 catalog_manager.cc:5484] T 3bc8f4968b9743baab5d25c8c7237c9c P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New cstate: current_term: 1 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:54.207854 31141 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bc8f4968b9743baab5d25c8c7237c9c with delete_type TABLET_DATA_TOMBSTONED (TS b00f244c69df401997ea99d492218a98 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45814
I20240503 08:13:54.224864 6998 tablet_replica.cc:330] T 3bc8f4968b9743baab5d25c8c7237c9c P b00f244c69df401997ea99d492218a98: stopping tablet replica
I20240503 08:13:54.225157 6998 raft_consensus.cc:2229] T 3bc8f4968b9743baab5d25c8c7237c9c P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:54.225415 6998 raft_consensus.cc:2258] T 3bc8f4968b9743baab5d25c8c7237c9c P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:54.226059 6998 ts_tablet_manager.cc:1875] T 3bc8f4968b9743baab5d25c8c7237c9c P b00f244c69df401997ea99d492218a98: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:54.232723 6998 ts_tablet_manager.cc:1888] T 3bc8f4968b9743baab5d25c8c7237c9c P b00f244c69df401997ea99d492218a98: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:13:54.232883 6998 log.cc:1196] T 3bc8f4968b9743baab5d25c8c7237c9c P b00f244c69df401997ea99d492218a98: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal/wals/3bc8f4968b9743baab5d25c8c7237c9c
I20240503 08:13:54.233839 27846 catalog_manager.cc:4830] TS b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): tablet 3bc8f4968b9743baab5d25c8c7237c9c (table rebalancer_extra_table_38 [id=1eba5f663686410fb06575b0f37c6126]) successfully deleted
I20240503 08:13:54.439002 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:54.468344 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:54.471562 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:54.487062 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:54.496001 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:54.498791 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:54.528911 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:54.543571 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:13:54.748822 31160 consensus_queue.cc:237] T 98b3c906cfe54f49bb88ce5e00e3cc04 P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:13:54.750859 30761 raft_consensus.cc:1261] T 98b3c906cfe54f49bb88ce5e00e3cc04 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:54.751063 31027 raft_consensus.cc:1261] T 98b3c906cfe54f49bb88ce5e00e3cc04 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:54.751355 6665 consensus_queue.cc:1035] T 98b3c906cfe54f49bb88ce5e00e3cc04 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:54.751566 6382 consensus_queue.cc:1035] T 98b3c906cfe54f49bb88ce5e00e3cc04 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:54.755100 6665 raft_consensus.cc:2932] T 98b3c906cfe54f49bb88ce5e00e3cc04 P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:54.758816 30761 raft_consensus.cc:2932] T 98b3c906cfe54f49bb88ce5e00e3cc04 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:54.766383 27862 catalog_manager.cc:5484] T 98b3c906cfe54f49bb88ce5e00e3cc04 P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: config changed from index 2 to 3, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New cstate: current_term: 1 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:54.772389 31028 raft_consensus.cc:2932] T 98b3c906cfe54f49bb88ce5e00e3cc04 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:13:54.786890 30874 tablet_service.cc:1514] Processing DeleteTablet for tablet 98b3c906cfe54f49bb88ce5e00e3cc04 with delete_type TABLET_DATA_TOMBSTONED (TS ce1496a47d084365ad786cd09e472a2c not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:34426
I20240503 08:13:54.788515 7076 tablet_replica.cc:330] T 98b3c906cfe54f49bb88ce5e00e3cc04 P ce1496a47d084365ad786cd09e472a2c: stopping tablet replica
I20240503 08:13:54.788766 7076 raft_consensus.cc:2229] T 98b3c906cfe54f49bb88ce5e00e3cc04 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:54.789038 7076 raft_consensus.cc:2258] T 98b3c906cfe54f49bb88ce5e00e3cc04 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:54.789639 7076 ts_tablet_manager.cc:1875] T 98b3c906cfe54f49bb88ce5e00e3cc04 P ce1496a47d084365ad786cd09e472a2c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:54.795032 7076 ts_tablet_manager.cc:1888] T 98b3c906cfe54f49bb88ce5e00e3cc04 P ce1496a47d084365ad786cd09e472a2c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:13:54.795185 7076 log.cc:1196] T 98b3c906cfe54f49bb88ce5e00e3cc04 P ce1496a47d084365ad786cd09e472a2c: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal/wals/98b3c906cfe54f49bb88ce5e00e3cc04
I20240503 08:13:54.796800 27846 catalog_manager.cc:4830] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): tablet 98b3c906cfe54f49bb88ce5e00e3cc04 (table rebalancer_extra_table_42 [id=3efe105f64254e5eb0be59c6b266462e]) successfully deleted
I20240503 08:13:55.041419 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:55.042349 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:55.071671 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:55.073608 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:55.123431 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:13:55.144660 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:55.158680 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:55.237071 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:55.403014 28262 consensus_queue.cc:237] T 41aa0e288ca4469ba34a07ff159ace91 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:13:55.404983 31028 raft_consensus.cc:1261] T 41aa0e288ca4469ba34a07ff159ace91 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:55.405074 28391 raft_consensus.cc:1261] T 41aa0e288ca4469ba34a07ff159ace91 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:55.405577 6004 consensus_queue.cc:1035] T 41aa0e288ca4469ba34a07ff159ace91 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:55.405968 6763 consensus_queue.cc:1035] T 41aa0e288ca4469ba34a07ff159ace91 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:55.408213 6004 raft_consensus.cc:2932] T 41aa0e288ca4469ba34a07ff159ace91 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:13:55.409238 31028 raft_consensus.cc:2932] T 41aa0e288ca4469ba34a07ff159ace91 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:13:55.413097 28391 raft_consensus.cc:2932] T 41aa0e288ca4469ba34a07ff159ace91 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:13:55.415675 27862 catalog_manager.cc:5484] T 41aa0e288ca4469ba34a07ff159ace91 P ced21ca983b749d0b4d4f6a0e2426bf8 reported cstate change: config changed from index 4 to 5, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:13:55.424628 28111 tablet_service.cc:1514] Processing DeleteTablet for tablet 41aa0e288ca4469ba34a07ff159ace91 with delete_type TABLET_DATA_TOMBSTONED (TS 1b411c6bb25549368041773d7b710d48 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57912
I20240503 08:13:55.430147 7157 tablet_replica.cc:330] T 41aa0e288ca4469ba34a07ff159ace91 P 1b411c6bb25549368041773d7b710d48: stopping tablet replica
I20240503 08:13:55.430492 7157 raft_consensus.cc:2229] T 41aa0e288ca4469ba34a07ff159ace91 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:55.430835 7157 raft_consensus.cc:2258] T 41aa0e288ca4469ba34a07ff159ace91 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:55.431514 7157 ts_tablet_manager.cc:1875] T 41aa0e288ca4469ba34a07ff159ace91 P 1b411c6bb25549368041773d7b710d48: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:55.437104 7157 ts_tablet_manager.cc:1888] T 41aa0e288ca4469ba34a07ff159ace91 P 1b411c6bb25549368041773d7b710d48: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:13:55.437196 7157 log.cc:1196] T 41aa0e288ca4469ba34a07ff159ace91 P 1b411c6bb25549368041773d7b710d48: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/41aa0e288ca4469ba34a07ff159ace91
I20240503 08:13:55.437942 27848 catalog_manager.cc:4830] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): tablet 41aa0e288ca4469ba34a07ff159ace91 (table rebalance_test_table_17 [id=7e5325b51bd7410e93410447bfd72f9c]) successfully deleted
I20240503 08:13:55.746939 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:55.752077 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:55.793403 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:55.842259 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:55.863055 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 08:13:55.873502 28066 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.70:33747, user_credentials={real_user=slave}} blocked reactor thread for 112684us
I20240503 08:13:55.895799 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:55.898574 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:13:55.914288 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240503 08:13:55.927858 28327 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.67:39919, user_credentials={real_user=slave}} blocked reactor thread for 45008.3us
W20240503 08:13:55.984125 30962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.68:46211, user_credentials={real_user=slave}} blocked reactor thread for 52747.9us
I20240503 08:13:56.058447 28000 consensus_queue.cc:237] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:56.060511 28262 raft_consensus.cc:1261] T 9fbe2eaba5344dfaa29e78a0d73f224a P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:56.060586 31027 raft_consensus.cc:1261] T 9fbe2eaba5344dfaa29e78a0d73f224a P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:56.061043 7212 consensus_queue.cc:1035] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:56.061267 6578 consensus_queue.cc:1035] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:56.063752 6765 raft_consensus.cc:2932] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:56.064360 31027 raft_consensus.cc:2932] T 9fbe2eaba5344dfaa29e78a0d73f224a P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:56.064332 28262 raft_consensus.cc:2932] T 9fbe2eaba5344dfaa29e78a0d73f224a P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:56.070672 27857 catalog_manager.cc:5484] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db reported cstate change: config changed from index 2 to 3, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:56.081629 30742 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fbe2eaba5344dfaa29e78a0d73f224a with delete_type TABLET_DATA_TOMBSTONED (TS 6d1d7de19459435fbde1999e3dfd226b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46794
I20240503 08:13:56.082211 7252 tablet_replica.cc:330] T 9fbe2eaba5344dfaa29e78a0d73f224a P 6d1d7de19459435fbde1999e3dfd226b: stopping tablet replica
I20240503 08:13:56.082469 7252 raft_consensus.cc:2229] T 9fbe2eaba5344dfaa29e78a0d73f224a P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:56.082788 7252 raft_consensus.cc:2258] T 9fbe2eaba5344dfaa29e78a0d73f224a P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:56.083417 7252 ts_tablet_manager.cc:1875] T 9fbe2eaba5344dfaa29e78a0d73f224a P 6d1d7de19459435fbde1999e3dfd226b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:56.089526 7252 ts_tablet_manager.cc:1888] T 9fbe2eaba5344dfaa29e78a0d73f224a P 6d1d7de19459435fbde1999e3dfd226b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:13:56.089722 7252 log.cc:1196] T 9fbe2eaba5344dfaa29e78a0d73f224a P 6d1d7de19459435fbde1999e3dfd226b: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal/wals/9fbe2eaba5344dfaa29e78a0d73f224a
I20240503 08:13:56.090401 27848 catalog_manager.cc:4830] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): tablet 9fbe2eaba5344dfaa29e78a0d73f224a (table rebalancer_extra_table_0 [id=10122b5266cf4446bb541a06acf3a16f]) successfully deleted
I20240503 08:13:56.335364 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:56.343820 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:56.360285 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:56.373602 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:56.389472 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:56.390754 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:56.425099 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:13:56.469892 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 08:13:56.539644 28329 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.65:39081, user_credentials={real_user=slave}} blocked reactor thread for 48221.3us
I20240503 08:13:56.626606 28390 consensus_queue.cc:237] T 1fca3d7f4b734eceb210e1186478eb72 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:13:56.628991 28133 raft_consensus.cc:1261] T 1fca3d7f4b734eceb210e1186478eb72 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:56.629127 30894 raft_consensus.cc:1261] T 1fca3d7f4b734eceb210e1186478eb72 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:56.629449 7050 consensus_queue.cc:1035] T 1fca3d7f4b734eceb210e1186478eb72 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:56.629953 7306 consensus_queue.cc:1035] T 1fca3d7f4b734eceb210e1186478eb72 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:56.635738 7050 raft_consensus.cc:2932] T 1fca3d7f4b734eceb210e1186478eb72 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:56.636138 30894 raft_consensus.cc:2932] T 1fca3d7f4b734eceb210e1186478eb72 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:56.643550 28132 raft_consensus.cc:2932] T 1fca3d7f4b734eceb210e1186478eb72 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:56.646626 27856 catalog_manager.cc:5484] T 1fca3d7f4b734eceb210e1186478eb72 P ce1496a47d084365ad786cd09e472a2c reported cstate change: config changed from index 2 to 3, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) evicted. New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:13:56.666285 28243 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fca3d7f4b734eceb210e1186478eb72 with delete_type TABLET_DATA_TOMBSTONED (TS aae860d7ae2b4002bf7794c278f9cc56 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:41960
I20240503 08:13:56.666889 7336 tablet_replica.cc:330] T 1fca3d7f4b734eceb210e1186478eb72 P aae860d7ae2b4002bf7794c278f9cc56: stopping tablet replica
I20240503 08:13:56.667089 7336 raft_consensus.cc:2229] T 1fca3d7f4b734eceb210e1186478eb72 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:56.667383 7336 raft_consensus.cc:2258] T 1fca3d7f4b734eceb210e1186478eb72 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:56.668391 7336 ts_tablet_manager.cc:1875] T 1fca3d7f4b734eceb210e1186478eb72 P aae860d7ae2b4002bf7794c278f9cc56: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:56.674758 7336 ts_tablet_manager.cc:1888] T 1fca3d7f4b734eceb210e1186478eb72 P aae860d7ae2b4002bf7794c278f9cc56: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:13:56.674930 7336 log.cc:1196] T 1fca3d7f4b734eceb210e1186478eb72 P aae860d7ae2b4002bf7794c278f9cc56: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal/wals/1fca3d7f4b734eceb210e1186478eb72
I20240503 08:13:56.675702 27847 catalog_manager.cc:4830] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): tablet 1fca3d7f4b734eceb210e1186478eb72 (table rebalancer_extra_table_66 [id=80792788b9164bd4af02769ad074bd3a]) successfully deleted
I20240503 08:13:56.899158 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:56.922761 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:56.931892 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:56.940492 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:56.946839 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:56.977429 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:13:56.978309 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:57.059433 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:57.166635 31027 consensus_queue.cc:237] T 48075c4a45d24e84bb2b9760710de3a0 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:13:57.168525 31160 raft_consensus.cc:1261] T 48075c4a45d24e84bb2b9760710de3a0 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer ced21ca983b749d0b4d4f6a0e2426bf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:57.168651 30894 raft_consensus.cc:1261] T 48075c4a45d24e84bb2b9760710de3a0 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer ced21ca983b749d0b4d4f6a0e2426bf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:57.169044 7219 consensus_queue.cc:1035] T 48075c4a45d24e84bb2b9760710de3a0 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:57.169262 7224 consensus_queue.cc:1035] T 48075c4a45d24e84bb2b9760710de3a0 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:57.171398 7224 raft_consensus.cc:2932] T 48075c4a45d24e84bb2b9760710de3a0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:57.172005 30894 raft_consensus.cc:2932] T 48075c4a45d24e84bb2b9760710de3a0 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:57.174239 31160 raft_consensus.cc:2932] T 48075c4a45d24e84bb2b9760710de3a0 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:57.177521 27857 catalog_manager.cc:5484] T 48075c4a45d24e84bb2b9760710de3a0 P ce1496a47d084365ad786cd09e472a2c reported cstate change: config changed from index 2 to 3, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New cstate: current_term: 1 leader_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:13:57.181517 30742 tablet_service.cc:1514] Processing DeleteTablet for tablet 48075c4a45d24e84bb2b9760710de3a0 with delete_type TABLET_DATA_TOMBSTONED (TS 6d1d7de19459435fbde1999e3dfd226b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46794
I20240503 08:13:57.182089 7413 tablet_replica.cc:330] T 48075c4a45d24e84bb2b9760710de3a0 P 6d1d7de19459435fbde1999e3dfd226b: stopping tablet replica
I20240503 08:13:57.182330 7413 raft_consensus.cc:2229] T 48075c4a45d24e84bb2b9760710de3a0 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:57.182570 7413 raft_consensus.cc:2258] T 48075c4a45d24e84bb2b9760710de3a0 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:57.183231 7413 ts_tablet_manager.cc:1875] T 48075c4a45d24e84bb2b9760710de3a0 P 6d1d7de19459435fbde1999e3dfd226b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:57.193593 7413 ts_tablet_manager.cc:1888] T 48075c4a45d24e84bb2b9760710de3a0 P 6d1d7de19459435fbde1999e3dfd226b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:13:57.193778 7413 log.cc:1196] T 48075c4a45d24e84bb2b9760710de3a0 P 6d1d7de19459435fbde1999e3dfd226b: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal/wals/48075c4a45d24e84bb2b9760710de3a0
I20240503 08:13:57.194757 27848 catalog_manager.cc:4830] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): tablet 48075c4a45d24e84bb2b9760710de3a0 (table rebalancer_extra_table_64 [id=00d43ce433a44235afa9ed3a6510e586]) successfully deleted
I20240503 08:13:57.456454 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:57.457322 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:57.476333 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:57.481004 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:13:57.482993 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:57.486361 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:57.555227 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:57.611644 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:57.770483 28133 consensus_queue.cc:237] T fa7e18041c294a239eb7b8f735b66c53 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:57.772517 28263 raft_consensus.cc:1261] T fa7e18041c294a239eb7b8f735b66c53 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:57.772946 7211 consensus_queue.cc:1035] T fa7e18041c294a239eb7b8f735b66c53 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:57.775702 7211 raft_consensus.cc:2932] T fa7e18041c294a239eb7b8f735b66c53 P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:57.779418 28263 raft_consensus.cc:2932] T fa7e18041c294a239eb7b8f735b66c53 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:57.780987 30895 raft_consensus.cc:1261] T fa7e18041c294a239eb7b8f735b66c53 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:57.781755 7211 consensus_queue.cc:1035] T fa7e18041c294a239eb7b8f735b66c53 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:57.783336 27859 catalog_manager.cc:5484] T fa7e18041c294a239eb7b8f735b66c53 P 1b411c6bb25549368041773d7b710d48 reported cstate change: config changed from index 2 to 3, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New cstate: current_term: 1 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:57.783761 30895 raft_consensus.cc:2932] T fa7e18041c294a239eb7b8f735b66c53 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:57.799786 30742 tablet_service.cc:1514] Processing DeleteTablet for tablet fa7e18041c294a239eb7b8f735b66c53 with delete_type TABLET_DATA_TOMBSTONED (TS 6d1d7de19459435fbde1999e3dfd226b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46794
I20240503 08:13:57.803321 7504 tablet_replica.cc:330] T fa7e18041c294a239eb7b8f735b66c53 P 6d1d7de19459435fbde1999e3dfd226b: stopping tablet replica
I20240503 08:13:57.803521 7504 raft_consensus.cc:2229] T fa7e18041c294a239eb7b8f735b66c53 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:57.803707 7504 raft_consensus.cc:2258] T fa7e18041c294a239eb7b8f735b66c53 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:57.804204 7504 ts_tablet_manager.cc:1875] T fa7e18041c294a239eb7b8f735b66c53 P 6d1d7de19459435fbde1999e3dfd226b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:57.810401 7504 ts_tablet_manager.cc:1888] T fa7e18041c294a239eb7b8f735b66c53 P 6d1d7de19459435fbde1999e3dfd226b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:13:57.810518 7504 log.cc:1196] T fa7e18041c294a239eb7b8f735b66c53 P 6d1d7de19459435fbde1999e3dfd226b: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal/wals/fa7e18041c294a239eb7b8f735b66c53
I20240503 08:13:57.811470 27848 catalog_manager.cc:4830] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): tablet fa7e18041c294a239eb7b8f735b66c53 (table rebalancer_extra_table_26 [id=1b577ea991674f799fea7ba7f25e80ad]) successfully deleted
I20240503 08:13:58.013669 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:58.031540 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:58.031984 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:58.088763 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:58.095300 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:58.150084 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:58.183001 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:58.195217 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:13:58.322830 31161 consensus_queue.cc:237] T 200387e6edce4ee9bf11331380923005 P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:58.324515 30895 raft_consensus.cc:1261] T 200387e6edce4ee9bf11331380923005 P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:58.324962 28390 raft_consensus.cc:1261] T 200387e6edce4ee9bf11331380923005 P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:58.325022 7330 consensus_queue.cc:1035] T 200387e6edce4ee9bf11331380923005 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:58.325857 7330 consensus_queue.cc:1035] T 200387e6edce4ee9bf11331380923005 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:58.327347 7221 raft_consensus.cc:2932] T 200387e6edce4ee9bf11331380923005 P b00f244c69df401997ea99d492218a98 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:58.328938 28390 raft_consensus.cc:2932] T 200387e6edce4ee9bf11331380923005 P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:58.333839 30895 raft_consensus.cc:2932] T 200387e6edce4ee9bf11331380923005 P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:58.336186 27857 catalog_manager.cc:5484] T 200387e6edce4ee9bf11331380923005 P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: config changed from index 2 to 4, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New cstate: current_term: 2 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:58.342619 31006 tablet_service.cc:1514] Processing DeleteTablet for tablet 200387e6edce4ee9bf11331380923005 with delete_type TABLET_DATA_TOMBSTONED (TS ced21ca983b749d0b4d4f6a0e2426bf8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47020
I20240503 08:13:58.343228 7593 tablet_replica.cc:330] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8: stopping tablet replica
I20240503 08:13:58.345978 7593 raft_consensus.cc:2229] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:58.346243 7593 raft_consensus.cc:2258] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:58.347129 7593 ts_tablet_manager.cc:1875] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:58.354252 7593 ts_tablet_manager.cc:1888] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240503 08:13:58.354421 7593 log.cc:1196] T 200387e6edce4ee9bf11331380923005 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal/wals/200387e6edce4ee9bf11331380923005
I20240503 08:13:58.355012 27847 catalog_manager.cc:4830] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): tablet 200387e6edce4ee9bf11331380923005 (table rebalancer_extra_table_20 [id=e37b245cb78d400384ac439a89fb8706]) successfully deleted
I20240503 08:13:58.430796 30759 consensus_queue.cc:237] T e3e58f895edb4095ba3d77653c9197af P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:58.432910 31161 raft_consensus.cc:1261] T e3e58f895edb4095ba3d77653c9197af P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:58.433079 30895 raft_consensus.cc:1261] T e3e58f895edb4095ba3d77653c9197af P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:58.433403 7620 consensus_queue.cc:1035] T e3e58f895edb4095ba3d77653c9197af P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 08:13:58.441933 30697 consensus_peers.cc:485] T e3e58f895edb4095ba3d77653c9197af P 6d1d7de19459435fbde1999e3dfd226b -> Peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Couldn't send request to peer 833582ff68c94269be364ebd187a75db. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e3e58f895edb4095ba3d77653c9197af. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:58.448668 7620 consensus_queue.cc:1035] T e3e58f895edb4095ba3d77653c9197af P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:58.450701 7062 raft_consensus.cc:2932] T e3e58f895edb4095ba3d77653c9197af P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:58.451186 30895 raft_consensus.cc:2932] T e3e58f895edb4095ba3d77653c9197af P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:58.455978 31161 raft_consensus.cc:2932] T e3e58f895edb4095ba3d77653c9197af P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:58.458387 27854 catalog_manager.cc:5484] T e3e58f895edb4095ba3d77653c9197af P ce1496a47d084365ad786cd09e472a2c reported cstate change: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:58.495821 27999 consensus_queue.cc:237] T dcc026d6d9bf40059fd6298c40b04c1a P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:58.509317 31027 raft_consensus.cc:1261] T dcc026d6d9bf40059fd6298c40b04c1a P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:58.509332 31161 raft_consensus.cc:1261] T dcc026d6d9bf40059fd6298c40b04c1a P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:58.509887 7623 consensus_queue.cc:1035] T dcc026d6d9bf40059fd6298c40b04c1a P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:58.510360 6765 consensus_queue.cc:1035] T dcc026d6d9bf40059fd6298c40b04c1a P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 08:13:58.514451 27939 consensus_peers.cc:485] T dcc026d6d9bf40059fd6298c40b04c1a P 833582ff68c94269be364ebd187a75db -> Peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): Couldn't send request to peer d09a11a03989421b9f28019dc1c14cc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dcc026d6d9bf40059fd6298c40b04c1a. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:58.526918 6765 raft_consensus.cc:2932] T dcc026d6d9bf40059fd6298c40b04c1a P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:58.527448 31027 raft_consensus.cc:2932] T dcc026d6d9bf40059fd6298c40b04c1a P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:58.534569 27860 catalog_manager.cc:5484] T dcc026d6d9bf40059fd6298c40b04c1a P ced21ca983b749d0b4d4f6a0e2426bf8 reported cstate change: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:58.537608 31161 raft_consensus.cc:2932] T dcc026d6d9bf40059fd6298c40b04c1a P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:13:58.608292 28389 consensus_queue.cc:237] T 68b1a80883614b1ca54fbd88bc769dff P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
W20240503 08:13:58.610517 28329 consensus_peers.cc:485] T 68b1a80883614b1ca54fbd88bc769dff P d09a11a03989421b9f28019dc1c14cc2 -> Peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Couldn't send request to peer 833582ff68c94269be364ebd187a75db. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 68b1a80883614b1ca54fbd88bc769dff. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:58.610546 31161 raft_consensus.cc:1261] T 68b1a80883614b1ca54fbd88bc769dff P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:58.610697 30894 raft_consensus.cc:1261] T 68b1a80883614b1ca54fbd88bc769dff P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:58.610996 7479 consensus_queue.cc:1035] T 68b1a80883614b1ca54fbd88bc769dff P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:58.611240 7306 consensus_queue.cc:1035] T 68b1a80883614b1ca54fbd88bc769dff P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:58.618644 7479 raft_consensus.cc:2932] T 68b1a80883614b1ca54fbd88bc769dff P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:58.619155 30894 raft_consensus.cc:2932] T 68b1a80883614b1ca54fbd88bc769dff P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:58.624526 31161 raft_consensus.cc:2932] T 68b1a80883614b1ca54fbd88bc769dff P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:58.625350 27860 catalog_manager.cc:5484] T 68b1a80883614b1ca54fbd88bc769dff P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:58.702250 30894 consensus_queue.cc:237] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:58.705523 28000 raft_consensus.cc:1261] T e7b54127ad4245838c4cc05071378472 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:58.705804 31027 raft_consensus.cc:1261] T e7b54127ad4245838c4cc05071378472 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:58.706327 7680 consensus_queue.cc:1035] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 08:13:58.706818 30828 consensus_peers.cc:485] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e7b54127ad4245838c4cc05071378472. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:58.706954 7289 consensus_queue.cc:1035] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:58.726840 7680 raft_consensus.cc:2932] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:58.727346 31027 raft_consensus.cc:2932] T e7b54127ad4245838c4cc05071378472 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:58.733075 28000 raft_consensus.cc:2932] T e7b54127ad4245838c4cc05071378472 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:58.734781 27860 catalog_manager.cc:5484] T e7b54127ad4245838c4cc05071378472 P ced21ca983b749d0b4d4f6a0e2426bf8 reported cstate change: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New cstate: current_term: 1 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:13:58.782501 30759 consensus_queue.cc:237] T 4ebf45bdc432451e87b7c98d435a76d2 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:58.785224 31161 raft_consensus.cc:1261] T 4ebf45bdc432451e87b7c98d435a76d2 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:13:58.785540 30697 consensus_peers.cc:485] T 4ebf45bdc432451e87b7c98d435a76d2 P 6d1d7de19459435fbde1999e3dfd226b -> Peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Couldn't send request to peer 833582ff68c94269be364ebd187a75db. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4ebf45bdc432451e87b7c98d435a76d2. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:58.786054 7620 consensus_queue.cc:1035] T 4ebf45bdc432451e87b7c98d435a76d2 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:58.787148 31027 raft_consensus.cc:1261] T 4ebf45bdc432451e87b7c98d435a76d2 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:58.787727 7620 consensus_queue.cc:1035] T 4ebf45bdc432451e87b7c98d435a76d2 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:58.789976 7624 raft_consensus.cc:2932] T 4ebf45bdc432451e87b7c98d435a76d2 P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:58.790477 31027 raft_consensus.cc:2932] T 4ebf45bdc432451e87b7c98d435a76d2 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:58.796357 27860 catalog_manager.cc:5484] T 4ebf45bdc432451e87b7c98d435a76d2 P ced21ca983b749d0b4d4f6a0e2426bf8 reported cstate change: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:58.796804 31161 raft_consensus.cc:2932] T 4ebf45bdc432451e87b7c98d435a76d2 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:13:58.974107 7745 ts_tablet_manager.cc:902] T dcc026d6d9bf40059fd6298c40b04c1a P d09a11a03989421b9f28019dc1c14cc2: Initiating tablet copy from peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:13:59.090050 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:59.116191 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:59.135327 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:59.136915 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:59.144956 7770 ts_tablet_manager.cc:902] T e3e58f895edb4095ba3d77653c9197af P 833582ff68c94269be364ebd187a75db: Initiating tablet copy from peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:13:59.155454 7757 ts_tablet_manager.cc:902] T 68b1a80883614b1ca54fbd88bc769dff P 833582ff68c94269be364ebd187a75db: Initiating tablet copy from peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:13:59.157531 7773 ts_tablet_manager.cc:902] T e7b54127ad4245838c4cc05071378472 P aae860d7ae2b4002bf7794c278f9cc56: Initiating tablet copy from peer ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:13:59.194420 7745 tablet_copy_client.cc:308] T dcc026d6d9bf40059fd6298c40b04c1a P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.65:39081
I20240503 08:13:59.194950 28023 tablet_copy_service.cc:140] P 833582ff68c94269be364ebd187a75db: Received BeginTabletCopySession request for tablet dcc026d6d9bf40059fd6298c40b04c1a from peer d09a11a03989421b9f28019dc1c14cc2 ({username='slave'} at 127.20.230.68:42231)
I20240503 08:13:59.195089 28023 tablet_copy_service.cc:161] P 833582ff68c94269be364ebd187a75db: Beginning new tablet copy session on tablet dcc026d6d9bf40059fd6298c40b04c1a from peer d09a11a03989421b9f28019dc1c14cc2 at {username='slave'} at 127.20.230.68:42231: session id = d09a11a03989421b9f28019dc1c14cc2-dcc026d6d9bf40059fd6298c40b04c1a
I20240503 08:13:59.201623 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:59.204494 28023 tablet_copy_source_session.cc:205] T dcc026d6d9bf40059fd6298c40b04c1a P 833582ff68c94269be364ebd187a75db: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:59.206341 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:59.206662 7745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc026d6d9bf40059fd6298c40b04c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:59.216153 7773 tablet_copy_client.cc:308] T e7b54127ad4245838c4cc05071378472 P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.70:33747
I20240503 08:13:59.216706 30915 tablet_copy_service.cc:140] P ce1496a47d084365ad786cd09e472a2c: Received BeginTabletCopySession request for tablet e7b54127ad4245838c4cc05071378472 from peer aae860d7ae2b4002bf7794c278f9cc56 ({username='slave'} at 127.20.230.67:50141)
I20240503 08:13:59.216830 30915 tablet_copy_service.cc:161] P ce1496a47d084365ad786cd09e472a2c: Beginning new tablet copy session on tablet e7b54127ad4245838c4cc05071378472 from peer aae860d7ae2b4002bf7794c278f9cc56 at {username='slave'} at 127.20.230.67:50141: session id = aae860d7ae2b4002bf7794c278f9cc56-e7b54127ad4245838c4cc05071378472
I20240503 08:13:59.219401 30915 tablet_copy_source_session.cc:205] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:59.220588 7773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7b54127ad4245838c4cc05071378472. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:59.228058 7745 tablet_copy_client.cc:783] T dcc026d6d9bf40059fd6298c40b04c1a P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:59.228264 7745 tablet_copy_client.cc:647] T dcc026d6d9bf40059fd6298c40b04c1a P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:59.228369 7773 tablet_copy_client.cc:783] T e7b54127ad4245838c4cc05071378472 P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:59.228549 7773 tablet_copy_client.cc:647] T e7b54127ad4245838c4cc05071378472 P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:59.231444 7773 tablet_copy_client.cc:523] T e7b54127ad4245838c4cc05071378472 P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:59.233208 7770 tablet_copy_client.cc:308] T e3e58f895edb4095ba3d77653c9197af P 833582ff68c94269be364ebd187a75db: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.69:34725
I20240503 08:13:59.234244 7773 tablet_bootstrap.cc:492] T e7b54127ad4245838c4cc05071378472 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:13:59.239874 30783 tablet_copy_service.cc:140] P 6d1d7de19459435fbde1999e3dfd226b: Received BeginTabletCopySession request for tablet e3e58f895edb4095ba3d77653c9197af from peer 833582ff68c94269be364ebd187a75db ({username='slave'} at 127.20.230.65:42993)
I20240503 08:13:59.240024 30783 tablet_copy_service.cc:161] P 6d1d7de19459435fbde1999e3dfd226b: Beginning new tablet copy session on tablet e3e58f895edb4095ba3d77653c9197af from peer 833582ff68c94269be364ebd187a75db at {username='slave'} at 127.20.230.65:42993: session id = 833582ff68c94269be364ebd187a75db-e3e58f895edb4095ba3d77653c9197af
I20240503 08:13:59.240677 7745 tablet_copy_client.cc:523] T dcc026d6d9bf40059fd6298c40b04c1a P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:59.243523 7745 tablet_bootstrap.cc:492] T dcc026d6d9bf40059fd6298c40b04c1a P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:13:59.250933 30783 tablet_copy_source_session.cc:205] T e3e58f895edb4095ba3d77653c9197af P 6d1d7de19459435fbde1999e3dfd226b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:59.255898 7773 tablet_bootstrap.cc:492] T e7b54127ad4245838c4cc05071378472 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:59.256310 7773 tablet_bootstrap.cc:492] T e7b54127ad4245838c4cc05071378472 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap complete.
I20240503 08:13:59.256601 7773 ts_tablet_manager.cc:1367] T e7b54127ad4245838c4cc05071378472 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.000s
I20240503 08:13:59.256855 7745 tablet_bootstrap.cc:492] T dcc026d6d9bf40059fd6298c40b04c1a P d09a11a03989421b9f28019dc1c14cc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:59.257207 7745 tablet_bootstrap.cc:492] T dcc026d6d9bf40059fd6298c40b04c1a P d09a11a03989421b9f28019dc1c14cc2: Bootstrap complete.
I20240503 08:13:59.257566 7745 ts_tablet_manager.cc:1367] T dcc026d6d9bf40059fd6298c40b04c1a P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.000s
I20240503 08:13:59.257428 7773 raft_consensus.cc:348] T e7b54127ad4245838c4cc05071378472 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:59.257810 7773 raft_consensus.cc:729] T e7b54127ad4245838c4cc05071378472 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:13:59.258078 7773 consensus_queue.cc:260] T e7b54127ad4245838c4cc05071378472 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:13:59.258311 7745 raft_consensus.cc:348] T dcc026d6d9bf40059fd6298c40b04c1a P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:59.258426 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:13:59.258508 7745 raft_consensus.cc:729] T dcc026d6d9bf40059fd6298c40b04c1a P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:13:59.258931 7773 ts_tablet_manager.cc:1398] T e7b54127ad4245838c4cc05071378472 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:59.258869 7745 consensus_queue.cc:260] T dcc026d6d9bf40059fd6298c40b04c1a P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:13:59.259847 7745 ts_tablet_manager.cc:1398] T dcc026d6d9bf40059fd6298c40b04c1a P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:59.260057 30915 tablet_copy_service.cc:342] P ce1496a47d084365ad786cd09e472a2c: Request end of tablet copy session aae860d7ae2b4002bf7794c278f9cc56-e7b54127ad4245838c4cc05071378472 received from {username='slave'} at 127.20.230.67:50141
I20240503 08:13:59.260174 30915 tablet_copy_service.cc:433] P ce1496a47d084365ad786cd09e472a2c: ending tablet copy session aae860d7ae2b4002bf7794c278f9cc56-e7b54127ad4245838c4cc05071378472 on tablet e7b54127ad4245838c4cc05071378472 with peer aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:13:59.260499 28023 tablet_copy_service.cc:342] P 833582ff68c94269be364ebd187a75db: Request end of tablet copy session d09a11a03989421b9f28019dc1c14cc2-dcc026d6d9bf40059fd6298c40b04c1a received from {username='slave'} at 127.20.230.68:42231
I20240503 08:13:59.260632 28023 tablet_copy_service.cc:433] P 833582ff68c94269be364ebd187a75db: ending tablet copy session d09a11a03989421b9f28019dc1c14cc2-dcc026d6d9bf40059fd6298c40b04c1a on tablet dcc026d6d9bf40059fd6298c40b04c1a with peer d09a11a03989421b9f28019dc1c14cc2
I20240503 08:13:59.268299 7757 tablet_copy_client.cc:308] T 68b1a80883614b1ca54fbd88bc769dff P 833582ff68c94269be364ebd187a75db: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.68:46211
I20240503 08:13:59.268990 28413 tablet_copy_service.cc:140] P d09a11a03989421b9f28019dc1c14cc2: Received BeginTabletCopySession request for tablet 68b1a80883614b1ca54fbd88bc769dff from peer 833582ff68c94269be364ebd187a75db ({username='slave'} at 127.20.230.65:36529)
I20240503 08:13:59.269121 28413 tablet_copy_service.cc:161] P d09a11a03989421b9f28019dc1c14cc2: Beginning new tablet copy session on tablet 68b1a80883614b1ca54fbd88bc769dff from peer 833582ff68c94269be364ebd187a75db at {username='slave'} at 127.20.230.65:36529: session id = 833582ff68c94269be364ebd187a75db-68b1a80883614b1ca54fbd88bc769dff
I20240503 08:13:59.273870 28413 tablet_copy_source_session.cc:205] T 68b1a80883614b1ca54fbd88bc769dff P d09a11a03989421b9f28019dc1c14cc2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:59.275461 7757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68b1a80883614b1ca54fbd88bc769dff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:59.281337 7770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3e58f895edb4095ba3d77653c9197af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:59.286968 7770 tablet_copy_client.cc:783] T e3e58f895edb4095ba3d77653c9197af P 833582ff68c94269be364ebd187a75db: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:59.287225 7770 tablet_copy_client.cc:647] T e3e58f895edb4095ba3d77653c9197af P 833582ff68c94269be364ebd187a75db: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:59.294245 7757 tablet_copy_client.cc:783] T 68b1a80883614b1ca54fbd88bc769dff P 833582ff68c94269be364ebd187a75db: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:59.294515 7757 tablet_copy_client.cc:647] T 68b1a80883614b1ca54fbd88bc769dff P 833582ff68c94269be364ebd187a75db: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:59.296114 7770 tablet_copy_client.cc:523] T e3e58f895edb4095ba3d77653c9197af P 833582ff68c94269be364ebd187a75db: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:59.299180 7770 tablet_bootstrap.cc:492] T e3e58f895edb4095ba3d77653c9197af P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:13:59.299901 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:59.302417 7757 tablet_copy_client.cc:523] T 68b1a80883614b1ca54fbd88bc769dff P 833582ff68c94269be364ebd187a75db: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:59.307179 7757 tablet_bootstrap.cc:492] T 68b1a80883614b1ca54fbd88bc769dff P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:13:59.327256 7770 tablet_bootstrap.cc:492] T e3e58f895edb4095ba3d77653c9197af P 833582ff68c94269be364ebd187a75db: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:59.327791 7770 tablet_bootstrap.cc:492] T e3e58f895edb4095ba3d77653c9197af P 833582ff68c94269be364ebd187a75db: Bootstrap complete.
I20240503 08:13:59.328089 7770 ts_tablet_manager.cc:1367] T e3e58f895edb4095ba3d77653c9197af P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.004s
I20240503 08:13:59.328905 7770 raft_consensus.cc:348] T e3e58f895edb4095ba3d77653c9197af P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:59.329245 7770 raft_consensus.cc:729] T e3e58f895edb4095ba3d77653c9197af P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:13:59.329524 7770 consensus_queue.cc:260] T e3e58f895edb4095ba3d77653c9197af P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:59.335702 7770 ts_tablet_manager.cc:1398] T e3e58f895edb4095ba3d77653c9197af P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240503 08:13:59.337600 30783 tablet_copy_service.cc:342] P 6d1d7de19459435fbde1999e3dfd226b: Request end of tablet copy session 833582ff68c94269be364ebd187a75db-e3e58f895edb4095ba3d77653c9197af received from {username='slave'} at 127.20.230.65:42993
I20240503 08:13:59.337723 30783 tablet_copy_service.cc:433] P 6d1d7de19459435fbde1999e3dfd226b: ending tablet copy session 833582ff68c94269be364ebd187a75db-e3e58f895edb4095ba3d77653c9197af on tablet e3e58f895edb4095ba3d77653c9197af with peer 833582ff68c94269be364ebd187a75db
I20240503 08:13:59.343329 7788 ts_tablet_manager.cc:902] T 4ebf45bdc432451e87b7c98d435a76d2 P 833582ff68c94269be364ebd187a75db: Initiating tablet copy from peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:13:59.354007 7757 tablet_bootstrap.cc:492] T 68b1a80883614b1ca54fbd88bc769dff P 833582ff68c94269be364ebd187a75db: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:59.354526 7757 tablet_bootstrap.cc:492] T 68b1a80883614b1ca54fbd88bc769dff P 833582ff68c94269be364ebd187a75db: Bootstrap complete.
I20240503 08:13:59.354903 7757 ts_tablet_manager.cc:1367] T 68b1a80883614b1ca54fbd88bc769dff P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.048s user 0.014s sys 0.000s
I20240503 08:13:59.355762 7757 raft_consensus.cc:348] T 68b1a80883614b1ca54fbd88bc769dff P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:59.356065 7757 raft_consensus.cc:729] T 68b1a80883614b1ca54fbd88bc769dff P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:13:59.356370 7757 consensus_queue.cc:260] T 68b1a80883614b1ca54fbd88bc769dff P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:59.356963 7757 ts_tablet_manager.cc:1398] T 68b1a80883614b1ca54fbd88bc769dff P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:59.357901 28413 tablet_copy_service.cc:342] P d09a11a03989421b9f28019dc1c14cc2: Request end of tablet copy session 833582ff68c94269be364ebd187a75db-68b1a80883614b1ca54fbd88bc769dff received from {username='slave'} at 127.20.230.65:36529
I20240503 08:13:59.358039 28413 tablet_copy_service.cc:433] P d09a11a03989421b9f28019dc1c14cc2: ending tablet copy session 833582ff68c94269be364ebd187a75db-68b1a80883614b1ca54fbd88bc769dff on tablet 68b1a80883614b1ca54fbd88bc769dff with peer 833582ff68c94269be364ebd187a75db
I20240503 08:13:59.397975 7788 tablet_copy_client.cc:308] T 4ebf45bdc432451e87b7c98d435a76d2 P 833582ff68c94269be364ebd187a75db: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.69:34725
I20240503 08:13:59.398928 30783 tablet_copy_service.cc:140] P 6d1d7de19459435fbde1999e3dfd226b: Received BeginTabletCopySession request for tablet 4ebf45bdc432451e87b7c98d435a76d2 from peer 833582ff68c94269be364ebd187a75db ({username='slave'} at 127.20.230.65:42993)
I20240503 08:13:59.399075 30783 tablet_copy_service.cc:161] P 6d1d7de19459435fbde1999e3dfd226b: Beginning new tablet copy session on tablet 4ebf45bdc432451e87b7c98d435a76d2 from peer 833582ff68c94269be364ebd187a75db at {username='slave'} at 127.20.230.65:42993: session id = 833582ff68c94269be364ebd187a75db-4ebf45bdc432451e87b7c98d435a76d2
I20240503 08:13:59.401793 30783 tablet_copy_source_session.cc:205] T 4ebf45bdc432451e87b7c98d435a76d2 P 6d1d7de19459435fbde1999e3dfd226b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:59.403787 7788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ebf45bdc432451e87b7c98d435a76d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:59.414858 7788 tablet_copy_client.cc:783] T 4ebf45bdc432451e87b7c98d435a76d2 P 833582ff68c94269be364ebd187a75db: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:59.415696 7788 tablet_copy_client.cc:647] T 4ebf45bdc432451e87b7c98d435a76d2 P 833582ff68c94269be364ebd187a75db: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:59.420931 7788 tablet_copy_client.cc:523] T 4ebf45bdc432451e87b7c98d435a76d2 P 833582ff68c94269be364ebd187a75db: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:59.425673 7788 tablet_bootstrap.cc:492] T 4ebf45bdc432451e87b7c98d435a76d2 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:13:59.434736 7788 tablet_bootstrap.cc:492] T 4ebf45bdc432451e87b7c98d435a76d2 P 833582ff68c94269be364ebd187a75db: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:59.435232 7788 tablet_bootstrap.cc:492] T 4ebf45bdc432451e87b7c98d435a76d2 P 833582ff68c94269be364ebd187a75db: Bootstrap complete.
I20240503 08:13:59.436532 7788 ts_tablet_manager.cc:1367] T 4ebf45bdc432451e87b7c98d435a76d2 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20240503 08:13:59.437351 7788 raft_consensus.cc:348] T 4ebf45bdc432451e87b7c98d435a76d2 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:59.437600 7788 raft_consensus.cc:729] T 4ebf45bdc432451e87b7c98d435a76d2 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:13:59.437917 7788 consensus_queue.cc:260] T 4ebf45bdc432451e87b7c98d435a76d2 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:13:59.438925 7788 ts_tablet_manager.cc:1398] T 4ebf45bdc432451e87b7c98d435a76d2 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:13:59.441644 30783 tablet_copy_service.cc:342] P 6d1d7de19459435fbde1999e3dfd226b: Request end of tablet copy session 833582ff68c94269be364ebd187a75db-4ebf45bdc432451e87b7c98d435a76d2 received from {username='slave'} at 127.20.230.65:42993
I20240503 08:13:59.441802 30783 tablet_copy_service.cc:433] P 6d1d7de19459435fbde1999e3dfd226b: ending tablet copy session 833582ff68c94269be364ebd187a75db-4ebf45bdc432451e87b7c98d435a76d2 on tablet 4ebf45bdc432451e87b7c98d435a76d2 with peer 833582ff68c94269be364ebd187a75db
I20240503 08:13:59.512944 28390 raft_consensus.cc:1203] T dcc026d6d9bf40059fd6298c40b04c1a P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:59.535199 28000 raft_consensus.cc:1203] T 68b1a80883614b1ca54fbd88bc769dff P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:59.558465 28000 raft_consensus.cc:1203] T e3e58f895edb4095ba3d77653c9197af P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:59.689397 28262 raft_consensus.cc:1203] T e7b54127ad4245838c4cc05071378472 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:59.716289 28000 raft_consensus.cc:1203] T 4ebf45bdc432451e87b7c98d435a76d2 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:59.789441 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:59.799634 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:59.833539 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:59.880100 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:13:59.885943 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:59.891252 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:13:59.933575 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:13:59.939246 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:14:00.054183 30894 consensus_queue.cc:237] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:00.055963 28262 raft_consensus.cc:1261] T e7b54127ad4245838c4cc05071378472 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:00.056479 7391 consensus_queue.cc:1035] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:00.058034 28003 raft_consensus.cc:1261] T e7b54127ad4245838c4cc05071378472 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:00.058491 7391 consensus_queue.cc:1035] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:00.058905 7391 raft_consensus.cc:2932] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:00.059631 28262 raft_consensus.cc:2932] T e7b54127ad4245838c4cc05071378472 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:00.065202 27860 catalog_manager.cc:5484] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c reported cstate change: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New cstate: current_term: 1 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:00.080097 28000 raft_consensus.cc:2932] T e7b54127ad4245838c4cc05071378472 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:00.091461 31006 tablet_service.cc:1514] Processing DeleteTablet for tablet e7b54127ad4245838c4cc05071378472 with delete_type TABLET_DATA_TOMBSTONED (TS ced21ca983b749d0b4d4f6a0e2426bf8 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47020
I20240503 08:14:00.092010 7894 tablet_replica.cc:330] T e7b54127ad4245838c4cc05071378472 P ced21ca983b749d0b4d4f6a0e2426bf8: stopping tablet replica
I20240503 08:14:00.092238 7894 raft_consensus.cc:2229] T e7b54127ad4245838c4cc05071378472 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:00.092468 7894 raft_consensus.cc:2258] T e7b54127ad4245838c4cc05071378472 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:00.093055 7894 ts_tablet_manager.cc:1875] T e7b54127ad4245838c4cc05071378472 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:00.098371 7894 ts_tablet_manager.cc:1888] T e7b54127ad4245838c4cc05071378472 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:14:00.098542 7894 log.cc:1196] T e7b54127ad4245838c4cc05071378472 P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal/wals/e7b54127ad4245838c4cc05071378472
I20240503 08:14:00.099107 27847 catalog_manager.cc:4830] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): tablet e7b54127ad4245838c4cc05071378472 (table rebalancer_extra_table_6 [id=a80c6942ff454bcfb6a142a36fb80741]) successfully deleted
I20240503 08:14:00.347723 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:00.355886 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:00.371398 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:00.394397 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:14:00.403618 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:00.420641 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:00.446810 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:14:00.454670 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:00.602376 28391 consensus_queue.cc:237] T 68b1a80883614b1ca54fbd88bc769dff P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:14:00.604029 28000 raft_consensus.cc:1261] T 68b1a80883614b1ca54fbd88bc769dff P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:00.604148 31160 raft_consensus.cc:1261] T 68b1a80883614b1ca54fbd88bc769dff P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:00.604499 7652 consensus_queue.cc:1035] T 68b1a80883614b1ca54fbd88bc769dff P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:00.604727 7479 consensus_queue.cc:1035] T 68b1a80883614b1ca54fbd88bc769dff P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:00.607034 7479 raft_consensus.cc:2932] T 68b1a80883614b1ca54fbd88bc769dff P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:14:00.607901 31160 raft_consensus.cc:2932] T 68b1a80883614b1ca54fbd88bc769dff P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:14:00.610360 28000 raft_consensus.cc:2932] T 68b1a80883614b1ca54fbd88bc769dff P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:14:00.614463 27855 catalog_manager.cc:5484] T 68b1a80883614b1ca54fbd88bc769dff P b00f244c69df401997ea99d492218a98 reported cstate change: config changed from index 2 to 3, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:14:00.625914 30874 tablet_service.cc:1514] Processing DeleteTablet for tablet 68b1a80883614b1ca54fbd88bc769dff with delete_type TABLET_DATA_TOMBSTONED (TS ce1496a47d084365ad786cd09e472a2c not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:34426
I20240503 08:14:00.626494 7965 tablet_replica.cc:330] T 68b1a80883614b1ca54fbd88bc769dff P ce1496a47d084365ad786cd09e472a2c: stopping tablet replica
I20240503 08:14:00.627151 7965 raft_consensus.cc:2229] T 68b1a80883614b1ca54fbd88bc769dff P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:00.627435 7965 raft_consensus.cc:2258] T 68b1a80883614b1ca54fbd88bc769dff P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:00.628010 7965 ts_tablet_manager.cc:1875] T 68b1a80883614b1ca54fbd88bc769dff P ce1496a47d084365ad786cd09e472a2c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:00.633483 7965 ts_tablet_manager.cc:1888] T 68b1a80883614b1ca54fbd88bc769dff P ce1496a47d084365ad786cd09e472a2c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:14:00.633603 7965 log.cc:1196] T 68b1a80883614b1ca54fbd88bc769dff P ce1496a47d084365ad786cd09e472a2c: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal/wals/68b1a80883614b1ca54fbd88bc769dff
I20240503 08:14:00.634282 27846 catalog_manager.cc:4830] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): tablet 68b1a80883614b1ca54fbd88bc769dff (table rebalancer_extra_table_18 [id=e4d6c0dac75042f984dc1aa8fa801cee]) successfully deleted
I20240503 08:14:00.875085 27851 maintenance_manager.cc:419] P d2b7d1c3c8a84f53b716ab764eecfa71: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033375
I20240503 08:14:00.940219 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:00.950626 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:00.976887 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:14:01.002643 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:01.011539 27850 maintenance_manager.cc:643] P d2b7d1c3c8a84f53b716ab764eecfa71: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.136s user 0.064s sys 0.007s Metrics: {"bytes_written":105772,"cfile_init":1,"dirs.queue_time_us":15922,"dirs.run_cpu_time_us":224,"dirs.run_wall_time_us":906,"drs_written":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1619,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":293,"thread_start_us":18519,"threads_started":1}
I20240503 08:14:01.013162 27851 maintenance_manager.cc:419] P d2b7d1c3c8a84f53b716ab764eecfa71: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 33474 bytes on disk
I20240503 08:14:01.017195 27850 maintenance_manager.cc:643] P d2b7d1c3c8a84f53b716ab764eecfa71: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240503 08:14:01.030200 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:14:01.047221 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:01.073563 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:01.124878 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:01.264459 28131 consensus_queue.cc:237] T 35c62760e1de4c2f88d06b97d0286c4c P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:14:01.266144 30761 raft_consensus.cc:1261] T 35c62760e1de4c2f88d06b97d0286c4c P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:01.266194 28000 raft_consensus.cc:1261] T 35c62760e1de4c2f88d06b97d0286c4c P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:01.266700 7840 consensus_queue.cc:1035] T 35c62760e1de4c2f88d06b97d0286c4c P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:01.266923 7211 consensus_queue.cc:1035] T 35c62760e1de4c2f88d06b97d0286c4c P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:01.269721 7211 raft_consensus.cc:2932] T 35c62760e1de4c2f88d06b97d0286c4c P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:14:01.270637 28003 raft_consensus.cc:2932] T 35c62760e1de4c2f88d06b97d0286c4c P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:14:01.277137 27856 catalog_manager.cc:5484] T 35c62760e1de4c2f88d06b97d0286c4c P 833582ff68c94269be364ebd187a75db reported cstate change: config changed from index 2 to 3, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New cstate: current_term: 1 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:14:01.291524 30761 raft_consensus.cc:2932] T 35c62760e1de4c2f88d06b97d0286c4c P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:14:01.306502 30874 tablet_service.cc:1514] Processing DeleteTablet for tablet 35c62760e1de4c2f88d06b97d0286c4c with delete_type TABLET_DATA_TOMBSTONED (TS ce1496a47d084365ad786cd09e472a2c not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:34426
I20240503 08:14:01.307042 8064 tablet_replica.cc:330] T 35c62760e1de4c2f88d06b97d0286c4c P ce1496a47d084365ad786cd09e472a2c: stopping tablet replica
I20240503 08:14:01.307257 8064 raft_consensus.cc:2229] T 35c62760e1de4c2f88d06b97d0286c4c P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:01.307523 8064 raft_consensus.cc:2258] T 35c62760e1de4c2f88d06b97d0286c4c P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:01.308254 8064 ts_tablet_manager.cc:1875] T 35c62760e1de4c2f88d06b97d0286c4c P ce1496a47d084365ad786cd09e472a2c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:01.317591 8064 ts_tablet_manager.cc:1888] T 35c62760e1de4c2f88d06b97d0286c4c P ce1496a47d084365ad786cd09e472a2c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:14:01.317716 8064 log.cc:1196] T 35c62760e1de4c2f88d06b97d0286c4c P ce1496a47d084365ad786cd09e472a2c: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal/wals/35c62760e1de4c2f88d06b97d0286c4c
I20240503 08:14:01.318480 27846 catalog_manager.cc:4830] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): tablet 35c62760e1de4c2f88d06b97d0286c4c (table rebalancer_extra_table_26 [id=1b577ea991674f799fea7ba7f25e80ad]) successfully deleted
I20240503 08:14:01.632999 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:01.636514 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:01.655714 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:14:01.687160 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:01.727741 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:01.728986 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:14:01.772855 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:01.798002 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:01.922233 30895 consensus_queue.cc:237] T 49b145f6946647248c967cf4badf8b40 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:14:01.924077 31026 raft_consensus.cc:1261] T 49b145f6946647248c967cf4badf8b40 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:14:01.924176 28003 raft_consensus.cc:1261] T 49b145f6946647248c967cf4badf8b40 P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:14:01.924568 7877 consensus_queue.cc:1035] T 49b145f6946647248c967cf4badf8b40 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:01.925606 7877 consensus_queue.cc:1035] T 49b145f6946647248c967cf4badf8b40 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:01.927249 7391 raft_consensus.cc:2932] T 49b145f6946647248c967cf4badf8b40 P ce1496a47d084365ad786cd09e472a2c [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:14:01.928658 28003 raft_consensus.cc:2932] T 49b145f6946647248c967cf4badf8b40 P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:14:01.930545 31026 raft_consensus.cc:2932] T 49b145f6946647248c967cf4badf8b40 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:14:01.936475 27854 catalog_manager.cc:5484] T 49b145f6946647248c967cf4badf8b40 P ce1496a47d084365ad786cd09e472a2c reported cstate change: config changed from index 2 to 4, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) evicted. New cstate: current_term: 2 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:14:01.949107 28373 tablet_service.cc:1514] Processing DeleteTablet for tablet 49b145f6946647248c967cf4badf8b40 with delete_type TABLET_DATA_TOMBSTONED (TS d09a11a03989421b9f28019dc1c14cc2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34786
I20240503 08:14:01.949654 8166 tablet_replica.cc:330] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2: stopping tablet replica
I20240503 08:14:01.949939 8166 raft_consensus.cc:2229] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:01.950085 8166 raft_consensus.cc:2258] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:01.950630 8166 ts_tablet_manager.cc:1875] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:01.958617 8166 ts_tablet_manager.cc:1888] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240503 08:14:01.958792 8166 log.cc:1196] T 49b145f6946647248c967cf4badf8b40 P d09a11a03989421b9f28019dc1c14cc2: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/wals/49b145f6946647248c967cf4badf8b40
I20240503 08:14:01.959848 27849 catalog_manager.cc:4830] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): tablet 49b145f6946647248c967cf4badf8b40 (table rebalancer_extra_table_66 [id=80792788b9164bd4af02769ad074bd3a]) successfully deleted
I20240503 08:14:02.232612 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:02.241962 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:02.268869 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:02.301358 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:02.324378 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:14:02.366493 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:02.367609 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:14:02.392740 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:02.518359 27999 consensus_queue.cc:237] T dcc026d6d9bf40059fd6298c40b04c1a P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:14:02.534014 28391 raft_consensus.cc:1261] T dcc026d6d9bf40059fd6298c40b04c1a P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:02.538197 31027 raft_consensus.cc:1261] T dcc026d6d9bf40059fd6298c40b04c1a P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:02.538146 8153 consensus_queue.cc:1035] T dcc026d6d9bf40059fd6298c40b04c1a P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:02.538697 7778 consensus_queue.cc:1035] T dcc026d6d9bf40059fd6298c40b04c1a P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:02.541530 7778 raft_consensus.cc:2932] T dcc026d6d9bf40059fd6298c40b04c1a P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:14:02.544358 31027 raft_consensus.cc:2932] T dcc026d6d9bf40059fd6298c40b04c1a P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:14:02.546478 28391 raft_consensus.cc:2932] T dcc026d6d9bf40059fd6298c40b04c1a P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:14:02.547161 27856 catalog_manager.cc:5484] T dcc026d6d9bf40059fd6298c40b04c1a P 833582ff68c94269be364ebd187a75db reported cstate change: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:14:02.564155 31141 tablet_service.cc:1514] Processing DeleteTablet for tablet dcc026d6d9bf40059fd6298c40b04c1a with delete_type TABLET_DATA_TOMBSTONED (TS b00f244c69df401997ea99d492218a98 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45814
I20240503 08:14:02.576407 8250 tablet_replica.cc:330] T dcc026d6d9bf40059fd6298c40b04c1a P b00f244c69df401997ea99d492218a98: stopping tablet replica
I20240503 08:14:02.576725 8250 raft_consensus.cc:2229] T dcc026d6d9bf40059fd6298c40b04c1a P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:02.576912 8250 raft_consensus.cc:2258] T dcc026d6d9bf40059fd6298c40b04c1a P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:02.577507 8250 ts_tablet_manager.cc:1875] T dcc026d6d9bf40059fd6298c40b04c1a P b00f244c69df401997ea99d492218a98: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:02.583318 8250 ts_tablet_manager.cc:1888] T dcc026d6d9bf40059fd6298c40b04c1a P b00f244c69df401997ea99d492218a98: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:14:02.583428 8250 log.cc:1196] T dcc026d6d9bf40059fd6298c40b04c1a P b00f244c69df401997ea99d492218a98: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal/wals/dcc026d6d9bf40059fd6298c40b04c1a
I20240503 08:14:02.584043 27846 catalog_manager.cc:4830] TS b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): tablet dcc026d6d9bf40059fd6298c40b04c1a (table rebalancer_extra_table_28 [id=49b49624ca8542a3a7429b4b6e32095d]) successfully deleted
I20240503 08:14:02.884524 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:02.885488 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:02.890352 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:02.909025 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:02.924890 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:14:02.969870 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:02.987352 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:03.034296 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:14:03.152225 30761 consensus_queue.cc:237] T e3e58f895edb4095ba3d77653c9197af P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:14:03.154258 28003 raft_consensus.cc:1261] T e3e58f895edb4095ba3d77653c9197af P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:03.154270 30894 raft_consensus.cc:1261] T e3e58f895edb4095ba3d77653c9197af P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:03.154863 8134 consensus_queue.cc:1035] T e3e58f895edb4095ba3d77653c9197af P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:03.155328 8134 consensus_queue.cc:1035] T e3e58f895edb4095ba3d77653c9197af P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:03.158255 8134 raft_consensus.cc:2932] T e3e58f895edb4095ba3d77653c9197af P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:14:03.161456 30895 raft_consensus.cc:2932] T e3e58f895edb4095ba3d77653c9197af P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:14:03.169097 28003 raft_consensus.cc:2932] T e3e58f895edb4095ba3d77653c9197af P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:14:03.172564 27858 catalog_manager.cc:5484] T e3e58f895edb4095ba3d77653c9197af P ce1496a47d084365ad786cd09e472a2c reported cstate change: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:14:03.188259 31141 tablet_service.cc:1514] Processing DeleteTablet for tablet e3e58f895edb4095ba3d77653c9197af with delete_type TABLET_DATA_TOMBSTONED (TS b00f244c69df401997ea99d492218a98 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45814
I20240503 08:14:03.188911 8349 tablet_replica.cc:330] T e3e58f895edb4095ba3d77653c9197af P b00f244c69df401997ea99d492218a98: stopping tablet replica
I20240503 08:14:03.189170 8349 raft_consensus.cc:2229] T e3e58f895edb4095ba3d77653c9197af P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:03.189481 8349 raft_consensus.cc:2258] T e3e58f895edb4095ba3d77653c9197af P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:03.190307 8349 ts_tablet_manager.cc:1875] T e3e58f895edb4095ba3d77653c9197af P b00f244c69df401997ea99d492218a98: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:03.196255 8349 ts_tablet_manager.cc:1888] T e3e58f895edb4095ba3d77653c9197af P b00f244c69df401997ea99d492218a98: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:14:03.196437 8349 log.cc:1196] T e3e58f895edb4095ba3d77653c9197af P b00f244c69df401997ea99d492218a98: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal/wals/e3e58f895edb4095ba3d77653c9197af
I20240503 08:14:03.197177 27846 catalog_manager.cc:4830] TS b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): tablet e3e58f895edb4095ba3d77653c9197af (table rebalancer_extra_table_24 [id=3cd9bb4078874a2e8da7dea4f43db999]) successfully deleted
I20240503 08:14:03.345310 31160 consensus_queue.cc:237] T f463dfe26ee94b5f9447c6dbdf8759c3 P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
W20240503 08:14:03.348337 31095 consensus_peers.cc:485] T f463dfe26ee94b5f9447c6dbdf8759c3 P b00f244c69df401997ea99d492218a98 -> Peer ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): Couldn't send request to peer ced21ca983b749d0b4d4f6a0e2426bf8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f463dfe26ee94b5f9447c6dbdf8759c3. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:03.348868 28390 raft_consensus.cc:1261] T f463dfe26ee94b5f9447c6dbdf8759c3 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:03.348938 28133 raft_consensus.cc:1261] T f463dfe26ee94b5f9447c6dbdf8759c3 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:03.349313 7221 consensus_queue.cc:1035] T f463dfe26ee94b5f9447c6dbdf8759c3 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:03.349619 8034 consensus_queue.cc:1035] T f463dfe26ee94b5f9447c6dbdf8759c3 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:03.356602 7221 raft_consensus.cc:2932] T f463dfe26ee94b5f9447c6dbdf8759c3 P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:14:03.357151 28390 raft_consensus.cc:2932] T f463dfe26ee94b5f9447c6dbdf8759c3 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:14:03.363852 27853 catalog_manager.cc:5484] T f463dfe26ee94b5f9447c6dbdf8759c3 P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: config changed from index -1 to 2, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) added. New cstate: current_term: 1 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:14:03.367254 28133 raft_consensus.cc:2932] T f463dfe26ee94b5f9447c6dbdf8759c3 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:14:03.470512 31160 consensus_queue.cc:237] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:03.472589 31026 raft_consensus.cc:1261] T 486e9843fd784148b9b49dfe27996dff P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:14:03.473107 31095 consensus_peers.cc:485] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 486e9843fd784148b9b49dfe27996dff. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:03.473160 30761 raft_consensus.cc:1261] T 486e9843fd784148b9b49dfe27996dff P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:03.473572 8388 consensus_queue.cc:1035] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:03.474133 8034 consensus_queue.cc:1035] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:03.476287 8034 raft_consensus.cc:2932] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:03.476814 30761 raft_consensus.cc:2932] T 486e9843fd784148b9b49dfe27996dff P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:03.482077 31027 raft_consensus.cc:2932] T 486e9843fd784148b9b49dfe27996dff P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:03.483371 27853 catalog_manager.cc:5484] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98 reported cstate change: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New cstate: current_term: 1 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:03.545164 28391 consensus_queue.cc:237] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:14:03.558897 28263 raft_consensus.cc:1261] T 775b8190036b4b828431fca19ce23ea9 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:03.559406 7652 consensus_queue.cc:1035] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:14:03.559686 28326 consensus_peers.cc:485] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2 -> Peer ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): Couldn't send request to peer ce1496a47d084365ad786cd09e472a2c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 775b8190036b4b828431fca19ce23ea9. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:03.560135 28133 raft_consensus.cc:1261] T 775b8190036b4b828431fca19ce23ea9 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:03.562608 7479 consensus_queue.cc:1035] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:03.570822 7479 raft_consensus.cc:2932] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:14:03.574633 28263 raft_consensus.cc:2932] T 775b8190036b4b828431fca19ce23ea9 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:14:03.577798 28133 raft_consensus.cc:2932] T 775b8190036b4b828431fca19ce23ea9 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:14:03.579792 27854 catalog_manager.cc:5484] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: config changed from index -1 to 4, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:14:03.621515 30895 consensus_queue.cc:237] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:03.624078 30761 raft_consensus.cc:1261] T f83a4778ffe640ee907e0a73e58abb73 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:03.624255 31027 raft_consensus.cc:1261] T f83a4778ffe640ee907e0a73e58abb73 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:14:03.624384 30828 consensus_peers.cc:485] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f83a4778ffe640ee907e0a73e58abb73. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:03.624547 8442 consensus_queue.cc:1035] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:03.624840 8322 consensus_queue.cc:1035] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:03.629139 8322 raft_consensus.cc:2932] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:03.629622 31027 raft_consensus.cc:2932] T f83a4778ffe640ee907e0a73e58abb73 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:03.632867 30761 raft_consensus.cc:2932] T f83a4778ffe640ee907e0a73e58abb73 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:03.637260 27853 catalog_manager.cc:5484] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c reported cstate change: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New cstate: current_term: 1 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:03.709347 28263 consensus_queue.cc:237] T 978bf5783b0e47e3bfb30441adc600ae P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:03.711656 28133 raft_consensus.cc:1261] T 978bf5783b0e47e3bfb30441adc600ae P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:03.712234 7308 consensus_queue.cc:1035] T 978bf5783b0e47e3bfb30441adc600ae P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:03.712576 28391 raft_consensus.cc:1261] T 978bf5783b0e47e3bfb30441adc600ae P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:14:03.713402 28197 consensus_peers.cc:485] T 978bf5783b0e47e3bfb30441adc600ae P aae860d7ae2b4002bf7794c278f9cc56 -> Peer ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): Couldn't send request to peer ced21ca983b749d0b4d4f6a0e2426bf8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 978bf5783b0e47e3bfb30441adc600ae. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:03.714672 7308 consensus_queue.cc:1035] T 978bf5783b0e47e3bfb30441adc600ae P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:03.736387 7308 raft_consensus.cc:2932] T 978bf5783b0e47e3bfb30441adc600ae P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:14:03.740665 28391 raft_consensus.cc:2932] T 978bf5783b0e47e3bfb30441adc600ae P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:14:03.741312 28133 raft_consensus.cc:2932] T 978bf5783b0e47e3bfb30441adc600ae P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:14:03.746469 27861 catalog_manager.cc:5484] T 978bf5783b0e47e3bfb30441adc600ae P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: config changed from index -1 to 4, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) added. New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:14:03.806038 28263 consensus_queue.cc:237] T d9098f3d36b24d578a3f8d7d3ccad052 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:14:03.808276 28133 raft_consensus.cc:1261] T d9098f3d36b24d578a3f8d7d3ccad052 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:03.808295 28003 raft_consensus.cc:1261] T d9098f3d36b24d578a3f8d7d3ccad052 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:14:03.808571 28196 consensus_peers.cc:485] T d9098f3d36b24d578a3f8d7d3ccad052 P aae860d7ae2b4002bf7794c278f9cc56 -> Peer b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): Couldn't send request to peer b00f244c69df401997ea99d492218a98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d9098f3d36b24d578a3f8d7d3ccad052. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:03.808872 8462 consensus_queue.cc:1035] T d9098f3d36b24d578a3f8d7d3ccad052 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:03.809156 7308 consensus_queue.cc:1035] T d9098f3d36b24d578a3f8d7d3ccad052 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:03.813957 8462 raft_consensus.cc:2932] T d9098f3d36b24d578a3f8d7d3ccad052 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:14:03.816802 28003 raft_consensus.cc:2932] T d9098f3d36b24d578a3f8d7d3ccad052 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:14:03.822538 28133 raft_consensus.cc:2932] T d9098f3d36b24d578a3f8d7d3ccad052 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:14:03.825736 27858 catalog_manager.cc:5484] T d9098f3d36b24d578a3f8d7d3ccad052 P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: config changed from index -1 to 4, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) added. New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:14:03.845232 8491 ts_tablet_manager.cc:902] T f463dfe26ee94b5f9447c6dbdf8759c3 P ced21ca983b749d0b4d4f6a0e2426bf8: Initiating tablet copy from peer b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:14:03.850564 8491 tablet_copy_client.cc:308] T f463dfe26ee94b5f9447c6dbdf8759c3 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.72:40893
I20240503 08:14:03.851347 31181 tablet_copy_service.cc:140] P b00f244c69df401997ea99d492218a98: Received BeginTabletCopySession request for tablet f463dfe26ee94b5f9447c6dbdf8759c3 from peer ced21ca983b749d0b4d4f6a0e2426bf8 ({username='slave'} at 127.20.230.71:59573)
I20240503 08:14:03.851508 31181 tablet_copy_service.cc:161] P b00f244c69df401997ea99d492218a98: Beginning new tablet copy session on tablet f463dfe26ee94b5f9447c6dbdf8759c3 from peer ced21ca983b749d0b4d4f6a0e2426bf8 at {username='slave'} at 127.20.230.71:59573: session id = ced21ca983b749d0b4d4f6a0e2426bf8-f463dfe26ee94b5f9447c6dbdf8759c3
I20240503 08:14:03.854488 31181 tablet_copy_source_session.cc:205] T f463dfe26ee94b5f9447c6dbdf8759c3 P b00f244c69df401997ea99d492218a98: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:03.857494 8491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f463dfe26ee94b5f9447c6dbdf8759c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:03.863348 8491 tablet_copy_client.cc:783] T f463dfe26ee94b5f9447c6dbdf8759c3 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:03.863576 8491 tablet_copy_client.cc:647] T f463dfe26ee94b5f9447c6dbdf8759c3 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:03.866120 8491 tablet_copy_client.cc:523] T f463dfe26ee94b5f9447c6dbdf8759c3 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:03.869078 8491 tablet_bootstrap.cc:492] T f463dfe26ee94b5f9447c6dbdf8759c3 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:14:03.874575 8491 tablet_bootstrap.cc:492] T f463dfe26ee94b5f9447c6dbdf8759c3 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:03.874933 8491 tablet_bootstrap.cc:492] T f463dfe26ee94b5f9447c6dbdf8759c3 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap complete.
I20240503 08:14:03.875164 8491 ts_tablet_manager.cc:1367] T f463dfe26ee94b5f9447c6dbdf8759c3 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.004s
I20240503 08:14:03.875757 8491 raft_consensus.cc:348] T f463dfe26ee94b5f9447c6dbdf8759c3 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:03.875952 8491 raft_consensus.cc:729] T f463dfe26ee94b5f9447c6dbdf8759c3 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:14:03.876173 8491 consensus_queue.cc:260] T f463dfe26ee94b5f9447c6dbdf8759c3 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:03.876667 8491 ts_tablet_manager.cc:1398] T f463dfe26ee94b5f9447c6dbdf8759c3 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 08:14:03.879930 31181 tablet_copy_service.cc:342] P b00f244c69df401997ea99d492218a98: Request end of tablet copy session ced21ca983b749d0b4d4f6a0e2426bf8-f463dfe26ee94b5f9447c6dbdf8759c3 received from {username='slave'} at 127.20.230.71:59573
I20240503 08:14:03.880066 31181 tablet_copy_service.cc:433] P b00f244c69df401997ea99d492218a98: ending tablet copy session ced21ca983b749d0b4d4f6a0e2426bf8-f463dfe26ee94b5f9447c6dbdf8759c3 on tablet f463dfe26ee94b5f9447c6dbdf8759c3 with peer ced21ca983b749d0b4d4f6a0e2426bf8
I20240503 08:14:03.893990 31027 consensus_queue.cc:237] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
W20240503 08:14:03.896569 30960 consensus_peers.cc:485] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 393abed63f334ac4b71986573c86c4d4. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:03.896876 31160 raft_consensus.cc:1261] T 393abed63f334ac4b71986573c86c4d4 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer ced21ca983b749d0b4d4f6a0e2426bf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:03.897032 30761 raft_consensus.cc:1261] T 393abed63f334ac4b71986573c86c4d4 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer ced21ca983b749d0b4d4f6a0e2426bf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:03.897599 8508 consensus_queue.cc:1035] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:03.898032 8137 consensus_queue.cc:1035] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:03.901288 8137 raft_consensus.cc:2932] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:03.904547 30761 raft_consensus.cc:2932] T 393abed63f334ac4b71986573c86c4d4 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:03.911432 27854 catalog_manager.cc:5484] T 393abed63f334ac4b71986573c86c4d4 P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New cstate: current_term: 1 leader_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:03.912140 31160 raft_consensus.cc:2932] T 393abed63f334ac4b71986573c86c4d4 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:03.930269 8519 ts_tablet_manager.cc:902] T 486e9843fd784148b9b49dfe27996dff P aae860d7ae2b4002bf7794c278f9cc56: Initiating tablet copy from peer b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:14:03.945983 8519 tablet_copy_client.cc:308] T 486e9843fd784148b9b49dfe27996dff P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.72:40893
I20240503 08:14:03.946746 31181 tablet_copy_service.cc:140] P b00f244c69df401997ea99d492218a98: Received BeginTabletCopySession request for tablet 486e9843fd784148b9b49dfe27996dff from peer aae860d7ae2b4002bf7794c278f9cc56 ({username='slave'} at 127.20.230.67:45737)
I20240503 08:14:03.946910 31181 tablet_copy_service.cc:161] P b00f244c69df401997ea99d492218a98: Beginning new tablet copy session on tablet 486e9843fd784148b9b49dfe27996dff from peer aae860d7ae2b4002bf7794c278f9cc56 at {username='slave'} at 127.20.230.67:45737: session id = aae860d7ae2b4002bf7794c278f9cc56-486e9843fd784148b9b49dfe27996dff
I20240503 08:14:03.949136 31181 tablet_copy_source_session.cc:205] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:03.950196 8519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 486e9843fd784148b9b49dfe27996dff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:03.955914 8519 tablet_copy_client.cc:783] T 486e9843fd784148b9b49dfe27996dff P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:03.956189 8519 tablet_copy_client.cc:647] T 486e9843fd784148b9b49dfe27996dff P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:03.957547 8519 tablet_copy_client.cc:523] T 486e9843fd784148b9b49dfe27996dff P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:03.960467 8519 tablet_bootstrap.cc:492] T 486e9843fd784148b9b49dfe27996dff P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:14:03.967320 8519 tablet_bootstrap.cc:492] T 486e9843fd784148b9b49dfe27996dff P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:03.967717 8519 tablet_bootstrap.cc:492] T 486e9843fd784148b9b49dfe27996dff P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap complete.
I20240503 08:14:03.967952 8519 ts_tablet_manager.cc:1367] T 486e9843fd784148b9b49dfe27996dff P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.005s
I20240503 08:14:03.968618 8519 raft_consensus.cc:348] T 486e9843fd784148b9b49dfe27996dff P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:03.968884 8519 raft_consensus.cc:729] T 486e9843fd784148b9b49dfe27996dff P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:14:03.969115 8519 consensus_queue.cc:260] T 486e9843fd784148b9b49dfe27996dff P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:03.969589 8519 ts_tablet_manager.cc:1398] T 486e9843fd784148b9b49dfe27996dff P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:14:03.970266 31181 tablet_copy_service.cc:342] P b00f244c69df401997ea99d492218a98: Request end of tablet copy session aae860d7ae2b4002bf7794c278f9cc56-486e9843fd784148b9b49dfe27996dff received from {username='slave'} at 127.20.230.67:45737
I20240503 08:14:03.970393 31181 tablet_copy_service.cc:433] P b00f244c69df401997ea99d492218a98: ending tablet copy session aae860d7ae2b4002bf7794c278f9cc56-486e9843fd784148b9b49dfe27996dff on tablet 486e9843fd784148b9b49dfe27996dff with peer aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:03.989281 31160 consensus_queue.cc:237] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:14:03.994895 30895 raft_consensus.cc:1261] T 620b0cdb874d4efe8d49c971a48ee924 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:03.995386 28262 raft_consensus.cc:1261] T 620b0cdb874d4efe8d49c971a48ee924 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:14:03.995816 31097 consensus_peers.cc:485] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98 -> Peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): Couldn't send request to peer d09a11a03989421b9f28019dc1c14cc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 620b0cdb874d4efe8d49c971a48ee924. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:03.996301 7221 consensus_queue.cc:1035] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:03.996594 8034 consensus_queue.cc:1035] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:03.999975 8034 raft_consensus.cc:2932] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:14:04.000450 28263 raft_consensus.cc:2932] T 620b0cdb874d4efe8d49c971a48ee924 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:14:04.007681 27854 catalog_manager.cc:5484] T 620b0cdb874d4efe8d49c971a48ee924 P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New cstate: current_term: 1 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:14:04.011113 30895 raft_consensus.cc:2932] T 620b0cdb874d4efe8d49c971a48ee924 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:14:04.033195 8543 ts_tablet_manager.cc:902] T 775b8190036b4b828431fca19ce23ea9 P ce1496a47d084365ad786cd09e472a2c: Initiating tablet copy from peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:14:04.046110 8543 tablet_copy_client.cc:308] T 775b8190036b4b828431fca19ce23ea9 P ce1496a47d084365ad786cd09e472a2c: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.68:46211
I20240503 08:14:04.046975 28413 tablet_copy_service.cc:140] P d09a11a03989421b9f28019dc1c14cc2: Received BeginTabletCopySession request for tablet 775b8190036b4b828431fca19ce23ea9 from peer ce1496a47d084365ad786cd09e472a2c ({username='slave'} at 127.20.230.70:51715)
I20240503 08:14:04.047133 28413 tablet_copy_service.cc:161] P d09a11a03989421b9f28019dc1c14cc2: Beginning new tablet copy session on tablet 775b8190036b4b828431fca19ce23ea9 from peer ce1496a47d084365ad786cd09e472a2c at {username='slave'} at 127.20.230.70:51715: session id = ce1496a47d084365ad786cd09e472a2c-775b8190036b4b828431fca19ce23ea9
I20240503 08:14:04.049526 28413 tablet_copy_source_session.cc:205] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:04.050938 8543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 775b8190036b4b828431fca19ce23ea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:04.056273 8543 tablet_copy_client.cc:783] T 775b8190036b4b828431fca19ce23ea9 P ce1496a47d084365ad786cd09e472a2c: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:04.056460 8543 tablet_copy_client.cc:647] T 775b8190036b4b828431fca19ce23ea9 P ce1496a47d084365ad786cd09e472a2c: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:04.058625 8543 tablet_copy_client.cc:523] T 775b8190036b4b828431fca19ce23ea9 P ce1496a47d084365ad786cd09e472a2c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:04.061357 8543 tablet_bootstrap.cc:492] T 775b8190036b4b828431fca19ce23ea9 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:14:04.081840 28263 consensus_queue.cc:237] T f46393257ce9471891698c36a07546ad P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:04.084123 28391 raft_consensus.cc:1261] T f46393257ce9471891698c36a07546ad P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:04.085427 7937 consensus_queue.cc:1035] T f46393257ce9471891698c36a07546ad P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:14:04.087250 28197 consensus_peers.cc:485] T f46393257ce9471891698c36a07546ad P aae860d7ae2b4002bf7794c278f9cc56 -> Peer ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): Couldn't send request to peer ced21ca983b749d0b4d4f6a0e2426bf8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f46393257ce9471891698c36a07546ad. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:04.088438 28131 raft_consensus.cc:1261] T f46393257ce9471891698c36a07546ad P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:04.089419 7937 consensus_queue.cc:1035] T f46393257ce9471891698c36a07546ad P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:04.092397 28131 raft_consensus.cc:2932] T f46393257ce9471891698c36a07546ad P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:14:04.091903 7937 raft_consensus.cc:2932] T f46393257ce9471891698c36a07546ad P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:14:04.098176 8543 tablet_bootstrap.cc:492] T 775b8190036b4b828431fca19ce23ea9 P ce1496a47d084365ad786cd09e472a2c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:04.098827 8543 tablet_bootstrap.cc:492] T 775b8190036b4b828431fca19ce23ea9 P ce1496a47d084365ad786cd09e472a2c: Bootstrap complete.
I20240503 08:14:04.099182 8543 ts_tablet_manager.cc:1367] T 775b8190036b4b828431fca19ce23ea9 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.038s user 0.035s sys 0.004s
I20240503 08:14:04.100239 8543 raft_consensus.cc:348] T 775b8190036b4b828431fca19ce23ea9 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:14:04.100571 8543 raft_consensus.cc:729] T 775b8190036b4b828431fca19ce23ea9 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:14:04.100896 8543 consensus_queue.cc:260] T 775b8190036b4b828431fca19ce23ea9 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:14:04.101696 8543 ts_tablet_manager.cc:1398] T 775b8190036b4b828431fca19ce23ea9 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:14:04.103037 28413 tablet_copy_service.cc:342] P d09a11a03989421b9f28019dc1c14cc2: Request end of tablet copy session ce1496a47d084365ad786cd09e472a2c-775b8190036b4b828431fca19ce23ea9 received from {username='slave'} at 127.20.230.70:51715
I20240503 08:14:04.103173 28413 tablet_copy_service.cc:433] P d09a11a03989421b9f28019dc1c14cc2: ending tablet copy session ce1496a47d084365ad786cd09e472a2c-775b8190036b4b828431fca19ce23ea9 on tablet 775b8190036b4b828431fca19ce23ea9 with peer ce1496a47d084365ad786cd09e472a2c
I20240503 08:14:04.103278 27858 catalog_manager.cc:5484] T f46393257ce9471891698c36a07546ad P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: config changed from index -1 to 4, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) added. New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:14:04.109269 28391 raft_consensus.cc:2932] T f46393257ce9471891698c36a07546ad P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:14:04.143839 8519 ts_tablet_manager.cc:902] T f83a4778ffe640ee907e0a73e58abb73 P aae860d7ae2b4002bf7794c278f9cc56: Initiating tablet copy from peer ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:14:04.159039 8519 tablet_copy_client.cc:308] T f83a4778ffe640ee907e0a73e58abb73 P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.70:33747
I20240503 08:14:04.159662 30915 tablet_copy_service.cc:140] P ce1496a47d084365ad786cd09e472a2c: Received BeginTabletCopySession request for tablet f83a4778ffe640ee907e0a73e58abb73 from peer aae860d7ae2b4002bf7794c278f9cc56 ({username='slave'} at 127.20.230.67:50141)
I20240503 08:14:04.159807 30915 tablet_copy_service.cc:161] P ce1496a47d084365ad786cd09e472a2c: Beginning new tablet copy session on tablet f83a4778ffe640ee907e0a73e58abb73 from peer aae860d7ae2b4002bf7794c278f9cc56 at {username='slave'} at 127.20.230.67:50141: session id = aae860d7ae2b4002bf7794c278f9cc56-f83a4778ffe640ee907e0a73e58abb73
I20240503 08:14:04.162739 30915 tablet_copy_source_session.cc:205] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:04.165799 8519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f83a4778ffe640ee907e0a73e58abb73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:04.172669 8519 tablet_copy_client.cc:783] T f83a4778ffe640ee907e0a73e58abb73 P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:04.173127 8519 tablet_copy_client.cc:647] T f83a4778ffe640ee907e0a73e58abb73 P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:04.175045 8519 tablet_copy_client.cc:523] T f83a4778ffe640ee907e0a73e58abb73 P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:04.178196 8519 tablet_bootstrap.cc:492] T f83a4778ffe640ee907e0a73e58abb73 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:14:04.183809 8519 tablet_bootstrap.cc:492] T f83a4778ffe640ee907e0a73e58abb73 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:04.184104 8519 tablet_bootstrap.cc:492] T f83a4778ffe640ee907e0a73e58abb73 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap complete.
I20240503 08:14:04.184309 8519 ts_tablet_manager.cc:1367] T f83a4778ffe640ee907e0a73e58abb73 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 08:14:04.184942 8519 raft_consensus.cc:348] T f83a4778ffe640ee907e0a73e58abb73 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:04.185154 8519 raft_consensus.cc:729] T f83a4778ffe640ee907e0a73e58abb73 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:14:04.185431 8519 consensus_queue.cc:260] T f83a4778ffe640ee907e0a73e58abb73 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:04.186192 8519 ts_tablet_manager.cc:1398] T f83a4778ffe640ee907e0a73e58abb73 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 08:14:04.187414 30915 tablet_copy_service.cc:342] P ce1496a47d084365ad786cd09e472a2c: Request end of tablet copy session aae860d7ae2b4002bf7794c278f9cc56-f83a4778ffe640ee907e0a73e58abb73 received from {username='slave'} at 127.20.230.67:50141
I20240503 08:14:04.187543 30915 tablet_copy_service.cc:433] P ce1496a47d084365ad786cd09e472a2c: ending tablet copy session aae860d7ae2b4002bf7794c278f9cc56-f83a4778ffe640ee907e0a73e58abb73 on tablet f83a4778ffe640ee907e0a73e58abb73 with peer aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:04.201591 28131 consensus_queue.cc:237] T 658865ee3f8d424ca5cd35e12e6d7c77 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:04.203902 30761 raft_consensus.cc:1261] T 658865ee3f8d424ca5cd35e12e6d7c77 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:04.203930 31027 raft_consensus.cc:1261] T 658865ee3f8d424ca5cd35e12e6d7c77 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:04.204511 8055 consensus_queue.cc:1035] T 658865ee3f8d424ca5cd35e12e6d7c77 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 08:14:04.204802 28067 consensus_peers.cc:485] T 658865ee3f8d424ca5cd35e12e6d7c77 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 658865ee3f8d424ca5cd35e12e6d7c77. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:04.205070 7211 consensus_queue.cc:1035] T 658865ee3f8d424ca5cd35e12e6d7c77 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:04.213912 8055 raft_consensus.cc:2932] T 658865ee3f8d424ca5cd35e12e6d7c77 P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:04.220974 27858 catalog_manager.cc:5484] T 658865ee3f8d424ca5cd35e12e6d7c77 P 1b411c6bb25549368041773d7b710d48 reported cstate change: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New cstate: current_term: 1 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:04.225203 30761 raft_consensus.cc:2932] T 658865ee3f8d424ca5cd35e12e6d7c77 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:04.228914 31027 raft_consensus.cc:2932] T 658865ee3f8d424ca5cd35e12e6d7c77 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:04.282749 28262 consensus_queue.cc:237] T 8c714f29abb140379b57e08eb927e2b7 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
W20240503 08:14:04.285162 28198 consensus_peers.cc:485] T 8c714f29abb140379b57e08eb927e2b7 P aae860d7ae2b4002bf7794c278f9cc56 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c714f29abb140379b57e08eb927e2b7. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:04.285223 30895 raft_consensus.cc:1261] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:04.285533 31160 raft_consensus.cc:1261] T 8c714f29abb140379b57e08eb927e2b7 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:04.285852 8560 consensus_queue.cc:1035] T 8c714f29abb140379b57e08eb927e2b7 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:04.286216 8462 consensus_queue.cc:1035] T 8c714f29abb140379b57e08eb927e2b7 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:04.288970 8462 raft_consensus.cc:2932] T 8c714f29abb140379b57e08eb927e2b7 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:14:04.289637 31160 raft_consensus.cc:2932] T 8c714f29abb140379b57e08eb927e2b7 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:14:04.293403 30895 raft_consensus.cc:2932] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:14:04.298354 27858 catalog_manager.cc:5484] T 8c714f29abb140379b57e08eb927e2b7 P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:14:04.306864 8491 ts_tablet_manager.cc:902] T 978bf5783b0e47e3bfb30441adc600ae P ced21ca983b749d0b4d4f6a0e2426bf8: Initiating tablet copy from peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:14:04.330757 8491 tablet_copy_client.cc:308] T 978bf5783b0e47e3bfb30441adc600ae P ced21ca983b749d0b4d4f6a0e2426bf8: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.67:39919
I20240503 08:14:04.332674 28283 tablet_copy_service.cc:140] P aae860d7ae2b4002bf7794c278f9cc56: Received BeginTabletCopySession request for tablet 978bf5783b0e47e3bfb30441adc600ae from peer ced21ca983b749d0b4d4f6a0e2426bf8 ({username='slave'} at 127.20.230.71:55281)
I20240503 08:14:04.332859 28283 tablet_copy_service.cc:161] P aae860d7ae2b4002bf7794c278f9cc56: Beginning new tablet copy session on tablet 978bf5783b0e47e3bfb30441adc600ae from peer ced21ca983b749d0b4d4f6a0e2426bf8 at {username='slave'} at 127.20.230.71:55281: session id = ced21ca983b749d0b4d4f6a0e2426bf8-978bf5783b0e47e3bfb30441adc600ae
I20240503 08:14:04.335511 28283 tablet_copy_source_session.cc:205] T 978bf5783b0e47e3bfb30441adc600ae P aae860d7ae2b4002bf7794c278f9cc56: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:04.337311 8491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 978bf5783b0e47e3bfb30441adc600ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:04.337651 8615 ts_tablet_manager.cc:902] T d9098f3d36b24d578a3f8d7d3ccad052 P b00f244c69df401997ea99d492218a98: Initiating tablet copy from peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:14:04.343056 8491 tablet_copy_client.cc:783] T 978bf5783b0e47e3bfb30441adc600ae P ced21ca983b749d0b4d4f6a0e2426bf8: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:04.343334 8491 tablet_copy_client.cc:647] T 978bf5783b0e47e3bfb30441adc600ae P ced21ca983b749d0b4d4f6a0e2426bf8: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:04.345269 8491 tablet_copy_client.cc:523] T 978bf5783b0e47e3bfb30441adc600ae P ced21ca983b749d0b4d4f6a0e2426bf8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:04.348907 8491 tablet_bootstrap.cc:492] T 978bf5783b0e47e3bfb30441adc600ae P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:14:04.354161 8615 tablet_copy_client.cc:308] T d9098f3d36b24d578a3f8d7d3ccad052 P b00f244c69df401997ea99d492218a98: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.67:39919
I20240503 08:14:04.354759 28283 tablet_copy_service.cc:140] P aae860d7ae2b4002bf7794c278f9cc56: Received BeginTabletCopySession request for tablet d9098f3d36b24d578a3f8d7d3ccad052 from peer b00f244c69df401997ea99d492218a98 ({username='slave'} at 127.20.230.72:55725)
I20240503 08:14:04.354893 28283 tablet_copy_service.cc:161] P aae860d7ae2b4002bf7794c278f9cc56: Beginning new tablet copy session on tablet d9098f3d36b24d578a3f8d7d3ccad052 from peer b00f244c69df401997ea99d492218a98 at {username='slave'} at 127.20.230.72:55725: session id = b00f244c69df401997ea99d492218a98-d9098f3d36b24d578a3f8d7d3ccad052
I20240503 08:14:04.357141 28283 tablet_copy_source_session.cc:205] T d9098f3d36b24d578a3f8d7d3ccad052 P aae860d7ae2b4002bf7794c278f9cc56: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:04.358213 8615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9098f3d36b24d578a3f8d7d3ccad052. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:04.363327 8615 tablet_copy_client.cc:783] T d9098f3d36b24d578a3f8d7d3ccad052 P b00f244c69df401997ea99d492218a98: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:04.363652 8615 tablet_copy_client.cc:647] T d9098f3d36b24d578a3f8d7d3ccad052 P b00f244c69df401997ea99d492218a98: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:04.365442 8615 tablet_copy_client.cc:523] T d9098f3d36b24d578a3f8d7d3ccad052 P b00f244c69df401997ea99d492218a98: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:04.368526 8615 tablet_bootstrap.cc:492] T d9098f3d36b24d578a3f8d7d3ccad052 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:14:04.384512 31027 raft_consensus.cc:1203] T f463dfe26ee94b5f9447c6dbdf8759c3 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:14:04.384994 8519 ts_tablet_manager.cc:902] T 393abed63f334ac4b71986573c86c4d4 P aae860d7ae2b4002bf7794c278f9cc56: Initiating tablet copy from peer ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:14:04.400862 8491 tablet_bootstrap.cc:492] T 978bf5783b0e47e3bfb30441adc600ae P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:04.401614 8491 tablet_bootstrap.cc:492] T 978bf5783b0e47e3bfb30441adc600ae P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap complete.
I20240503 08:14:04.401993 8491 ts_tablet_manager.cc:1367] T 978bf5783b0e47e3bfb30441adc600ae P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.053s user 0.042s sys 0.001s
I20240503 08:14:04.403126 8491 raft_consensus.cc:348] T 978bf5783b0e47e3bfb30441adc600ae P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:04.403438 8491 raft_consensus.cc:729] T 978bf5783b0e47e3bfb30441adc600ae P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:14:04.403769 8491 consensus_queue.cc:260] T 978bf5783b0e47e3bfb30441adc600ae P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:04.404562 8491 ts_tablet_manager.cc:1398] T 978bf5783b0e47e3bfb30441adc600ae P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:14:04.405752 28283 tablet_copy_service.cc:342] P aae860d7ae2b4002bf7794c278f9cc56: Request end of tablet copy session ced21ca983b749d0b4d4f6a0e2426bf8-978bf5783b0e47e3bfb30441adc600ae received from {username='slave'} at 127.20.230.71:55281
I20240503 08:14:04.405949 28283 tablet_copy_service.cc:433] P aae860d7ae2b4002bf7794c278f9cc56: ending tablet copy session ced21ca983b749d0b4d4f6a0e2426bf8-978bf5783b0e47e3bfb30441adc600ae on tablet 978bf5783b0e47e3bfb30441adc600ae with peer ced21ca983b749d0b4d4f6a0e2426bf8
I20240503 08:14:04.409554 8519 tablet_copy_client.cc:308] T 393abed63f334ac4b71986573c86c4d4 P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.71:41915
I20240503 08:14:04.410212 31048 tablet_copy_service.cc:140] P ced21ca983b749d0b4d4f6a0e2426bf8: Received BeginTabletCopySession request for tablet 393abed63f334ac4b71986573c86c4d4 from peer aae860d7ae2b4002bf7794c278f9cc56 ({username='slave'} at 127.20.230.67:49979)
I20240503 08:14:04.410355 31048 tablet_copy_service.cc:161] P ced21ca983b749d0b4d4f6a0e2426bf8: Beginning new tablet copy session on tablet 393abed63f334ac4b71986573c86c4d4 from peer aae860d7ae2b4002bf7794c278f9cc56 at {username='slave'} at 127.20.230.67:49979: session id = aae860d7ae2b4002bf7794c278f9cc56-393abed63f334ac4b71986573c86c4d4
I20240503 08:14:04.412776 31048 tablet_copy_source_session.cc:205] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:04.416209 8519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 393abed63f334ac4b71986573c86c4d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:04.424127 8636 ts_tablet_manager.cc:902] T 620b0cdb874d4efe8d49c971a48ee924 P d09a11a03989421b9f28019dc1c14cc2: Initiating tablet copy from peer b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:14:04.427031 8519 tablet_copy_client.cc:783] T 393abed63f334ac4b71986573c86c4d4 P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:04.427296 8519 tablet_copy_client.cc:647] T 393abed63f334ac4b71986573c86c4d4 P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:04.428737 8519 tablet_copy_client.cc:523] T 393abed63f334ac4b71986573c86c4d4 P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:04.433984 8519 tablet_bootstrap.cc:492] T 393abed63f334ac4b71986573c86c4d4 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:14:04.435736 28263 raft_consensus.cc:1203] T 486e9843fd784148b9b49dfe27996dff P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:14:04.450678 31160 consensus_queue.cc:237] T fd6f4b81101b46588c3c7711f6bafcfa P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:14:04.453951 30895 raft_consensus.cc:1261] T fd6f4b81101b46588c3c7711f6bafcfa P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:04.454130 30761 raft_consensus.cc:1261] T fd6f4b81101b46588c3c7711f6bafcfa P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:04.454581 8034 consensus_queue.cc:1035] T fd6f4b81101b46588c3c7711f6bafcfa P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:04.454881 7221 consensus_queue.cc:1035] T fd6f4b81101b46588c3c7711f6bafcfa P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 08:14:04.458802 31097 consensus_peers.cc:485] T fd6f4b81101b46588c3c7711f6bafcfa P b00f244c69df401997ea99d492218a98 -> Peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): Couldn't send request to peer d09a11a03989421b9f28019dc1c14cc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd6f4b81101b46588c3c7711f6bafcfa. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:04.459259 8519 tablet_bootstrap.cc:492] T 393abed63f334ac4b71986573c86c4d4 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:04.459646 8519 tablet_bootstrap.cc:492] T 393abed63f334ac4b71986573c86c4d4 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap complete.
I20240503 08:14:04.459905 8519 ts_tablet_manager.cc:1367] T 393abed63f334ac4b71986573c86c4d4 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.003s
I20240503 08:14:04.460616 8519 raft_consensus.cc:348] T 393abed63f334ac4b71986573c86c4d4 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:04.460880 8519 raft_consensus.cc:729] T 393abed63f334ac4b71986573c86c4d4 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:14:04.461127 8519 consensus_queue.cc:260] T 393abed63f334ac4b71986573c86c4d4 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:04.461802 8519 ts_tablet_manager.cc:1398] T 393abed63f334ac4b71986573c86c4d4 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:14:04.462980 31048 tablet_copy_service.cc:342] P ced21ca983b749d0b4d4f6a0e2426bf8: Request end of tablet copy session aae860d7ae2b4002bf7794c278f9cc56-393abed63f334ac4b71986573c86c4d4 received from {username='slave'} at 127.20.230.67:49979
I20240503 08:14:04.463114 31048 tablet_copy_service.cc:433] P ced21ca983b749d0b4d4f6a0e2426bf8: ending tablet copy session aae860d7ae2b4002bf7794c278f9cc56-393abed63f334ac4b71986573c86c4d4 on tablet 393abed63f334ac4b71986573c86c4d4 with peer aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:04.470690 8615 tablet_bootstrap.cc:492] T d9098f3d36b24d578a3f8d7d3ccad052 P b00f244c69df401997ea99d492218a98: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:04.471181 8615 tablet_bootstrap.cc:492] T d9098f3d36b24d578a3f8d7d3ccad052 P b00f244c69df401997ea99d492218a98: Bootstrap complete.
I20240503 08:14:04.470955 8034 raft_consensus.cc:2932] T fd6f4b81101b46588c3c7711f6bafcfa P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:14:04.471441 8615 ts_tablet_manager.cc:1367] T d9098f3d36b24d578a3f8d7d3ccad052 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.103s user 0.045s sys 0.005s
I20240503 08:14:04.472344 8615 raft_consensus.cc:348] T d9098f3d36b24d578a3f8d7d3ccad052 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:14:04.472591 8615 raft_consensus.cc:729] T d9098f3d36b24d578a3f8d7d3ccad052 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:14:04.474391 8615 consensus_queue.cc:260] T d9098f3d36b24d578a3f8d7d3ccad052 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:14:04.474318 30894 raft_consensus.cc:2932] T fd6f4b81101b46588c3c7711f6bafcfa P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:14:04.475065 8615 ts_tablet_manager.cc:1398] T d9098f3d36b24d578a3f8d7d3ccad052 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:14:04.475984 28283 tablet_copy_service.cc:342] P aae860d7ae2b4002bf7794c278f9cc56: Request end of tablet copy session b00f244c69df401997ea99d492218a98-d9098f3d36b24d578a3f8d7d3ccad052 received from {username='slave'} at 127.20.230.72:55725
I20240503 08:14:04.476094 28283 tablet_copy_service.cc:433] P aae860d7ae2b4002bf7794c278f9cc56: ending tablet copy session b00f244c69df401997ea99d492218a98-d9098f3d36b24d578a3f8d7d3ccad052 on tablet d9098f3d36b24d578a3f8d7d3ccad052 with peer b00f244c69df401997ea99d492218a98
I20240503 08:14:04.482249 27858 catalog_manager.cc:5484] T fd6f4b81101b46588c3c7711f6bafcfa P ce1496a47d084365ad786cd09e472a2c reported cstate change: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New cstate: current_term: 1 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:14:04.487689 30761 raft_consensus.cc:2932] T fd6f4b81101b46588c3c7711f6bafcfa P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:14:04.510478 8636 tablet_copy_client.cc:308] T 620b0cdb874d4efe8d49c971a48ee924 P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.72:40893
I20240503 08:14:04.511163 31181 tablet_copy_service.cc:140] P b00f244c69df401997ea99d492218a98: Received BeginTabletCopySession request for tablet 620b0cdb874d4efe8d49c971a48ee924 from peer d09a11a03989421b9f28019dc1c14cc2 ({username='slave'} at 127.20.230.68:37433)
I20240503 08:14:04.511335 31181 tablet_copy_service.cc:161] P b00f244c69df401997ea99d492218a98: Beginning new tablet copy session on tablet 620b0cdb874d4efe8d49c971a48ee924 from peer d09a11a03989421b9f28019dc1c14cc2 at {username='slave'} at 127.20.230.68:37433: session id = d09a11a03989421b9f28019dc1c14cc2-620b0cdb874d4efe8d49c971a48ee924
I20240503 08:14:04.514117 31181 tablet_copy_source_session.cc:205] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:04.515605 8636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 620b0cdb874d4efe8d49c971a48ee924. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:04.523738 8636 tablet_copy_client.cc:783] T 620b0cdb874d4efe8d49c971a48ee924 P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:04.524333 8636 tablet_copy_client.cc:647] T 620b0cdb874d4efe8d49c971a48ee924 P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:04.526047 8636 tablet_copy_client.cc:523] T 620b0cdb874d4efe8d49c971a48ee924 P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:04.531356 8636 tablet_bootstrap.cc:492] T 620b0cdb874d4efe8d49c971a48ee924 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:14:04.543241 30895 raft_consensus.cc:1203] T 775b8190036b4b828431fca19ce23ea9 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:14:04.558167 8636 tablet_bootstrap.cc:492] T 620b0cdb874d4efe8d49c971a48ee924 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:04.558513 8636 tablet_bootstrap.cc:492] T 620b0cdb874d4efe8d49c971a48ee924 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap complete.
I20240503 08:14:04.558763 8636 ts_tablet_manager.cc:1367] T 620b0cdb874d4efe8d49c971a48ee924 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.019s
I20240503 08:14:04.559415 8636 raft_consensus.cc:348] T 620b0cdb874d4efe8d49c971a48ee924 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:14:04.559625 8636 raft_consensus.cc:729] T 620b0cdb874d4efe8d49c971a48ee924 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:14:04.559842 8636 consensus_queue.cc:260] T 620b0cdb874d4efe8d49c971a48ee924 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:14:04.561740 8636 ts_tablet_manager.cc:1398] T 620b0cdb874d4efe8d49c971a48ee924 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240503 08:14:04.562358 31181 tablet_copy_service.cc:342] P b00f244c69df401997ea99d492218a98: Request end of tablet copy session d09a11a03989421b9f28019dc1c14cc2-620b0cdb874d4efe8d49c971a48ee924 received from {username='slave'} at 127.20.230.68:37433
I20240503 08:14:04.562482 31181 tablet_copy_service.cc:433] P b00f244c69df401997ea99d492218a98: ending tablet copy session d09a11a03989421b9f28019dc1c14cc2-620b0cdb874d4efe8d49c971a48ee924 on tablet 620b0cdb874d4efe8d49c971a48ee924 with peer d09a11a03989421b9f28019dc1c14cc2
I20240503 08:14:04.675387 8491 ts_tablet_manager.cc:902] T f46393257ce9471891698c36a07546ad P ced21ca983b749d0b4d4f6a0e2426bf8: Initiating tablet copy from peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:14:04.696520 31027 raft_consensus.cc:1203] T 978bf5783b0e47e3bfb30441adc600ae P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:14:04.706534 8491 tablet_copy_client.cc:308] T f46393257ce9471891698c36a07546ad P ced21ca983b749d0b4d4f6a0e2426bf8: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.67:39919
I20240503 08:14:04.708089 28283 tablet_copy_service.cc:140] P aae860d7ae2b4002bf7794c278f9cc56: Received BeginTabletCopySession request for tablet f46393257ce9471891698c36a07546ad from peer ced21ca983b749d0b4d4f6a0e2426bf8 ({username='slave'} at 127.20.230.71:55281)
I20240503 08:14:04.708287 28283 tablet_copy_service.cc:161] P aae860d7ae2b4002bf7794c278f9cc56: Beginning new tablet copy session on tablet f46393257ce9471891698c36a07546ad from peer ced21ca983b749d0b4d4f6a0e2426bf8 at {username='slave'} at 127.20.230.71:55281: session id = ced21ca983b749d0b4d4f6a0e2426bf8-f46393257ce9471891698c36a07546ad
I20240503 08:14:04.711390 28283 tablet_copy_source_session.cc:205] T f46393257ce9471891698c36a07546ad P aae860d7ae2b4002bf7794c278f9cc56: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:04.724445 8491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f46393257ce9471891698c36a07546ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:04.733867 8491 tablet_copy_client.cc:783] T f46393257ce9471891698c36a07546ad P ced21ca983b749d0b4d4f6a0e2426bf8: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:04.734277 8491 tablet_copy_client.cc:647] T f46393257ce9471891698c36a07546ad P ced21ca983b749d0b4d4f6a0e2426bf8: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:04.736075 8491 tablet_copy_client.cc:523] T f46393257ce9471891698c36a07546ad P ced21ca983b749d0b4d4f6a0e2426bf8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:04.739487 8491 tablet_bootstrap.cc:492] T f46393257ce9471891698c36a07546ad P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:14:04.759044 31159 raft_consensus.cc:1203] T d9098f3d36b24d578a3f8d7d3ccad052 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:14:04.764745 28262 raft_consensus.cc:1203] T f83a4778ffe640ee907e0a73e58abb73 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:14:04.800030 8519 ts_tablet_manager.cc:902] T 658865ee3f8d424ca5cd35e12e6d7c77 P aae860d7ae2b4002bf7794c278f9cc56: Initiating tablet copy from peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:14:04.825855 8519 tablet_copy_client.cc:308] T 658865ee3f8d424ca5cd35e12e6d7c77 P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.66:37143
I20240503 08:14:04.826498 28153 tablet_copy_service.cc:140] P 1b411c6bb25549368041773d7b710d48: Received BeginTabletCopySession request for tablet 658865ee3f8d424ca5cd35e12e6d7c77 from peer aae860d7ae2b4002bf7794c278f9cc56 ({username='slave'} at 127.20.230.67:59481)
I20240503 08:14:04.826687 28153 tablet_copy_service.cc:161] P 1b411c6bb25549368041773d7b710d48: Beginning new tablet copy session on tablet 658865ee3f8d424ca5cd35e12e6d7c77 from peer aae860d7ae2b4002bf7794c278f9cc56 at {username='slave'} at 127.20.230.67:59481: session id = aae860d7ae2b4002bf7794c278f9cc56-658865ee3f8d424ca5cd35e12e6d7c77
I20240503 08:14:04.829456 28153 tablet_copy_source_session.cc:205] T 658865ee3f8d424ca5cd35e12e6d7c77 P 1b411c6bb25549368041773d7b710d48: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:04.830916 8519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 658865ee3f8d424ca5cd35e12e6d7c77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:04.838121 8519 tablet_copy_client.cc:783] T 658865ee3f8d424ca5cd35e12e6d7c77 P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:04.849650 8491 tablet_bootstrap.cc:492] T f46393257ce9471891698c36a07546ad P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:04.851814 8519 tablet_copy_client.cc:647] T 658865ee3f8d424ca5cd35e12e6d7c77 P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:04.852838 8491 tablet_bootstrap.cc:492] T f46393257ce9471891698c36a07546ad P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap complete.
I20240503 08:14:04.853319 8491 ts_tablet_manager.cc:1367] T f46393257ce9471891698c36a07546ad P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.114s user 0.051s sys 0.004s
I20240503 08:14:04.854918 8491 raft_consensus.cc:348] T f46393257ce9471891698c36a07546ad P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:04.855775 8491 raft_consensus.cc:729] T f46393257ce9471891698c36a07546ad P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:14:04.856029 8491 consensus_queue.cc:260] T f46393257ce9471891698c36a07546ad P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:04.856482 8491 ts_tablet_manager.cc:1398] T f46393257ce9471891698c36a07546ad P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:14:04.861290 28283 tablet_copy_service.cc:342] P aae860d7ae2b4002bf7794c278f9cc56: Request end of tablet copy session ced21ca983b749d0b4d4f6a0e2426bf8-f46393257ce9471891698c36a07546ad received from {username='slave'} at 127.20.230.71:55281
I20240503 08:14:04.861389 28283 tablet_copy_service.cc:433] P aae860d7ae2b4002bf7794c278f9cc56: ending tablet copy session ced21ca983b749d0b4d4f6a0e2426bf8-f46393257ce9471891698c36a07546ad on tablet f46393257ce9471891698c36a07546ad with peer ced21ca983b749d0b4d4f6a0e2426bf8
I20240503 08:14:04.865733 8519 tablet_copy_client.cc:523] T 658865ee3f8d424ca5cd35e12e6d7c77 P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:04.869235 8519 tablet_bootstrap.cc:492] T 658865ee3f8d424ca5cd35e12e6d7c77 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:14:04.872216 28391 raft_consensus.cc:1203] T 620b0cdb874d4efe8d49c971a48ee924 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:14:04.883838 8519 tablet_bootstrap.cc:492] T 658865ee3f8d424ca5cd35e12e6d7c77 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:04.884425 8519 tablet_bootstrap.cc:492] T 658865ee3f8d424ca5cd35e12e6d7c77 P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap complete.
I20240503 08:14:04.887404 8519 ts_tablet_manager.cc:1367] T 658865ee3f8d424ca5cd35e12e6d7c77 P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.001s
I20240503 08:14:04.888031 8519 raft_consensus.cc:348] T 658865ee3f8d424ca5cd35e12e6d7c77 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:04.888207 8519 raft_consensus.cc:729] T 658865ee3f8d424ca5cd35e12e6d7c77 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:14:04.888384 8519 consensus_queue.cc:260] T 658865ee3f8d424ca5cd35e12e6d7c77 P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:04.888911 8519 ts_tablet_manager.cc:1398] T 658865ee3f8d424ca5cd35e12e6d7c77 P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:14:04.890354 28153 tablet_copy_service.cc:342] P 1b411c6bb25549368041773d7b710d48: Request end of tablet copy session aae860d7ae2b4002bf7794c278f9cc56-658865ee3f8d424ca5cd35e12e6d7c77 received from {username='slave'} at 127.20.230.67:59481
I20240503 08:14:04.890496 28153 tablet_copy_service.cc:433] P 1b411c6bb25549368041773d7b710d48: ending tablet copy session aae860d7ae2b4002bf7794c278f9cc56-658865ee3f8d424ca5cd35e12e6d7c77 on tablet 658865ee3f8d424ca5cd35e12e6d7c77 with peer aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:04.901880 28262 raft_consensus.cc:1203] T 393abed63f334ac4b71986573c86c4d4 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:14:04.907159 8705 ts_tablet_manager.cc:902] T 8c714f29abb140379b57e08eb927e2b7 P 1b411c6bb25549368041773d7b710d48: Initiating tablet copy from peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:14:04.949400 8705 tablet_copy_client.cc:308] T 8c714f29abb140379b57e08eb927e2b7 P 1b411c6bb25549368041773d7b710d48: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.67:39919
I20240503 08:14:04.950863 28283 tablet_copy_service.cc:140] P aae860d7ae2b4002bf7794c278f9cc56: Received BeginTabletCopySession request for tablet 8c714f29abb140379b57e08eb927e2b7 from peer 1b411c6bb25549368041773d7b710d48 ({username='slave'} at 127.20.230.66:43521)
I20240503 08:14:04.951025 28283 tablet_copy_service.cc:161] P aae860d7ae2b4002bf7794c278f9cc56: Beginning new tablet copy session on tablet 8c714f29abb140379b57e08eb927e2b7 from peer 1b411c6bb25549368041773d7b710d48 at {username='slave'} at 127.20.230.66:43521: session id = 1b411c6bb25549368041773d7b710d48-8c714f29abb140379b57e08eb927e2b7
I20240503 08:14:04.953871 28283 tablet_copy_source_session.cc:205] T 8c714f29abb140379b57e08eb927e2b7 P aae860d7ae2b4002bf7794c278f9cc56: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:04.955669 8705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c714f29abb140379b57e08eb927e2b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:04.963866 8705 tablet_copy_client.cc:783] T 8c714f29abb140379b57e08eb927e2b7 P 1b411c6bb25549368041773d7b710d48: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:04.964129 8705 tablet_copy_client.cc:647] T 8c714f29abb140379b57e08eb927e2b7 P 1b411c6bb25549368041773d7b710d48: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:04.967579 8705 tablet_copy_client.cc:523] T 8c714f29abb140379b57e08eb927e2b7 P 1b411c6bb25549368041773d7b710d48: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:04.969746 8636 ts_tablet_manager.cc:902] T fd6f4b81101b46588c3c7711f6bafcfa P d09a11a03989421b9f28019dc1c14cc2: Initiating tablet copy from peer b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:14:04.971096 8636 tablet_copy_client.cc:308] T fd6f4b81101b46588c3c7711f6bafcfa P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.72:40893
I20240503 08:14:04.971781 31181 tablet_copy_service.cc:140] P b00f244c69df401997ea99d492218a98: Received BeginTabletCopySession request for tablet fd6f4b81101b46588c3c7711f6bafcfa from peer d09a11a03989421b9f28019dc1c14cc2 ({username='slave'} at 127.20.230.68:37433)
I20240503 08:14:04.971937 31181 tablet_copy_service.cc:161] P b00f244c69df401997ea99d492218a98: Beginning new tablet copy session on tablet fd6f4b81101b46588c3c7711f6bafcfa from peer d09a11a03989421b9f28019dc1c14cc2 at {username='slave'} at 127.20.230.68:37433: session id = d09a11a03989421b9f28019dc1c14cc2-fd6f4b81101b46588c3c7711f6bafcfa
I20240503 08:14:04.974885 31181 tablet_copy_source_session.cc:205] T fd6f4b81101b46588c3c7711f6bafcfa P b00f244c69df401997ea99d492218a98: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:04.977547 8705 tablet_bootstrap.cc:492] T 8c714f29abb140379b57e08eb927e2b7 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:14:04.977566 8636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd6f4b81101b46588c3c7711f6bafcfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:04.993149 8636 tablet_copy_client.cc:783] T fd6f4b81101b46588c3c7711f6bafcfa P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:04.994732 8636 tablet_copy_client.cc:647] T fd6f4b81101b46588c3c7711f6bafcfa P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:04.999742 8636 tablet_copy_client.cc:523] T fd6f4b81101b46588c3c7711f6bafcfa P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:05.001999 8705 tablet_bootstrap.cc:492] T 8c714f29abb140379b57e08eb927e2b7 P 1b411c6bb25549368041773d7b710d48: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:05.002529 8705 tablet_bootstrap.cc:492] T 8c714f29abb140379b57e08eb927e2b7 P 1b411c6bb25549368041773d7b710d48: Bootstrap complete.
I20240503 08:14:05.002791 8705 ts_tablet_manager.cc:1367] T 8c714f29abb140379b57e08eb927e2b7 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.026s user 0.012s sys 0.008s
I20240503 08:14:05.003598 8705 raft_consensus.cc:348] T 8c714f29abb140379b57e08eb927e2b7 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:14:05.003863 8705 raft_consensus.cc:729] T 8c714f29abb140379b57e08eb927e2b7 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:14:05.004141 8705 consensus_queue.cc:260] T 8c714f29abb140379b57e08eb927e2b7 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:14:05.004652 8636 tablet_bootstrap.cc:492] T fd6f4b81101b46588c3c7711f6bafcfa P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:14:05.004684 8705 ts_tablet_manager.cc:1398] T 8c714f29abb140379b57e08eb927e2b7 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:14:05.006683 28283 tablet_copy_service.cc:342] P aae860d7ae2b4002bf7794c278f9cc56: Request end of tablet copy session 1b411c6bb25549368041773d7b710d48-8c714f29abb140379b57e08eb927e2b7 received from {username='slave'} at 127.20.230.66:43521
I20240503 08:14:05.006805 28283 tablet_copy_service.cc:433] P aae860d7ae2b4002bf7794c278f9cc56: ending tablet copy session 1b411c6bb25549368041773d7b710d48-8c714f29abb140379b57e08eb927e2b7 on tablet 8c714f29abb140379b57e08eb927e2b7 with peer 1b411c6bb25549368041773d7b710d48
I20240503 08:14:05.021050 8636 tablet_bootstrap.cc:492] T fd6f4b81101b46588c3c7711f6bafcfa P d09a11a03989421b9f28019dc1c14cc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:05.021550 8636 tablet_bootstrap.cc:492] T fd6f4b81101b46588c3c7711f6bafcfa P d09a11a03989421b9f28019dc1c14cc2: Bootstrap complete.
I20240503 08:14:05.021968 8636 ts_tablet_manager.cc:1367] T fd6f4b81101b46588c3c7711f6bafcfa P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.002s
I20240503 08:14:05.022687 8636 raft_consensus.cc:348] T fd6f4b81101b46588c3c7711f6bafcfa P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:14:05.022967 8636 raft_consensus.cc:729] T fd6f4b81101b46588c3c7711f6bafcfa P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:14:05.023218 8636 consensus_queue.cc:260] T fd6f4b81101b46588c3c7711f6bafcfa P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:14:05.023837 8636 ts_tablet_manager.cc:1398] T fd6f4b81101b46588c3c7711f6bafcfa P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:14:05.024842 31181 tablet_copy_service.cc:342] P b00f244c69df401997ea99d492218a98: Request end of tablet copy session d09a11a03989421b9f28019dc1c14cc2-fd6f4b81101b46588c3c7711f6bafcfa received from {username='slave'} at 127.20.230.68:37433
I20240503 08:14:05.024984 31181 tablet_copy_service.cc:433] P b00f244c69df401997ea99d492218a98: ending tablet copy session d09a11a03989421b9f28019dc1c14cc2-fd6f4b81101b46588c3c7711f6bafcfa on tablet fd6f4b81101b46588c3c7711f6bafcfa with peer d09a11a03989421b9f28019dc1c14cc2
I20240503 08:14:05.035689 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:05.103665 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:05.149171 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:05.170050 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240503 08:14:05.170362 28199 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.68:46211, user_credentials={real_user=slave}} blocked reactor thread for 69593.5us
I20240503 08:14:05.188795 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:14:05.206962 28262 raft_consensus.cc:1203] T 658865ee3f8d424ca5cd35e12e6d7c77 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:14:05.259914 31027 raft_consensus.cc:1203] T f46393257ce9471891698c36a07546ad P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:14:05.295552 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:05.297741 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:05.309146 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:05.318630 28131 raft_consensus.cc:1203] T 8c714f29abb140379b57e08eb927e2b7 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:14:05.570729 28390 raft_consensus.cc:1203] T fd6f4b81101b46588c3c7711f6bafcfa P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:14:05.745646 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:05.770418 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:05.786448 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:05.913719 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:14:05.929279 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:05.946058 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:05.947058 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:05.948246 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:14:06.095523 28133 consensus_queue.cc:237] T 658865ee3f8d424ca5cd35e12e6d7c77 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:06.098731 31027 raft_consensus.cc:1261] T 658865ee3f8d424ca5cd35e12e6d7c77 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:06.099094 28263 raft_consensus.cc:1261] T 658865ee3f8d424ca5cd35e12e6d7c77 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:06.099228 8055 consensus_queue.cc:1035] T 658865ee3f8d424ca5cd35e12e6d7c77 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:06.099620 8742 consensus_queue.cc:1035] T 658865ee3f8d424ca5cd35e12e6d7c77 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:06.101646 8055 raft_consensus.cc:2932] T 658865ee3f8d424ca5cd35e12e6d7c77 P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:06.102675 28263 raft_consensus.cc:2932] T 658865ee3f8d424ca5cd35e12e6d7c77 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:06.104954 31026 raft_consensus.cc:2932] T 658865ee3f8d424ca5cd35e12e6d7c77 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:06.108002 27858 catalog_manager.cc:5484] T 658865ee3f8d424ca5cd35e12e6d7c77 P 1b411c6bb25549368041773d7b710d48 reported cstate change: config changed from index 2 to 3, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New cstate: current_term: 1 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:06.122822 30742 tablet_service.cc:1514] Processing DeleteTablet for tablet 658865ee3f8d424ca5cd35e12e6d7c77 with delete_type TABLET_DATA_TOMBSTONED (TS 6d1d7de19459435fbde1999e3dfd226b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46794
I20240503 08:14:06.134073 8835 tablet_replica.cc:330] T 658865ee3f8d424ca5cd35e12e6d7c77 P 6d1d7de19459435fbde1999e3dfd226b: stopping tablet replica
I20240503 08:14:06.134294 8835 raft_consensus.cc:2229] T 658865ee3f8d424ca5cd35e12e6d7c77 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:06.136234 8835 raft_consensus.cc:2258] T 658865ee3f8d424ca5cd35e12e6d7c77 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:06.140446 8835 ts_tablet_manager.cc:1875] T 658865ee3f8d424ca5cd35e12e6d7c77 P 6d1d7de19459435fbde1999e3dfd226b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:06.151762 8835 ts_tablet_manager.cc:1888] T 658865ee3f8d424ca5cd35e12e6d7c77 P 6d1d7de19459435fbde1999e3dfd226b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:14:06.155707 8835 log.cc:1196] T 658865ee3f8d424ca5cd35e12e6d7c77 P 6d1d7de19459435fbde1999e3dfd226b: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal/wals/658865ee3f8d424ca5cd35e12e6d7c77
I20240503 08:14:06.157243 27848 catalog_manager.cc:4830] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): tablet 658865ee3f8d424ca5cd35e12e6d7c77 (table rebalancer_extra_table_12 [id=7f3bcd2f061e424892943977bcde8c80]) successfully deleted
I20240503 08:14:06.429388 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:06.441295 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:06.448400 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:06.476016 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:06.476630 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:14:06.558840 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:14:06.643224 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:06.693017 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:06.810627 31161 consensus_queue.cc:237] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:14:06.812927 28261 raft_consensus.cc:1261] T 620b0cdb874d4efe8d49c971a48ee924 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:06.813046 28391 raft_consensus.cc:1261] T 620b0cdb874d4efe8d49c971a48ee924 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:06.813464 8537 consensus_queue.cc:1035] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:06.813782 8743 consensus_queue.cc:1035] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:06.816620 8743 raft_consensus.cc:2932] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:14:06.819399 28391 raft_consensus.cc:2932] T 620b0cdb874d4efe8d49c971a48ee924 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:14:06.825796 28262 raft_consensus.cc:2932] T 620b0cdb874d4efe8d49c971a48ee924 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:14:06.831477 27854 catalog_manager.cc:5484] T 620b0cdb874d4efe8d49c971a48ee924 P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: config changed from index 2 to 3, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New cstate: current_term: 1 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:14:06.862221 30874 tablet_service.cc:1514] Processing DeleteTablet for tablet 620b0cdb874d4efe8d49c971a48ee924 with delete_type TABLET_DATA_TOMBSTONED (TS ce1496a47d084365ad786cd09e472a2c not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:34426
I20240503 08:14:06.863009 8919 tablet_replica.cc:330] T 620b0cdb874d4efe8d49c971a48ee924 P ce1496a47d084365ad786cd09e472a2c: stopping tablet replica
I20240503 08:14:06.863255 8919 raft_consensus.cc:2229] T 620b0cdb874d4efe8d49c971a48ee924 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:06.863592 8919 raft_consensus.cc:2258] T 620b0cdb874d4efe8d49c971a48ee924 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:06.864369 8919 ts_tablet_manager.cc:1875] T 620b0cdb874d4efe8d49c971a48ee924 P ce1496a47d084365ad786cd09e472a2c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:06.870504 8919 ts_tablet_manager.cc:1888] T 620b0cdb874d4efe8d49c971a48ee924 P ce1496a47d084365ad786cd09e472a2c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:14:06.870623 8919 log.cc:1196] T 620b0cdb874d4efe8d49c971a48ee924 P ce1496a47d084365ad786cd09e472a2c: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal/wals/620b0cdb874d4efe8d49c971a48ee924
I20240503 08:14:06.871305 27846 catalog_manager.cc:4830] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): tablet 620b0cdb874d4efe8d49c971a48ee924 (table rebalancer_extra_table_48 [id=65fb43133ebb48fa9b6c1fe9b9dd6f27]) successfully deleted
I20240503 08:14:07.121506 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:07.121935 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:07.165729 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:14:07.185671 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:14:07.200160 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:07.240923 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:07.248008 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:07.254407 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:07.412941 31161 consensus_queue.cc:237] T fd6f4b81101b46588c3c7711f6bafcfa P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:14:07.414966 30761 raft_consensus.cc:1261] T fd6f4b81101b46588c3c7711f6bafcfa P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:07.414966 28391 raft_consensus.cc:1261] T fd6f4b81101b46588c3c7711f6bafcfa P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:07.415665 8743 consensus_queue.cc:1035] T fd6f4b81101b46588c3c7711f6bafcfa P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:07.416281 8913 consensus_queue.cc:1035] T fd6f4b81101b46588c3c7711f6bafcfa P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:07.418954 8995 raft_consensus.cc:2932] T fd6f4b81101b46588c3c7711f6bafcfa P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:14:07.419577 28391 raft_consensus.cc:2932] T fd6f4b81101b46588c3c7711f6bafcfa P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:14:07.420225 30761 raft_consensus.cc:2932] T fd6f4b81101b46588c3c7711f6bafcfa P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:14:07.427079 27854 catalog_manager.cc:5484] T fd6f4b81101b46588c3c7711f6bafcfa P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: config changed from index 2 to 3, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New cstate: current_term: 1 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:14:07.437100 30874 tablet_service.cc:1514] Processing DeleteTablet for tablet fd6f4b81101b46588c3c7711f6bafcfa with delete_type TABLET_DATA_TOMBSTONED (TS ce1496a47d084365ad786cd09e472a2c not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:34426
I20240503 08:14:07.437693 9003 tablet_replica.cc:330] T fd6f4b81101b46588c3c7711f6bafcfa P ce1496a47d084365ad786cd09e472a2c: stopping tablet replica
I20240503 08:14:07.439260 9003 raft_consensus.cc:2229] T fd6f4b81101b46588c3c7711f6bafcfa P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:07.439560 9003 raft_consensus.cc:2258] T fd6f4b81101b46588c3c7711f6bafcfa P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:07.441061 9003 ts_tablet_manager.cc:1875] T fd6f4b81101b46588c3c7711f6bafcfa P ce1496a47d084365ad786cd09e472a2c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:07.450286 9003 ts_tablet_manager.cc:1888] T fd6f4b81101b46588c3c7711f6bafcfa P ce1496a47d084365ad786cd09e472a2c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:14:07.450484 9003 log.cc:1196] T fd6f4b81101b46588c3c7711f6bafcfa P ce1496a47d084365ad786cd09e472a2c: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal/wals/fd6f4b81101b46588c3c7711f6bafcfa
I20240503 08:14:07.451316 27846 catalog_manager.cc:4830] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): tablet fd6f4b81101b46588c3c7711f6bafcfa (table rebalancer_extra_table_34 [id=505d74ce858645248ed95157f710daf1]) successfully deleted
I20240503 08:14:07.789392 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:07.860431 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:14:07.868842 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:07.884996 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:07.899691 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:14:07.947836 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:07.969359 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:07.989568 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:08.113641 31026 consensus_queue.cc:237] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:08.115448 28261 raft_consensus.cc:1261] T 393abed63f334ac4b71986573c86c4d4 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer ced21ca983b749d0b4d4f6a0e2426bf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:08.115573 31161 raft_consensus.cc:1261] T 393abed63f334ac4b71986573c86c4d4 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer ced21ca983b749d0b4d4f6a0e2426bf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:08.116081 9064 consensus_queue.cc:1035] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:08.116350 8137 consensus_queue.cc:1035] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:08.123009 9064 raft_consensus.cc:2932] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:08.123895 28263 raft_consensus.cc:2932] T 393abed63f334ac4b71986573c86c4d4 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:08.131904 27855 catalog_manager.cc:5484] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8 reported cstate change: config changed from index 2 to 3, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New cstate: current_term: 1 leader_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:08.123656 31161 raft_consensus.cc:2932] T 393abed63f334ac4b71986573c86c4d4 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:08.142256 30742 tablet_service.cc:1514] Processing DeleteTablet for tablet 393abed63f334ac4b71986573c86c4d4 with delete_type TABLET_DATA_TOMBSTONED (TS 6d1d7de19459435fbde1999e3dfd226b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46794
I20240503 08:14:08.142820 9091 tablet_replica.cc:330] T 393abed63f334ac4b71986573c86c4d4 P 6d1d7de19459435fbde1999e3dfd226b: stopping tablet replica
I20240503 08:14:08.143074 9091 raft_consensus.cc:2229] T 393abed63f334ac4b71986573c86c4d4 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:08.143347 9091 raft_consensus.cc:2258] T 393abed63f334ac4b71986573c86c4d4 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:08.144335 9091 ts_tablet_manager.cc:1875] T 393abed63f334ac4b71986573c86c4d4 P 6d1d7de19459435fbde1999e3dfd226b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:08.151680 9091 ts_tablet_manager.cc:1888] T 393abed63f334ac4b71986573c86c4d4 P 6d1d7de19459435fbde1999e3dfd226b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:14:08.151896 9091 log.cc:1196] T 393abed63f334ac4b71986573c86c4d4 P 6d1d7de19459435fbde1999e3dfd226b: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal/wals/393abed63f334ac4b71986573c86c4d4
I20240503 08:14:08.152765 27848 catalog_manager.cc:4830] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): tablet 393abed63f334ac4b71986573c86c4d4 (table rebalancer_extra_table_8 [id=ae93b0e34a8d4e648e3405fef1bfcbf8]) successfully deleted
I20240503 08:14:08.409591 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:08.450181 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:08.455895 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:08.468183 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:08.510916 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:08.570226 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:14:08.602809 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:14:08.670482 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:08.814256 28263 consensus_queue.cc:237] T d9098f3d36b24d578a3f8d7d3ccad052 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:14:08.816051 31159 raft_consensus.cc:1261] T d9098f3d36b24d578a3f8d7d3ccad052 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:08.816082 28003 raft_consensus.cc:1261] T d9098f3d36b24d578a3f8d7d3ccad052 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:08.816558 9153 consensus_queue.cc:1035] T d9098f3d36b24d578a3f8d7d3ccad052 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:08.818666 9004 consensus_queue.cc:1035] T d9098f3d36b24d578a3f8d7d3ccad052 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:08.820653 9153 raft_consensus.cc:2932] T d9098f3d36b24d578a3f8d7d3ccad052 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:14:08.821192 31159 raft_consensus.cc:2932] T d9098f3d36b24d578a3f8d7d3ccad052 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:14:08.822165 28003 raft_consensus.cc:2932] T d9098f3d36b24d578a3f8d7d3ccad052 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:14:08.827389 27856 catalog_manager.cc:5484] T d9098f3d36b24d578a3f8d7d3ccad052 P b00f244c69df401997ea99d492218a98 reported cstate change: config changed from index 4 to 5, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:14:08.834573 28111 tablet_service.cc:1514] Processing DeleteTablet for tablet d9098f3d36b24d578a3f8d7d3ccad052 with delete_type TABLET_DATA_TOMBSTONED (TS 1b411c6bb25549368041773d7b710d48 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57912
I20240503 08:14:08.835172 9178 tablet_replica.cc:330] T d9098f3d36b24d578a3f8d7d3ccad052 P 1b411c6bb25549368041773d7b710d48: stopping tablet replica
I20240503 08:14:08.835414 9178 raft_consensus.cc:2229] T d9098f3d36b24d578a3f8d7d3ccad052 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:08.835741 9178 raft_consensus.cc:2258] T d9098f3d36b24d578a3f8d7d3ccad052 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:08.836333 9178 ts_tablet_manager.cc:1875] T d9098f3d36b24d578a3f8d7d3ccad052 P 1b411c6bb25549368041773d7b710d48: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:08.841475 9178 ts_tablet_manager.cc:1888] T d9098f3d36b24d578a3f8d7d3ccad052 P 1b411c6bb25549368041773d7b710d48: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:14:08.841584 9178 log.cc:1196] T d9098f3d36b24d578a3f8d7d3ccad052 P 1b411c6bb25549368041773d7b710d48: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/d9098f3d36b24d578a3f8d7d3ccad052
I20240503 08:14:08.842278 27848 catalog_manager.cc:4830] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): tablet d9098f3d36b24d578a3f8d7d3ccad052 (table rebalance_test_table_1 [id=05d29869a30847b6a0c137521965898a]) successfully deleted
I20240503 08:14:09.063879 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:09.064549 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:09.100622 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:09.106003 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:14:09.122476 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:09.153409 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:14:09.164321 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:09.192473 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:09.348332 28263 consensus_queue.cc:237] T 978bf5783b0e47e3bfb30441adc600ae P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:09.354836 28133 raft_consensus.cc:1261] T 978bf5783b0e47e3bfb30441adc600ae P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:09.354898 31026 raft_consensus.cc:1261] T 978bf5783b0e47e3bfb30441adc600ae P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:09.355382 9153 consensus_queue.cc:1035] T 978bf5783b0e47e3bfb30441adc600ae P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:09.355674 9004 consensus_queue.cc:1035] T 978bf5783b0e47e3bfb30441adc600ae P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:09.358556 9004 raft_consensus.cc:2932] T 978bf5783b0e47e3bfb30441adc600ae P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:14:09.361462 28133 raft_consensus.cc:2932] T 978bf5783b0e47e3bfb30441adc600ae P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:14:09.364058 27858 catalog_manager.cc:5484] T 978bf5783b0e47e3bfb30441adc600ae P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: config changed from index 4 to 5, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) evicted. New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:14:09.367894 31026 raft_consensus.cc:2932] T 978bf5783b0e47e3bfb30441adc600ae P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:14:09.384297 28373 tablet_service.cc:1514] Processing DeleteTablet for tablet 978bf5783b0e47e3bfb30441adc600ae with delete_type TABLET_DATA_TOMBSTONED (TS d09a11a03989421b9f28019dc1c14cc2 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34786
I20240503 08:14:09.403934 9261 tablet_replica.cc:330] T 978bf5783b0e47e3bfb30441adc600ae P d09a11a03989421b9f28019dc1c14cc2: stopping tablet replica
I20240503 08:14:09.404292 9261 raft_consensus.cc:2229] T 978bf5783b0e47e3bfb30441adc600ae P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:09.405985 9261 raft_consensus.cc:2258] T 978bf5783b0e47e3bfb30441adc600ae P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:09.410075 9261 ts_tablet_manager.cc:1875] T 978bf5783b0e47e3bfb30441adc600ae P d09a11a03989421b9f28019dc1c14cc2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:09.417728 9261 ts_tablet_manager.cc:1888] T 978bf5783b0e47e3bfb30441adc600ae P d09a11a03989421b9f28019dc1c14cc2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:14:09.417953 9261 log.cc:1196] T 978bf5783b0e47e3bfb30441adc600ae P d09a11a03989421b9f28019dc1c14cc2: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/wals/978bf5783b0e47e3bfb30441adc600ae
I20240503 08:14:09.419138 27849 catalog_manager.cc:4830] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): tablet 978bf5783b0e47e3bfb30441adc600ae (table rebalance_test_table_19 [id=18d795e9c19046fdba033edcea369490]) successfully deleted
I20240503 08:14:09.793498 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:09.794790 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:09.813613 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:09.844769 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:14:09.855818 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:09.869542 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:14:09.875819 28049 maintenance_manager.cc:419] P 833582ff68c94269be364ebd187a75db: Scheduling FlushMRSOp(d9098f3d36b24d578a3f8d7d3ccad052): perf score=0.033334
I20240503 08:14:09.914067 28179 maintenance_manager.cc:419] P 1b411c6bb25549368041773d7b710d48: Scheduling FlushMRSOp(775b8190036b4b828431fca19ce23ea9): perf score=0.033350
I20240503 08:14:09.925051 28309 maintenance_manager.cc:419] P aae860d7ae2b4002bf7794c278f9cc56: Scheduling FlushMRSOp(d94f5824a71f454f8a1bfa13dd032684): perf score=0.033356
I20240503 08:14:09.968638 28439 maintenance_manager.cc:419] P d09a11a03989421b9f28019dc1c14cc2: Scheduling FlushMRSOp(d94f5824a71f454f8a1bfa13dd032684): perf score=0.033368
I20240503 08:14:10.001091 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:10.010387 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:10.179782 28070 maintenance_manager.cc:643] P 1b411c6bb25549368041773d7b710d48: FlushMRSOp(775b8190036b4b828431fca19ce23ea9) complete. Timing: real 0.264s user 0.055s sys 0.029s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":45096,"dirs.queue_time_us":23550,"dirs.run_cpu_time_us":247,"dirs.run_wall_time_us":8218,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1385,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":90189,"threads_started":3,"wal-append.queue_time_us":6467}
I20240503 08:14:10.183699 28179 maintenance_manager.cc:419] P 1b411c6bb25549368041773d7b710d48: Scheduling FlushMRSOp(d94f5824a71f454f8a1bfa13dd032684): perf score=0.033422
I20240503 08:14:10.260181 28070 maintenance_manager.cc:643] P 1b411c6bb25549368041773d7b710d48: FlushMRSOp(d94f5824a71f454f8a1bfa13dd032684) complete. Timing: real 0.076s user 0.046s sys 0.011s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":39,"dirs.run_cpu_time_us":180,"dirs.run_wall_time_us":1470,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":811,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":12800,"threads_started":1,"wal-append.queue_time_us":14258}
I20240503 08:14:10.264389 28179 maintenance_manager.cc:419] P 1b411c6bb25549368041773d7b710d48: Scheduling UndoDeltaBlockGCOp(d94f5824a71f454f8a1bfa13dd032684): 4923 bytes on disk
I20240503 08:14:10.266572 28070 maintenance_manager.cc:643] P 1b411c6bb25549368041773d7b710d48: UndoDeltaBlockGCOp(d94f5824a71f454f8a1bfa13dd032684) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
I20240503 08:14:10.271492 28179 maintenance_manager.cc:419] P 1b411c6bb25549368041773d7b710d48: Scheduling UndoDeltaBlockGCOp(775b8190036b4b828431fca19ce23ea9): 4583 bytes on disk
I20240503 08:14:10.273921 28070 maintenance_manager.cc:643] P 1b411c6bb25549368041773d7b710d48: UndoDeltaBlockGCOp(775b8190036b4b828431fca19ce23ea9) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":169,"lbm_reads_lt_1ms":4}
I20240503 08:14:10.296222 28200 maintenance_manager.cc:643] P aae860d7ae2b4002bf7794c278f9cc56: FlushMRSOp(d94f5824a71f454f8a1bfa13dd032684) complete. Timing: real 0.370s user 0.073s sys 0.020s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":27695,"dirs.queue_time_us":9428,"dirs.run_cpu_time_us":174,"dirs.run_wall_time_us":3778,"drs_written":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4,"lbm_write_time_us":17498,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":60946,"threads_started":3,"wal-append.queue_time_us":13799}
I20240503 08:14:10.306285 28309 maintenance_manager.cc:419] P aae860d7ae2b4002bf7794c278f9cc56: Scheduling FlushMRSOp(d9098f3d36b24d578a3f8d7d3ccad052): perf score=0.033458
I20240503 08:14:10.316642 27940 maintenance_manager.cc:643] P 833582ff68c94269be364ebd187a75db: FlushMRSOp(d9098f3d36b24d578a3f8d7d3ccad052) complete. Timing: real 0.440s user 0.063s sys 0.017s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":42778,"dirs.queue_time_us":12200,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":2411,"drs_written":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4,"lbm_write_time_us":919,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":57897,"threads_started":3,"wal-append.queue_time_us":2070}
I20240503 08:14:10.327378 28049 maintenance_manager.cc:419] P 833582ff68c94269be364ebd187a75db: Scheduling UndoDeltaBlockGCOp(d9098f3d36b24d578a3f8d7d3ccad052): 4838 bytes on disk
I20240503 08:14:10.328362 27940 maintenance_manager.cc:643] P 833582ff68c94269be364ebd187a75db: UndoDeltaBlockGCOp(d9098f3d36b24d578a3f8d7d3ccad052) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240503 08:14:10.429419 28330 maintenance_manager.cc:643] P d09a11a03989421b9f28019dc1c14cc2: FlushMRSOp(d94f5824a71f454f8a1bfa13dd032684) complete. Timing: real 0.459s user 0.053s sys 0.024s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":30408,"dirs.queue_time_us":8524,"dirs.run_cpu_time_us":234,"dirs.run_wall_time_us":2583,"drs_written":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4,"lbm_write_time_us":12711,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":70230,"threads_started":3,"wal-append.queue_time_us":11789}
I20240503 08:14:10.448606 28439 maintenance_manager.cc:419] P d09a11a03989421b9f28019dc1c14cc2: Scheduling FlushMRSOp(775b8190036b4b828431fca19ce23ea9): perf score=0.033496
W20240503 08:14:10.522235 30697 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.65:39081, user_credentials={real_user=slave}} blocked reactor thread for 48736.4us
I20240503 08:14:10.554435 28260 consensus_queue.cc:237] T f46393257ce9471891698c36a07546ad P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:10.564317 31026 raft_consensus.cc:1261] T f46393257ce9471891698c36a07546ad P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:10.564289 28391 raft_consensus.cc:1261] T f46393257ce9471891698c36a07546ad P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:10.564821 9342 consensus_queue.cc:1035] T f46393257ce9471891698c36a07546ad P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:10.565093 9153 consensus_queue.cc:1035] T f46393257ce9471891698c36a07546ad P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:10.618301 9342 raft_consensus.cc:2932] T f46393257ce9471891698c36a07546ad P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:14:10.623679 28389 raft_consensus.cc:2932] T f46393257ce9471891698c36a07546ad P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:14:10.630802 27853 catalog_manager.cc:5484] T f46393257ce9471891698c36a07546ad P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: config changed from index 4 to 5, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:14:10.646239 31027 raft_consensus.cc:2932] T f46393257ce9471891698c36a07546ad P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:14:10.668036 28200 maintenance_manager.cc:643] P aae860d7ae2b4002bf7794c278f9cc56: FlushMRSOp(d9098f3d36b24d578a3f8d7d3ccad052) complete. Timing: real 0.361s user 0.040s sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":7684,"dirs.run_cpu_time_us":209,"dirs.run_wall_time_us":1455,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":813,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":38128,"threads_started":1,"wal-append.queue_time_us":34588}
I20240503 08:14:10.671414 28309 maintenance_manager.cc:419] P aae860d7ae2b4002bf7794c278f9cc56: Scheduling FlushMRSOp(775b8190036b4b828431fca19ce23ea9): perf score=0.033556
I20240503 08:14:10.725935 28111 tablet_service.cc:1514] Processing DeleteTablet for tablet f46393257ce9471891698c36a07546ad with delete_type TABLET_DATA_TOMBSTONED (TS 1b411c6bb25549368041773d7b710d48 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57912
I20240503 08:14:10.731793 9370 tablet_replica.cc:330] T f46393257ce9471891698c36a07546ad P 1b411c6bb25549368041773d7b710d48: stopping tablet replica
I20240503 08:14:10.732036 9370 raft_consensus.cc:2229] T f46393257ce9471891698c36a07546ad P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:10.732277 9370 raft_consensus.cc:2258] T f46393257ce9471891698c36a07546ad P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:10.736629 9370 ts_tablet_manager.cc:1875] T f46393257ce9471891698c36a07546ad P 1b411c6bb25549368041773d7b710d48: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:10.761885 28330 maintenance_manager.cc:643] P d09a11a03989421b9f28019dc1c14cc2: FlushMRSOp(775b8190036b4b828431fca19ce23ea9) complete. Timing: real 0.311s user 0.039s sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":1213,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":1475,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3186,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":14973,"threads_started":1,"wal-append.queue_time_us":11166}
I20240503 08:14:10.765501 28439 maintenance_manager.cc:419] P d09a11a03989421b9f28019dc1c14cc2: Scheduling UndoDeltaBlockGCOp(d94f5824a71f454f8a1bfa13dd032684): 4923 bytes on disk
I20240503 08:14:10.766598 28330 maintenance_manager.cc:643] P d09a11a03989421b9f28019dc1c14cc2: UndoDeltaBlockGCOp(d94f5824a71f454f8a1bfa13dd032684) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
I20240503 08:14:10.769910 28439 maintenance_manager.cc:419] P d09a11a03989421b9f28019dc1c14cc2: Scheduling UndoDeltaBlockGCOp(775b8190036b4b828431fca19ce23ea9): 4583 bytes on disk
I20240503 08:14:10.770493 28330 maintenance_manager.cc:643] P d09a11a03989421b9f28019dc1c14cc2: UndoDeltaBlockGCOp(775b8190036b4b828431fca19ce23ea9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20240503 08:14:10.788631 9370 ts_tablet_manager.cc:1888] T f46393257ce9471891698c36a07546ad P 1b411c6bb25549368041773d7b710d48: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:14:10.791771 9370 log.cc:1196] T f46393257ce9471891698c36a07546ad P 1b411c6bb25549368041773d7b710d48: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/f46393257ce9471891698c36a07546ad
I20240503 08:14:10.794291 27848 catalog_manager.cc:4830] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): tablet f46393257ce9471891698c36a07546ad (table rebalance_test_table_7 [id=f65dccb36e67468c8e28e95e66b5e4ef]) successfully deleted
I20240503 08:14:10.905000 28200 maintenance_manager.cc:643] P aae860d7ae2b4002bf7794c278f9cc56: FlushMRSOp(775b8190036b4b828431fca19ce23ea9) complete. Timing: real 0.227s user 0.050s sys 0.014s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":47,"dirs.run_cpu_time_us":161,"dirs.run_wall_time_us":1328,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1436,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":28435,"threads_started":1,"wal-append.queue_time_us":28543}
I20240503 08:14:10.913275 28309 maintenance_manager.cc:419] P aae860d7ae2b4002bf7794c278f9cc56: Scheduling UndoDeltaBlockGCOp(d94f5824a71f454f8a1bfa13dd032684): 4923 bytes on disk
I20240503 08:14:10.914894 28200 maintenance_manager.cc:643] P aae860d7ae2b4002bf7794c278f9cc56: UndoDeltaBlockGCOp(d94f5824a71f454f8a1bfa13dd032684) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
I20240503 08:14:10.918262 28309 maintenance_manager.cc:419] P aae860d7ae2b4002bf7794c278f9cc56: Scheduling UndoDeltaBlockGCOp(d9098f3d36b24d578a3f8d7d3ccad052): 4838 bytes on disk
I20240503 08:14:10.918879 28200 maintenance_manager.cc:643] P aae860d7ae2b4002bf7794c278f9cc56: UndoDeltaBlockGCOp(d9098f3d36b24d578a3f8d7d3ccad052) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
I20240503 08:14:10.924063 28309 maintenance_manager.cc:419] P aae860d7ae2b4002bf7794c278f9cc56: Scheduling UndoDeltaBlockGCOp(775b8190036b4b828431fca19ce23ea9): 4583 bytes on disk
I20240503 08:14:10.924698 28200 maintenance_manager.cc:643] P aae860d7ae2b4002bf7794c278f9cc56: UndoDeltaBlockGCOp(775b8190036b4b828431fca19ce23ea9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
I20240503 08:14:11.735359 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 08:14:11.779228 30694 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.72:40893, user_credentials={real_user=slave}} blocked reactor thread for 81241.6us
I20240503 08:14:11.786356 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:14:11.809293 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:14:11.851940 28198 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.69:34725, user_credentials={real_user=slave}} blocked reactor thread for 139403us
I20240503 08:14:11.875244 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:11.887094 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:11.921200 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:11.951907 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:14:12.003716 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:12.138690 30894 consensus_queue.cc:237] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:12.141041 28263 raft_consensus.cc:1261] T f83a4778ffe640ee907e0a73e58abb73 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:12.141058 31026 raft_consensus.cc:1261] T f83a4778ffe640ee907e0a73e58abb73 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:12.141603 8876 consensus_queue.cc:1035] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:12.141912 9443 consensus_queue.cc:1035] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:12.156267 8876 raft_consensus.cc:2932] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:12.162031 28260 raft_consensus.cc:2932] T f83a4778ffe640ee907e0a73e58abb73 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:12.169441 27853 catalog_manager.cc:5484] T f83a4778ffe640ee907e0a73e58abb73 P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: config changed from index 2 to 3, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New cstate: current_term: 1 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:12.183065 31027 raft_consensus.cc:2932] T f83a4778ffe640ee907e0a73e58abb73 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:12.206465 30742 tablet_service.cc:1514] Processing DeleteTablet for tablet f83a4778ffe640ee907e0a73e58abb73 with delete_type TABLET_DATA_TOMBSTONED (TS 6d1d7de19459435fbde1999e3dfd226b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46794
I20240503 08:14:12.207088 9476 tablet_replica.cc:330] T f83a4778ffe640ee907e0a73e58abb73 P 6d1d7de19459435fbde1999e3dfd226b: stopping tablet replica
I20240503 08:14:12.207309 9476 raft_consensus.cc:2229] T f83a4778ffe640ee907e0a73e58abb73 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:12.207527 9476 raft_consensus.cc:2258] T f83a4778ffe640ee907e0a73e58abb73 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:12.208878 9476 ts_tablet_manager.cc:1875] T f83a4778ffe640ee907e0a73e58abb73 P 6d1d7de19459435fbde1999e3dfd226b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:12.214758 9476 ts_tablet_manager.cc:1888] T f83a4778ffe640ee907e0a73e58abb73 P 6d1d7de19459435fbde1999e3dfd226b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:14:12.214898 9476 log.cc:1196] T f83a4778ffe640ee907e0a73e58abb73 P 6d1d7de19459435fbde1999e3dfd226b: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal/wals/f83a4778ffe640ee907e0a73e58abb73
I20240503 08:14:12.221961 27848 catalog_manager.cc:4830] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): tablet f83a4778ffe640ee907e0a73e58abb73 (table rebalancer_extra_table_2 [id=7358500a74314a5a8795ddb89337f666]) successfully deleted
I20240503 08:14:12.453022 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:12.462767 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:12.467135 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:12.504734 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:14:12.516109 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:12.546134 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:12.557685 28439 maintenance_manager.cc:419] P d09a11a03989421b9f28019dc1c14cc2: Scheduling FlushMRSOp(8bcbc022caa94bb08a2db4de9b060557): perf score=0.033350
I20240503 08:14:12.562940 28179 maintenance_manager.cc:419] P 1b411c6bb25549368041773d7b710d48: Scheduling FlushMRSOp(8450e62352a74d2ea1471166d4cb1132): perf score=0.033355
I20240503 08:14:12.583909 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:14:12.610478 28049 maintenance_manager.cc:419] P 833582ff68c94269be364ebd187a75db: Scheduling FlushMRSOp(d3da6397e0c34c8f96933963dd4fa65e): perf score=0.033357
I20240503 08:14:12.643602 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:12.700755 28309 maintenance_manager.cc:419] P aae860d7ae2b4002bf7794c278f9cc56: Scheduling FlushMRSOp(8450e62352a74d2ea1471166d4cb1132): perf score=0.033393
I20240503 08:14:12.776667 27940 maintenance_manager.cc:643] P 833582ff68c94269be364ebd187a75db: FlushMRSOp(d3da6397e0c34c8f96933963dd4fa65e) complete. Timing: real 0.166s user 0.042s sys 0.005s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":9190,"dirs.run_cpu_time_us":230,"dirs.run_wall_time_us":30461,"drs_written":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4,"lbm_write_time_us":727,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":22162,"threads_started":2,"wal-append.queue_time_us":10238}
I20240503 08:14:12.780987 28049 maintenance_manager.cc:419] P 833582ff68c94269be364ebd187a75db: Scheduling UndoDeltaBlockGCOp(d3da6397e0c34c8f96933963dd4fa65e): 4581 bytes on disk
I20240503 08:14:12.781607 27940 maintenance_manager.cc:643] P 833582ff68c94269be364ebd187a75db: UndoDeltaBlockGCOp(d3da6397e0c34c8f96933963dd4fa65e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
I20240503 08:14:12.795543 28070 maintenance_manager.cc:643] P 1b411c6bb25549368041773d7b710d48: FlushMRSOp(8450e62352a74d2ea1471166d4cb1132) complete. Timing: real 0.232s user 0.058s sys 0.009s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":38489,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":8484,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":782,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":38808,"threads_started":2,"wal-append.queue_time_us":411}
I20240503 08:14:12.798398 28179 maintenance_manager.cc:419] P 1b411c6bb25549368041773d7b710d48: Scheduling UndoDeltaBlockGCOp(8450e62352a74d2ea1471166d4cb1132): 4923 bytes on disk
I20240503 08:14:12.799005 28070 maintenance_manager.cc:643] P 1b411c6bb25549368041773d7b710d48: UndoDeltaBlockGCOp(8450e62352a74d2ea1471166d4cb1132) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240503 08:14:12.823539 28330 maintenance_manager.cc:643] P d09a11a03989421b9f28019dc1c14cc2: FlushMRSOp(8bcbc022caa94bb08a2db4de9b060557) complete. Timing: real 0.265s user 0.045s sys 0.014s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":10493,"dirs.run_cpu_time_us":169,"dirs.run_wall_time_us":3377,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":754,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":29856,"threads_started":2,"wal-append.queue_time_us":16003}
I20240503 08:14:12.826689 28439 maintenance_manager.cc:419] P d09a11a03989421b9f28019dc1c14cc2: Scheduling FlushMRSOp(d3da6397e0c34c8f96933963dd4fa65e): perf score=0.033420
I20240503 08:14:12.835927 28179 maintenance_manager.cc:419] P 1b411c6bb25549368041773d7b710d48: Scheduling FlushMRSOp(8bcbc022caa94bb08a2db4de9b060557): perf score=0.033420
I20240503 08:14:12.864243 28200 maintenance_manager.cc:643] P aae860d7ae2b4002bf7794c278f9cc56: FlushMRSOp(8450e62352a74d2ea1471166d4cb1132) complete. Timing: real 0.163s user 0.062s sys 0.015s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":332,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":2633,"drs_written":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4,"lbm_write_time_us":619,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":10921,"threads_started":2,"wal-append.queue_time_us":6472}
I20240503 08:14:12.867882 28309 maintenance_manager.cc:419] P aae860d7ae2b4002bf7794c278f9cc56: Scheduling FlushMRSOp(8bcbc022caa94bb08a2db4de9b060557): perf score=0.033428
I20240503 08:14:12.898993 28070 maintenance_manager.cc:643] P 1b411c6bb25549368041773d7b710d48: FlushMRSOp(8bcbc022caa94bb08a2db4de9b060557) complete. Timing: real 0.062s user 0.034s sys 0.016s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":56,"dirs.run_cpu_time_us":165,"dirs.run_wall_time_us":1438,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":803,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":6491,"threads_started":1,"wal-append.queue_time_us":6390}
I20240503 08:14:12.903728 28330 maintenance_manager.cc:643] P d09a11a03989421b9f28019dc1c14cc2: FlushMRSOp(d3da6397e0c34c8f96933963dd4fa65e) complete. Timing: real 0.077s user 0.035s sys 0.015s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":149,"dirs.run_wall_time_us":2119,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":871,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":8481,"threads_started":1,"wal-append.queue_time_us":8569}
I20240503 08:14:12.906275 28179 maintenance_manager.cc:419] P 1b411c6bb25549368041773d7b710d48: Scheduling UndoDeltaBlockGCOp(8bcbc022caa94bb08a2db4de9b060557): 4838 bytes on disk
I20240503 08:14:12.906885 28070 maintenance_manager.cc:643] P 1b411c6bb25549368041773d7b710d48: UndoDeltaBlockGCOp(8bcbc022caa94bb08a2db4de9b060557) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240503 08:14:12.907088 28439 maintenance_manager.cc:419] P d09a11a03989421b9f28019dc1c14cc2: Scheduling UndoDeltaBlockGCOp(8bcbc022caa94bb08a2db4de9b060557): 4838 bytes on disk
I20240503 08:14:12.908026 28330 maintenance_manager.cc:643] P d09a11a03989421b9f28019dc1c14cc2: UndoDeltaBlockGCOp(8bcbc022caa94bb08a2db4de9b060557) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240503 08:14:12.911264 28439 maintenance_manager.cc:419] P d09a11a03989421b9f28019dc1c14cc2: Scheduling UndoDeltaBlockGCOp(d3da6397e0c34c8f96933963dd4fa65e): 4581 bytes on disk
I20240503 08:14:12.911865 28330 maintenance_manager.cc:643] P d09a11a03989421b9f28019dc1c14cc2: UndoDeltaBlockGCOp(d3da6397e0c34c8f96933963dd4fa65e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20240503 08:14:12.925540 28200 maintenance_manager.cc:643] P aae860d7ae2b4002bf7794c278f9cc56: FlushMRSOp(8bcbc022caa94bb08a2db4de9b060557) complete. Timing: real 0.056s user 0.031s sys 0.011s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":137,"dirs.run_wall_time_us":1008,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":881,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":1589,"threads_started":1,"wal-append.queue_time_us":1690}
I20240503 08:14:12.929389 28309 maintenance_manager.cc:419] P aae860d7ae2b4002bf7794c278f9cc56: Scheduling UndoDeltaBlockGCOp(8450e62352a74d2ea1471166d4cb1132): 4923 bytes on disk
I20240503 08:14:12.930105 28200 maintenance_manager.cc:643] P aae860d7ae2b4002bf7794c278f9cc56: UndoDeltaBlockGCOp(8450e62352a74d2ea1471166d4cb1132) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
I20240503 08:14:12.933373 28309 maintenance_manager.cc:419] P aae860d7ae2b4002bf7794c278f9cc56: Scheduling UndoDeltaBlockGCOp(8bcbc022caa94bb08a2db4de9b060557): 4838 bytes on disk
I20240503 08:14:12.934000 28200 maintenance_manager.cc:643] P aae860d7ae2b4002bf7794c278f9cc56: UndoDeltaBlockGCOp(8bcbc022caa94bb08a2db4de9b060557) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
I20240503 08:14:12.958889 28391 consensus_queue.cc:237] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:14:12.960723 30893 raft_consensus.cc:1261] T 775b8190036b4b828431fca19ce23ea9 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:12.960847 28260 raft_consensus.cc:1261] T 775b8190036b4b828431fca19ce23ea9 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:12.961259 9360 consensus_queue.cc:1035] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:12.961562 9152 consensus_queue.cc:1035] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:12.965051 9152 raft_consensus.cc:2932] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:14:12.967823 28260 raft_consensus.cc:2932] T 775b8190036b4b828431fca19ce23ea9 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:14:12.968096 30893 raft_consensus.cc:2932] T 775b8190036b4b828431fca19ce23ea9 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:14:12.972960 27861 catalog_manager.cc:5484] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: config changed from index 4 to 5, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:14:12.985682 28111 tablet_service.cc:1514] Processing DeleteTablet for tablet 775b8190036b4b828431fca19ce23ea9 with delete_type TABLET_DATA_TOMBSTONED (TS 1b411c6bb25549368041773d7b710d48 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57912
I20240503 08:14:12.986269 9567 tablet_replica.cc:330] T 775b8190036b4b828431fca19ce23ea9 P 1b411c6bb25549368041773d7b710d48: stopping tablet replica
I20240503 08:14:12.986446 9567 raft_consensus.cc:2229] T 775b8190036b4b828431fca19ce23ea9 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:12.986749 9567 raft_consensus.cc:2258] T 775b8190036b4b828431fca19ce23ea9 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:12.987866 9567 ts_tablet_manager.cc:1875] T 775b8190036b4b828431fca19ce23ea9 P 1b411c6bb25549368041773d7b710d48: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:13.003610 9567 ts_tablet_manager.cc:1888] T 775b8190036b4b828431fca19ce23ea9 P 1b411c6bb25549368041773d7b710d48: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:14:13.003842 9567 log.cc:1196] T 775b8190036b4b828431fca19ce23ea9 P 1b411c6bb25549368041773d7b710d48: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/775b8190036b4b828431fca19ce23ea9
I20240503 08:14:13.004556 27848 catalog_manager.cc:4830] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): tablet 775b8190036b4b828431fca19ce23ea9 (table rebalance_test_table_1 [id=05d29869a30847b6a0c137521965898a]) successfully deleted
I20240503 08:14:13.345352 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:13.362979 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:14:13.373363 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:13.383945 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:14:13.394241 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:13.414371 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:13.425262 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:13.457952 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:13.650676 31161 consensus_queue.cc:237] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:13.653074 28260 raft_consensus.cc:1261] T 486e9843fd784148b9b49dfe27996dff P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:13.653308 31027 raft_consensus.cc:1261] T 486e9843fd784148b9b49dfe27996dff P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:13.654145 8995 consensus_queue.cc:1035] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:13.654752 8995 consensus_queue.cc:1035] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 08:14:13.658389 9564 raft_consensus.cc:2932] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:13.662725 28260 raft_consensus.cc:2932] T 486e9843fd784148b9b49dfe27996dff P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:13.669260 27857 catalog_manager.cc:5484] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98 reported cstate change: config changed from index 2 to 3, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New cstate: current_term: 1 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:13.688994 31026 raft_consensus.cc:2932] T 486e9843fd784148b9b49dfe27996dff P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:13.704231 30742 tablet_service.cc:1514] Processing DeleteTablet for tablet 486e9843fd784148b9b49dfe27996dff with delete_type TABLET_DATA_TOMBSTONED (TS 6d1d7de19459435fbde1999e3dfd226b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46794
I20240503 08:14:13.706313 9665 tablet_replica.cc:330] T 486e9843fd784148b9b49dfe27996dff P 6d1d7de19459435fbde1999e3dfd226b: stopping tablet replica
I20240503 08:14:13.706586 9665 raft_consensus.cc:2229] T 486e9843fd784148b9b49dfe27996dff P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:13.707173 9665 raft_consensus.cc:2258] T 486e9843fd784148b9b49dfe27996dff P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:13.707988 9665 ts_tablet_manager.cc:1875] T 486e9843fd784148b9b49dfe27996dff P 6d1d7de19459435fbde1999e3dfd226b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:13.717815 9665 ts_tablet_manager.cc:1888] T 486e9843fd784148b9b49dfe27996dff P 6d1d7de19459435fbde1999e3dfd226b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:14:13.718003 9665 log.cc:1196] T 486e9843fd784148b9b49dfe27996dff P 6d1d7de19459435fbde1999e3dfd226b: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal/wals/486e9843fd784148b9b49dfe27996dff
I20240503 08:14:13.718719 27848 catalog_manager.cc:4830] TS 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): tablet 486e9843fd784148b9b49dfe27996dff (table rebalancer_extra_table_22 [id=11f85d35fe5f401d990c665eefacf247]) successfully deleted
I20240503 08:14:13.986188 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:13.987435 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:13.994077 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:14.019656 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:14:14.023276 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:14:14.066434 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:14.100708 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:14.156401 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:14.311122 31159 consensus_queue.cc:237] T f463dfe26ee94b5f9447c6dbdf8759c3 P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:14.312911 31027 raft_consensus.cc:1261] T f463dfe26ee94b5f9447c6dbdf8759c3 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:14.313206 28391 raft_consensus.cc:1261] T f463dfe26ee94b5f9447c6dbdf8759c3 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:14.313661 9564 consensus_queue.cc:1035] T f463dfe26ee94b5f9447c6dbdf8759c3 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:14.313998 8995 consensus_queue.cc:1035] T f463dfe26ee94b5f9447c6dbdf8759c3 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:14.316769 8995 raft_consensus.cc:2932] T f463dfe26ee94b5f9447c6dbdf8759c3 P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:14:14.317369 28389 raft_consensus.cc:2932] T f463dfe26ee94b5f9447c6dbdf8759c3 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:14:14.317606 31027 raft_consensus.cc:2932] T f463dfe26ee94b5f9447c6dbdf8759c3 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:14:14.322932 27853 catalog_manager.cc:5484] T f463dfe26ee94b5f9447c6dbdf8759c3 P b00f244c69df401997ea99d492218a98 reported cstate change: config changed from index 2 to 3, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New cstate: current_term: 1 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:14:14.332056 28111 tablet_service.cc:1514] Processing DeleteTablet for tablet f463dfe26ee94b5f9447c6dbdf8759c3 with delete_type TABLET_DATA_TOMBSTONED (TS 1b411c6bb25549368041773d7b710d48 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:57912
I20240503 08:14:14.334582 9751 tablet_replica.cc:330] T f463dfe26ee94b5f9447c6dbdf8759c3 P 1b411c6bb25549368041773d7b710d48: stopping tablet replica
I20240503 08:14:14.334784 9751 raft_consensus.cc:2229] T f463dfe26ee94b5f9447c6dbdf8759c3 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:14.335000 9751 raft_consensus.cc:2258] T f463dfe26ee94b5f9447c6dbdf8759c3 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:14.335673 9751 ts_tablet_manager.cc:1875] T f463dfe26ee94b5f9447c6dbdf8759c3 P 1b411c6bb25549368041773d7b710d48: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:14.341292 9751 ts_tablet_manager.cc:1888] T f463dfe26ee94b5f9447c6dbdf8759c3 P 1b411c6bb25549368041773d7b710d48: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:14:14.341406 9751 log.cc:1196] T f463dfe26ee94b5f9447c6dbdf8759c3 P 1b411c6bb25549368041773d7b710d48: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/f463dfe26ee94b5f9447c6dbdf8759c3
I20240503 08:14:14.342104 27848 catalog_manager.cc:4830] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): tablet f463dfe26ee94b5f9447c6dbdf8759c3 (table rebalancer_extra_table_58 [id=69e4cfedb64b4938988b9ed5d473eec0]) successfully deleted
I20240503 08:14:14.727334 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:14.746428 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:14:14.758770 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:14.759703 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:14.779129 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:14.838848 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:14.846406 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:14:14.917840 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:15.022143 30761 consensus_queue.cc:237] T 4ebf45bdc432451e87b7c98d435a76d2 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:14:15.024552 28000 raft_consensus.cc:1261] T 4ebf45bdc432451e87b7c98d435a76d2 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:15.024685 31027 raft_consensus.cc:1261] T 4ebf45bdc432451e87b7c98d435a76d2 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:15.025009 9442 consensus_queue.cc:1035] T 4ebf45bdc432451e87b7c98d435a76d2 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:15.025393 9752 consensus_queue.cc:1035] T 4ebf45bdc432451e87b7c98d435a76d2 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:15.032856 9752 raft_consensus.cc:2932] T 4ebf45bdc432451e87b7c98d435a76d2 P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:14:15.035691 28000 raft_consensus.cc:2932] T 4ebf45bdc432451e87b7c98d435a76d2 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:14:15.043264 27860 catalog_manager.cc:5484] T 4ebf45bdc432451e87b7c98d435a76d2 P 833582ff68c94269be364ebd187a75db reported cstate change: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:14:15.044214 31027 raft_consensus.cc:2932] T 4ebf45bdc432451e87b7c98d435a76d2 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:14:15.051959 31141 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ebf45bdc432451e87b7c98d435a76d2 with delete_type TABLET_DATA_TOMBSTONED (TS b00f244c69df401997ea99d492218a98 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45814
I20240503 08:14:15.053164 9841 tablet_replica.cc:330] T 4ebf45bdc432451e87b7c98d435a76d2 P b00f244c69df401997ea99d492218a98: stopping tablet replica
I20240503 08:14:15.053473 9841 raft_consensus.cc:2229] T 4ebf45bdc432451e87b7c98d435a76d2 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:15.053820 9841 raft_consensus.cc:2258] T 4ebf45bdc432451e87b7c98d435a76d2 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:15.054744 9841 ts_tablet_manager.cc:1875] T 4ebf45bdc432451e87b7c98d435a76d2 P b00f244c69df401997ea99d492218a98: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:15.061110 9841 ts_tablet_manager.cc:1888] T 4ebf45bdc432451e87b7c98d435a76d2 P b00f244c69df401997ea99d492218a98: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:14:15.061242 9841 log.cc:1196] T 4ebf45bdc432451e87b7c98d435a76d2 P b00f244c69df401997ea99d492218a98: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal/wals/4ebf45bdc432451e87b7c98d435a76d2
I20240503 08:14:15.062067 27846 catalog_manager.cc:4830] TS b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): tablet 4ebf45bdc432451e87b7c98d435a76d2 (table rebalancer_extra_table_48 [id=65fb43133ebb48fa9b6c1fe9b9dd6f27]) successfully deleted
I20240503 08:14:15.154644 30893 consensus_queue.cc:237] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:14:15.157006 31161 raft_consensus.cc:1261] T 2401f8f101f94189aa3c0582f08cef88 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:15.157497 31027 raft_consensus.cc:1261] T 2401f8f101f94189aa3c0582f08cef88 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:15.158082 9666 consensus_queue.cc:1035] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 08:14:15.159408 30830 consensus_peers.cc:485] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c -> Peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Couldn't send request to peer 833582ff68c94269be364ebd187a75db. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2401f8f101f94189aa3c0582f08cef88. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:15.159986 9666 consensus_queue.cc:1035] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:15.166945 9666 raft_consensus.cc:2932] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:14:15.167598 31161 raft_consensus.cc:2932] T 2401f8f101f94189aa3c0582f08cef88 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:14:15.171990 31027 raft_consensus.cc:2932] T 2401f8f101f94189aa3c0582f08cef88 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:14:15.173465 27853 catalog_manager.cc:5484] T 2401f8f101f94189aa3c0582f08cef88 P b00f244c69df401997ea99d492218a98 reported cstate change: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New cstate: current_term: 1 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:14:15.208004 31161 consensus_queue.cc:237] T 644bb089a4064ecfb88438d8b859bf01 P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:14:15.210296 31027 raft_consensus.cc:1261] T 644bb089a4064ecfb88438d8b859bf01 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:15.210296 28391 raft_consensus.cc:1261] T 644bb089a4064ecfb88438d8b859bf01 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:14:15.210498 31097 consensus_peers.cc:485] T 644bb089a4064ecfb88438d8b859bf01 P b00f244c69df401997ea99d492218a98 -> Peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Couldn't send request to peer 833582ff68c94269be364ebd187a75db. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 644bb089a4064ecfb88438d8b859bf01. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:15.210834 9564 consensus_queue.cc:1035] T 644bb089a4064ecfb88438d8b859bf01 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:15.211081 8995 consensus_queue.cc:1035] T 644bb089a4064ecfb88438d8b859bf01 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:15.213847 9903 raft_consensus.cc:2932] T 644bb089a4064ecfb88438d8b859bf01 P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:14:15.214395 28391 raft_consensus.cc:2932] T 644bb089a4064ecfb88438d8b859bf01 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:14:15.221066 31027 raft_consensus.cc:2932] T 644bb089a4064ecfb88438d8b859bf01 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:14:15.225731 27860 catalog_manager.cc:5484] T 644bb089a4064ecfb88438d8b859bf01 P b00f244c69df401997ea99d492218a98 reported cstate change: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New cstate: current_term: 1 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:14:15.286108 28391 consensus_queue.cc:237] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:15.288199 28133 raft_consensus.cc:1261] T 03b02a4425784a30b1007b3a14139d8d P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:15.288492 31027 raft_consensus.cc:1261] T 03b02a4425784a30b1007b3a14139d8d P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:14:15.288815 28327 consensus_peers.cc:485] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 03b02a4425784a30b1007b3a14139d8d. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:15.289110 9152 consensus_queue.cc:1035] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:15.289649 9152 consensus_queue.cc:1035] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:15.294118 9152 raft_consensus.cc:2932] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:15.294605 31027 raft_consensus.cc:2932] T 03b02a4425784a30b1007b3a14139d8d P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:15.302402 27853 catalog_manager.cc:5484] T 03b02a4425784a30b1007b3a14139d8d P ced21ca983b749d0b4d4f6a0e2426bf8 reported cstate change: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:15.307961 28133 raft_consensus.cc:2932] T 03b02a4425784a30b1007b3a14139d8d P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:15.342339 30761 consensus_queue.cc:237] T e30557ffd4594f7d9cda13025e4c5e60 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:14:15.344715 28000 raft_consensus.cc:1261] T e30557ffd4594f7d9cda13025e4c5e60 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:14:15.344924 30694 consensus_peers.cc:485] T e30557ffd4594f7d9cda13025e4c5e60 P 6d1d7de19459435fbde1999e3dfd226b -> Peer b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): Couldn't send request to peer b00f244c69df401997ea99d492218a98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e30557ffd4594f7d9cda13025e4c5e60. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:15.345139 31027 raft_consensus.cc:1261] T e30557ffd4594f7d9cda13025e4c5e60 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:15.345479 9752 consensus_queue.cc:1035] T e30557ffd4594f7d9cda13025e4c5e60 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:15.345810 9832 consensus_queue.cc:1035] T e30557ffd4594f7d9cda13025e4c5e60 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:15.349126 9752 raft_consensus.cc:2932] T e30557ffd4594f7d9cda13025e4c5e60 P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:14:15.349674 31027 raft_consensus.cc:2932] T e30557ffd4594f7d9cda13025e4c5e60 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:14:15.355983 28000 raft_consensus.cc:2932] T e30557ffd4594f7d9cda13025e4c5e60 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:14:15.357672 27853 catalog_manager.cc:5484] T e30557ffd4594f7d9cda13025e4c5e60 P ced21ca983b749d0b4d4f6a0e2426bf8 reported cstate change: config changed from index -1 to 2, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) added. New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:14:15.390980 30761 consensus_queue.cc:237] T 71fa62206f744de0b97e3aa9ea753e37 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:14:15.393617 28391 raft_consensus.cc:1261] T 71fa62206f744de0b97e3aa9ea753e37 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:14:15.393977 30694 consensus_peers.cc:485] T 71fa62206f744de0b97e3aa9ea753e37 P 6d1d7de19459435fbde1999e3dfd226b -> Peer ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): Couldn't send request to peer ce1496a47d084365ad786cd09e472a2c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71fa62206f744de0b97e3aa9ea753e37. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:15.394173 31161 raft_consensus.cc:1261] T 71fa62206f744de0b97e3aa9ea753e37 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:15.394385 9442 consensus_queue.cc:1035] T 71fa62206f744de0b97e3aa9ea753e37 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:15.394690 9752 consensus_queue.cc:1035] T 71fa62206f744de0b97e3aa9ea753e37 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:15.407562 9442 raft_consensus.cc:2932] T 71fa62206f744de0b97e3aa9ea753e37 P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:14:15.408334 31159 raft_consensus.cc:2932] T 71fa62206f744de0b97e3aa9ea753e37 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:14:15.415288 27853 catalog_manager.cc:5484] T 71fa62206f744de0b97e3aa9ea753e37 P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: config changed from index -1 to 2, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:14:15.426841 28391 raft_consensus.cc:2932] T 71fa62206f744de0b97e3aa9ea753e37 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:14:15.751307 10010 ts_tablet_manager.cc:902] T 644bb089a4064ecfb88438d8b859bf01 P 833582ff68c94269be364ebd187a75db: Initiating tablet copy from peer b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:14:15.764287 10008 ts_tablet_manager.cc:902] T 03b02a4425784a30b1007b3a14139d8d P aae860d7ae2b4002bf7794c278f9cc56: Initiating tablet copy from peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:14:15.766968 10013 ts_tablet_manager.cc:902] T 2401f8f101f94189aa3c0582f08cef88 P 833582ff68c94269be364ebd187a75db: Initiating tablet copy from peer ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:14:15.796260 10013 tablet_copy_client.cc:308] T 2401f8f101f94189aa3c0582f08cef88 P 833582ff68c94269be364ebd187a75db: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.70:33747
I20240503 08:14:15.797034 30915 tablet_copy_service.cc:140] P ce1496a47d084365ad786cd09e472a2c: Received BeginTabletCopySession request for tablet 2401f8f101f94189aa3c0582f08cef88 from peer 833582ff68c94269be364ebd187a75db ({username='slave'} at 127.20.230.65:60309)
I20240503 08:14:15.797201 30915 tablet_copy_service.cc:161] P ce1496a47d084365ad786cd09e472a2c: Beginning new tablet copy session on tablet 2401f8f101f94189aa3c0582f08cef88 from peer 833582ff68c94269be364ebd187a75db at {username='slave'} at 127.20.230.65:60309: session id = 833582ff68c94269be364ebd187a75db-2401f8f101f94189aa3c0582f08cef88
I20240503 08:14:15.800160 30915 tablet_copy_source_session.cc:205] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:15.801591 10013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2401f8f101f94189aa3c0582f08cef88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:15.805047 10010 tablet_copy_client.cc:308] T 644bb089a4064ecfb88438d8b859bf01 P 833582ff68c94269be364ebd187a75db: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.72:40893
I20240503 08:14:15.805578 31181 tablet_copy_service.cc:140] P b00f244c69df401997ea99d492218a98: Received BeginTabletCopySession request for tablet 644bb089a4064ecfb88438d8b859bf01 from peer 833582ff68c94269be364ebd187a75db ({username='slave'} at 127.20.230.65:40115)
I20240503 08:14:15.805714 31181 tablet_copy_service.cc:161] P b00f244c69df401997ea99d492218a98: Beginning new tablet copy session on tablet 644bb089a4064ecfb88438d8b859bf01 from peer 833582ff68c94269be364ebd187a75db at {username='slave'} at 127.20.230.65:40115: session id = 833582ff68c94269be364ebd187a75db-644bb089a4064ecfb88438d8b859bf01
I20240503 08:14:15.808568 31181 tablet_copy_source_session.cc:205] T 644bb089a4064ecfb88438d8b859bf01 P b00f244c69df401997ea99d492218a98: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:15.811103 10010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 644bb089a4064ecfb88438d8b859bf01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:15.812908 10013 tablet_copy_client.cc:783] T 2401f8f101f94189aa3c0582f08cef88 P 833582ff68c94269be364ebd187a75db: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:15.813143 10013 tablet_copy_client.cc:647] T 2401f8f101f94189aa3c0582f08cef88 P 833582ff68c94269be364ebd187a75db: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:15.816182 10008 tablet_copy_client.cc:308] T 03b02a4425784a30b1007b3a14139d8d P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.68:46211
I20240503 08:14:15.817209 28413 tablet_copy_service.cc:140] P d09a11a03989421b9f28019dc1c14cc2: Received BeginTabletCopySession request for tablet 03b02a4425784a30b1007b3a14139d8d from peer aae860d7ae2b4002bf7794c278f9cc56 ({username='slave'} at 127.20.230.67:39219)
I20240503 08:14:15.817338 28413 tablet_copy_service.cc:161] P d09a11a03989421b9f28019dc1c14cc2: Beginning new tablet copy session on tablet 03b02a4425784a30b1007b3a14139d8d from peer aae860d7ae2b4002bf7794c278f9cc56 at {username='slave'} at 127.20.230.67:39219: session id = aae860d7ae2b4002bf7794c278f9cc56-03b02a4425784a30b1007b3a14139d8d
I20240503 08:14:15.818938 10013 tablet_copy_client.cc:523] T 2401f8f101f94189aa3c0582f08cef88 P 833582ff68c94269be364ebd187a75db: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:15.822002 10013 tablet_bootstrap.cc:492] T 2401f8f101f94189aa3c0582f08cef88 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:14:15.827925 28413 tablet_copy_source_session.cc:205] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:15.829579 10008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03b02a4425784a30b1007b3a14139d8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:15.836877 10010 tablet_copy_client.cc:783] T 644bb089a4064ecfb88438d8b859bf01 P 833582ff68c94269be364ebd187a75db: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:15.837086 10010 tablet_copy_client.cc:647] T 644bb089a4064ecfb88438d8b859bf01 P 833582ff68c94269be364ebd187a75db: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:15.840620 10010 tablet_copy_client.cc:523] T 644bb089a4064ecfb88438d8b859bf01 P 833582ff68c94269be364ebd187a75db: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:15.841943 10008 tablet_copy_client.cc:783] T 03b02a4425784a30b1007b3a14139d8d P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:15.842156 10008 tablet_copy_client.cc:647] T 03b02a4425784a30b1007b3a14139d8d P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:15.843973 10008 tablet_copy_client.cc:523] T 03b02a4425784a30b1007b3a14139d8d P aae860d7ae2b4002bf7794c278f9cc56: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:15.850049 10008 tablet_bootstrap.cc:492] T 03b02a4425784a30b1007b3a14139d8d P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap starting.
I20240503 08:14:15.851658 10010 tablet_bootstrap.cc:492] T 644bb089a4064ecfb88438d8b859bf01 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:14:15.866649 10013 tablet_bootstrap.cc:492] T 2401f8f101f94189aa3c0582f08cef88 P 833582ff68c94269be364ebd187a75db: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:15.867084 10013 tablet_bootstrap.cc:492] T 2401f8f101f94189aa3c0582f08cef88 P 833582ff68c94269be364ebd187a75db: Bootstrap complete.
I20240503 08:14:15.867334 10013 ts_tablet_manager.cc:1367] T 2401f8f101f94189aa3c0582f08cef88 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.045s user 0.031s sys 0.012s
I20240503 08:14:15.868031 10013 raft_consensus.cc:348] T 2401f8f101f94189aa3c0582f08cef88 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:14:15.868326 10013 raft_consensus.cc:729] T 2401f8f101f94189aa3c0582f08cef88 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:14:15.868562 10013 consensus_queue.cc:260] T 2401f8f101f94189aa3c0582f08cef88 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:14:15.869055 10013 ts_tablet_manager.cc:1398] T 2401f8f101f94189aa3c0582f08cef88 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:14:15.869599 30915 tablet_copy_service.cc:342] P ce1496a47d084365ad786cd09e472a2c: Request end of tablet copy session 833582ff68c94269be364ebd187a75db-2401f8f101f94189aa3c0582f08cef88 received from {username='slave'} at 127.20.230.65:60309
I20240503 08:14:15.869704 30915 tablet_copy_service.cc:433] P ce1496a47d084365ad786cd09e472a2c: ending tablet copy session 833582ff68c94269be364ebd187a75db-2401f8f101f94189aa3c0582f08cef88 on tablet 2401f8f101f94189aa3c0582f08cef88 with peer 833582ff68c94269be364ebd187a75db
I20240503 08:14:15.876811 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:15.877374 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:15.892547 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:15.905361 10008 tablet_bootstrap.cc:492] T 03b02a4425784a30b1007b3a14139d8d P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:15.905784 10008 tablet_bootstrap.cc:492] T 03b02a4425784a30b1007b3a14139d8d P aae860d7ae2b4002bf7794c278f9cc56: Bootstrap complete.
I20240503 08:14:15.906037 10008 ts_tablet_manager.cc:1367] T 03b02a4425784a30b1007b3a14139d8d P aae860d7ae2b4002bf7794c278f9cc56: Time spent bootstrapping tablet: real 0.056s user 0.006s sys 0.000s
I20240503 08:14:15.906661 10008 raft_consensus.cc:348] T 03b02a4425784a30b1007b3a14139d8d P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:15.906889 10008 raft_consensus.cc:729] T 03b02a4425784a30b1007b3a14139d8d P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae860d7ae2b4002bf7794c278f9cc56, State: Initialized, Role: FOLLOWER
I20240503 08:14:15.908283 10008 consensus_queue.cc:260] T 03b02a4425784a30b1007b3a14139d8d P aae860d7ae2b4002bf7794c278f9cc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:15.908820 10008 ts_tablet_manager.cc:1398] T 03b02a4425784a30b1007b3a14139d8d P aae860d7ae2b4002bf7794c278f9cc56: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:14:15.909386 28413 tablet_copy_service.cc:342] P d09a11a03989421b9f28019dc1c14cc2: Request end of tablet copy session aae860d7ae2b4002bf7794c278f9cc56-03b02a4425784a30b1007b3a14139d8d received from {username='slave'} at 127.20.230.67:39219
I20240503 08:14:15.909487 28413 tablet_copy_service.cc:433] P d09a11a03989421b9f28019dc1c14cc2: ending tablet copy session aae860d7ae2b4002bf7794c278f9cc56-03b02a4425784a30b1007b3a14139d8d on tablet 03b02a4425784a30b1007b3a14139d8d with peer aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:15.913197 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:15.928740 10043 ts_tablet_manager.cc:902] T e30557ffd4594f7d9cda13025e4c5e60 P b00f244c69df401997ea99d492218a98: Initiating tablet copy from peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:14:15.950157 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:14:15.950203 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:15.959183 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:14:15.964771 10010 tablet_bootstrap.cc:492] T 644bb089a4064ecfb88438d8b859bf01 P 833582ff68c94269be364ebd187a75db: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:15.965276 10010 tablet_bootstrap.cc:492] T 644bb089a4064ecfb88438d8b859bf01 P 833582ff68c94269be364ebd187a75db: Bootstrap complete.
I20240503 08:14:15.965569 10010 ts_tablet_manager.cc:1367] T 644bb089a4064ecfb88438d8b859bf01 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.114s user 0.011s sys 0.000s
I20240503 08:14:15.967892 10010 raft_consensus.cc:348] T 644bb089a4064ecfb88438d8b859bf01 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:14:15.969117 10010 raft_consensus.cc:729] T 644bb089a4064ecfb88438d8b859bf01 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:14:15.969393 10010 consensus_queue.cc:260] T 644bb089a4064ecfb88438d8b859bf01 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:14:15.970023 10010 ts_tablet_manager.cc:1398] T 644bb089a4064ecfb88438d8b859bf01 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:14:15.994163 31181 tablet_copy_service.cc:342] P b00f244c69df401997ea99d492218a98: Request end of tablet copy session 833582ff68c94269be364ebd187a75db-644bb089a4064ecfb88438d8b859bf01 received from {username='slave'} at 127.20.230.65:40115
I20240503 08:14:15.994328 31181 tablet_copy_service.cc:433] P b00f244c69df401997ea99d492218a98: ending tablet copy session 833582ff68c94269be364ebd187a75db-644bb089a4064ecfb88438d8b859bf01 on tablet 644bb089a4064ecfb88438d8b859bf01 with peer 833582ff68c94269be364ebd187a75db
I20240503 08:14:16.038368 10053 ts_tablet_manager.cc:902] T 71fa62206f744de0b97e3aa9ea753e37 P ce1496a47d084365ad786cd09e472a2c: Initiating tablet copy from peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:14:16.060299 10043 tablet_copy_client.cc:308] T e30557ffd4594f7d9cda13025e4c5e60 P b00f244c69df401997ea99d492218a98: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.69:34725
I20240503 08:14:16.061009 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:16.060979 30783 tablet_copy_service.cc:140] P 6d1d7de19459435fbde1999e3dfd226b: Received BeginTabletCopySession request for tablet e30557ffd4594f7d9cda13025e4c5e60 from peer b00f244c69df401997ea99d492218a98 ({username='slave'} at 127.20.230.72:52827)
I20240503 08:14:16.061146 30783 tablet_copy_service.cc:161] P 6d1d7de19459435fbde1999e3dfd226b: Beginning new tablet copy session on tablet e30557ffd4594f7d9cda13025e4c5e60 from peer b00f244c69df401997ea99d492218a98 at {username='slave'} at 127.20.230.72:52827: session id = b00f244c69df401997ea99d492218a98-e30557ffd4594f7d9cda13025e4c5e60
I20240503 08:14:16.064041 30783 tablet_copy_source_session.cc:205] T e30557ffd4594f7d9cda13025e4c5e60 P 6d1d7de19459435fbde1999e3dfd226b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:16.065312 10043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e30557ffd4594f7d9cda13025e4c5e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:16.072356 10043 tablet_copy_client.cc:783] T e30557ffd4594f7d9cda13025e4c5e60 P b00f244c69df401997ea99d492218a98: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:16.072665 10043 tablet_copy_client.cc:647] T e30557ffd4594f7d9cda13025e4c5e60 P b00f244c69df401997ea99d492218a98: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:16.075235 10053 tablet_copy_client.cc:308] T 71fa62206f744de0b97e3aa9ea753e37 P ce1496a47d084365ad786cd09e472a2c: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.69:34725
I20240503 08:14:16.075806 30783 tablet_copy_service.cc:140] P 6d1d7de19459435fbde1999e3dfd226b: Received BeginTabletCopySession request for tablet 71fa62206f744de0b97e3aa9ea753e37 from peer ce1496a47d084365ad786cd09e472a2c ({username='slave'} at 127.20.230.70:41691)
I20240503 08:14:16.075940 30783 tablet_copy_service.cc:161] P 6d1d7de19459435fbde1999e3dfd226b: Beginning new tablet copy session on tablet 71fa62206f744de0b97e3aa9ea753e37 from peer ce1496a47d084365ad786cd09e472a2c at {username='slave'} at 127.20.230.70:41691: session id = ce1496a47d084365ad786cd09e472a2c-71fa62206f744de0b97e3aa9ea753e37
I20240503 08:14:16.079210 30783 tablet_copy_source_session.cc:205] T 71fa62206f744de0b97e3aa9ea753e37 P 6d1d7de19459435fbde1999e3dfd226b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:16.080360 10053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71fa62206f744de0b97e3aa9ea753e37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:16.085505 10053 tablet_copy_client.cc:783] T 71fa62206f744de0b97e3aa9ea753e37 P ce1496a47d084365ad786cd09e472a2c: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:16.085721 10053 tablet_copy_client.cc:647] T 71fa62206f744de0b97e3aa9ea753e37 P ce1496a47d084365ad786cd09e472a2c: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:16.088271 10043 tablet_copy_client.cc:523] T e30557ffd4594f7d9cda13025e4c5e60 P b00f244c69df401997ea99d492218a98: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:16.096304 10043 tablet_bootstrap.cc:492] T e30557ffd4594f7d9cda13025e4c5e60 P b00f244c69df401997ea99d492218a98: Bootstrap starting.
I20240503 08:14:16.102012 10053 tablet_copy_client.cc:523] T 71fa62206f744de0b97e3aa9ea753e37 P ce1496a47d084365ad786cd09e472a2c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:16.105146 10053 tablet_bootstrap.cc:492] T 71fa62206f744de0b97e3aa9ea753e37 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:14:16.112229 10043 tablet_bootstrap.cc:492] T e30557ffd4594f7d9cda13025e4c5e60 P b00f244c69df401997ea99d492218a98: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:16.112639 10043 tablet_bootstrap.cc:492] T e30557ffd4594f7d9cda13025e4c5e60 P b00f244c69df401997ea99d492218a98: Bootstrap complete.
I20240503 08:14:16.112871 10043 ts_tablet_manager.cc:1367] T e30557ffd4594f7d9cda13025e4c5e60 P b00f244c69df401997ea99d492218a98: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240503 08:14:16.113502 10043 raft_consensus.cc:348] T e30557ffd4594f7d9cda13025e4c5e60 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:14:16.113714 10043 raft_consensus.cc:729] T e30557ffd4594f7d9cda13025e4c5e60 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00f244c69df401997ea99d492218a98, State: Initialized, Role: FOLLOWER
I20240503 08:14:16.113962 10043 consensus_queue.cc:260] T e30557ffd4594f7d9cda13025e4c5e60 P b00f244c69df401997ea99d492218a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:14:16.116021 10043 ts_tablet_manager.cc:1398] T e30557ffd4594f7d9cda13025e4c5e60 P b00f244c69df401997ea99d492218a98: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:14:16.118352 30783 tablet_copy_service.cc:342] P 6d1d7de19459435fbde1999e3dfd226b: Request end of tablet copy session b00f244c69df401997ea99d492218a98-e30557ffd4594f7d9cda13025e4c5e60 received from {username='slave'} at 127.20.230.72:52827
I20240503 08:14:16.118517 30783 tablet_copy_service.cc:433] P 6d1d7de19459435fbde1999e3dfd226b: ending tablet copy session b00f244c69df401997ea99d492218a98-e30557ffd4594f7d9cda13025e4c5e60 on tablet e30557ffd4594f7d9cda13025e4c5e60 with peer b00f244c69df401997ea99d492218a98
I20240503 08:14:16.131723 10053 tablet_bootstrap.cc:492] T 71fa62206f744de0b97e3aa9ea753e37 P ce1496a47d084365ad786cd09e472a2c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:16.132160 10053 tablet_bootstrap.cc:492] T 71fa62206f744de0b97e3aa9ea753e37 P ce1496a47d084365ad786cd09e472a2c: Bootstrap complete.
I20240503 08:14:16.132400 10053 ts_tablet_manager.cc:1367] T 71fa62206f744de0b97e3aa9ea753e37 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.002s
I20240503 08:14:16.133050 10053 raft_consensus.cc:348] T 71fa62206f744de0b97e3aa9ea753e37 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:14:16.133277 10053 raft_consensus.cc:729] T 71fa62206f744de0b97e3aa9ea753e37 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:14:16.133495 10053 consensus_queue.cc:260] T 71fa62206f744de0b97e3aa9ea753e37 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:14:16.136116 10053 ts_tablet_manager.cc:1398] T 71fa62206f744de0b97e3aa9ea753e37 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 08:14:16.137007 30783 tablet_copy_service.cc:342] P 6d1d7de19459435fbde1999e3dfd226b: Request end of tablet copy session ce1496a47d084365ad786cd09e472a2c-71fa62206f744de0b97e3aa9ea753e37 received from {username='slave'} at 127.20.230.70:41691
I20240503 08:14:16.137142 30783 tablet_copy_service.cc:433] P 6d1d7de19459435fbde1999e3dfd226b: ending tablet copy session ce1496a47d084365ad786cd09e472a2c-71fa62206f744de0b97e3aa9ea753e37 on tablet 71fa62206f744de0b97e3aa9ea753e37 with peer ce1496a47d084365ad786cd09e472a2c
I20240503 08:14:16.210923 28000 raft_consensus.cc:1203] T 2401f8f101f94189aa3c0582f08cef88 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:14:16.272948 28260 raft_consensus.cc:1203] T 03b02a4425784a30b1007b3a14139d8d P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:14:16.294279 28000 raft_consensus.cc:1203] T 644bb089a4064ecfb88438d8b859bf01 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:14:16.326074 28049 maintenance_manager.cc:419] P 833582ff68c94269be364ebd187a75db: Scheduling FlushMRSOp(099e37089b2e42b4839e681f8f37365d): perf score=0.033346
I20240503 08:14:16.441720 27940 maintenance_manager.cc:643] P 833582ff68c94269be364ebd187a75db: FlushMRSOp(099e37089b2e42b4839e681f8f37365d) complete. Timing: real 0.115s user 0.067s sys 0.005s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":6096,"dirs.run_cpu_time_us":181,"dirs.run_wall_time_us":3605,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1497,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":9346,"threads_started":2,"wal-append.queue_time_us":3451}
I20240503 08:14:16.444643 28049 maintenance_manager.cc:419] P 833582ff68c94269be364ebd187a75db: Scheduling UndoDeltaBlockGCOp(099e37089b2e42b4839e681f8f37365d): 4838 bytes on disk
I20240503 08:14:16.448820 28179 maintenance_manager.cc:419] P 1b411c6bb25549368041773d7b710d48: Scheduling FlushMRSOp(bab5c25c31a64b578489b1006f631cc0): perf score=0.033386
I20240503 08:14:16.445272 27940 maintenance_manager.cc:643] P 833582ff68c94269be364ebd187a75db: UndoDeltaBlockGCOp(099e37089b2e42b4839e681f8f37365d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
I20240503 08:14:16.456005 28439 maintenance_manager.cc:419] P d09a11a03989421b9f28019dc1c14cc2: Scheduling FlushMRSOp(bab5c25c31a64b578489b1006f631cc0): perf score=0.033382
I20240503 08:14:16.494566 28309 maintenance_manager.cc:419] P aae860d7ae2b4002bf7794c278f9cc56: Scheduling FlushMRSOp(bab5c25c31a64b578489b1006f631cc0): perf score=0.033395
I20240503 08:14:16.521642 31161 raft_consensus.cc:1203] T e30557ffd4594f7d9cda13025e4c5e60 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:14:16.556267 30893 raft_consensus.cc:1203] T 71fa62206f744de0b97e3aa9ea753e37 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:14:16.566867 28070 maintenance_manager.cc:643] P 1b411c6bb25549368041773d7b710d48: FlushMRSOp(bab5c25c31a64b578489b1006f631cc0) complete. Timing: real 0.118s user 0.048s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":3498,"dirs.run_cpu_time_us":215,"dirs.run_wall_time_us":8808,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":827,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":12969,"threads_started":2,"wal-append.queue_time_us":9468}
I20240503 08:14:16.567734 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:16.570741 28179 maintenance_manager.cc:419] P 1b411c6bb25549368041773d7b710d48: Scheduling UndoDeltaBlockGCOp(bab5c25c31a64b578489b1006f631cc0): 4923 bytes on disk
I20240503 08:14:16.571743 28070 maintenance_manager.cc:643] P 1b411c6bb25549368041773d7b710d48: UndoDeltaBlockGCOp(bab5c25c31a64b578489b1006f631cc0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240503 08:14:16.574542 28179 maintenance_manager.cc:419] P 1b411c6bb25549368041773d7b710d48: Scheduling FlushMRSOp(099e37089b2e42b4839e681f8f37365d): perf score=0.033414
I20240503 08:14:16.608089 28330 maintenance_manager.cc:643] P d09a11a03989421b9f28019dc1c14cc2: FlushMRSOp(bab5c25c31a64b578489b1006f631cc0) complete. Timing: real 0.152s user 0.074s sys 0.045s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":54011,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":17005,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":894,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":86035,"threads_started":2,"wal-append.queue_time_us":32185}
I20240503 08:14:16.611708 28439 maintenance_manager.cc:419] P d09a11a03989421b9f28019dc1c14cc2: Scheduling UndoDeltaBlockGCOp(bab5c25c31a64b578489b1006f631cc0): 4923 bytes on disk
I20240503 08:14:16.612273 28330 maintenance_manager.cc:643] P d09a11a03989421b9f28019dc1c14cc2: UndoDeltaBlockGCOp(bab5c25c31a64b578489b1006f631cc0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20240503 08:14:16.616046 28439 maintenance_manager.cc:419] P d09a11a03989421b9f28019dc1c14cc2: Scheduling FlushMRSOp(ba1fc93781714efb8eb7ec67c2fceb8a): perf score=0.033422
I20240503 08:14:16.678485 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:16.710404 28070 maintenance_manager.cc:643] P 1b411c6bb25549368041773d7b710d48: FlushMRSOp(099e37089b2e42b4839e681f8f37365d) complete. Timing: real 0.136s user 0.042s sys 0.006s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":35,"dirs.run_cpu_time_us":162,"dirs.run_wall_time_us":1399,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":910,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":6301,"threads_started":1,"wal-append.queue_time_us":6198}
I20240503 08:14:16.713587 28179 maintenance_manager.cc:419] P 1b411c6bb25549368041773d7b710d48: Scheduling FlushMRSOp(ba1fc93781714efb8eb7ec67c2fceb8a): perf score=0.033449
I20240503 08:14:16.726730 28330 maintenance_manager.cc:643] P d09a11a03989421b9f28019dc1c14cc2: FlushMRSOp(ba1fc93781714efb8eb7ec67c2fceb8a) complete. Timing: real 0.110s user 0.064s sys 0.035s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":747,"dirs.run_cpu_time_us":203,"dirs.run_wall_time_us":7997,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":829,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":44081,"threads_started":1,"wal-append.queue_time_us":44161}
I20240503 08:14:16.729980 28439 maintenance_manager.cc:419] P d09a11a03989421b9f28019dc1c14cc2: Scheduling UndoDeltaBlockGCOp(ba1fc93781714efb8eb7ec67c2fceb8a): 4583 bytes on disk
I20240503 08:14:16.730664 28330 maintenance_manager.cc:643] P d09a11a03989421b9f28019dc1c14cc2: UndoDeltaBlockGCOp(ba1fc93781714efb8eb7ec67c2fceb8a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20240503 08:14:16.780014 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:16.798302 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:14:16.806100 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:16.818168 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 08:14:16.818894 28199 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.68:46211, user_credentials={real_user=slave}} blocked reactor thread for 57145.3us
I20240503 08:14:16.849006 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240503 08:14:16.877259 28199 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.65:39081, user_credentials={real_user=slave}} blocked reactor thread for 57860.9us
I20240503 08:14:16.891499 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:16.892239 28200 maintenance_manager.cc:643] P aae860d7ae2b4002bf7794c278f9cc56: FlushMRSOp(bab5c25c31a64b578489b1006f631cc0) complete. Timing: real 0.391s user 0.056s sys 0.034s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":158599,"dirs.run_cpu_time_us":212,"dirs.run_wall_time_us":26123,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":877,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":5655936,"thread_start_us":180306,"threads_started":2,"wal-append.queue_time_us":368}
I20240503 08:14:16.895638 28309 maintenance_manager.cc:419] P aae860d7ae2b4002bf7794c278f9cc56: Scheduling FlushMRSOp(ba1fc93781714efb8eb7ec67c2fceb8a): perf score=0.033501
I20240503 08:14:16.959565 28070 maintenance_manager.cc:643] P 1b411c6bb25549368041773d7b710d48: FlushMRSOp(ba1fc93781714efb8eb7ec67c2fceb8a) complete. Timing: real 0.246s user 0.075s sys 0.061s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":41,"dirs.run_cpu_time_us":167,"dirs.run_wall_time_us":1215,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5893,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":114543,"threads_started":1,"wal-append.queue_time_us":114628}
I20240503 08:14:16.962507 28179 maintenance_manager.cc:419] P 1b411c6bb25549368041773d7b710d48: Scheduling UndoDeltaBlockGCOp(099e37089b2e42b4839e681f8f37365d): 4838 bytes on disk
I20240503 08:14:16.963798 28070 maintenance_manager.cc:643] P 1b411c6bb25549368041773d7b710d48: UndoDeltaBlockGCOp(099e37089b2e42b4839e681f8f37365d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240503 08:14:16.966282 28179 maintenance_manager.cc:419] P 1b411c6bb25549368041773d7b710d48: Scheduling UndoDeltaBlockGCOp(ba1fc93781714efb8eb7ec67c2fceb8a): 4583 bytes on disk
I20240503 08:14:16.967250 28070 maintenance_manager.cc:643] P 1b411c6bb25549368041773d7b710d48: UndoDeltaBlockGCOp(ba1fc93781714efb8eb7ec67c2fceb8a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20240503 08:14:17.024123 28200 maintenance_manager.cc:643] P aae860d7ae2b4002bf7794c278f9cc56: FlushMRSOp(ba1fc93781714efb8eb7ec67c2fceb8a) complete. Timing: real 0.126s user 0.037s sys 0.007s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":132,"dirs.run_cpu_time_us":153,"dirs.run_wall_time_us":2081,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":818,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":234,"threads_started":1,"wal-append.queue_time_us":347}
I20240503 08:14:17.027446 28309 maintenance_manager.cc:419] P aae860d7ae2b4002bf7794c278f9cc56: Scheduling FlushMRSOp(099e37089b2e42b4839e681f8f37365d): perf score=0.033535
I20240503 08:14:17.089354 30761 consensus_queue.cc:237] T e30557ffd4594f7d9cda13025e4c5e60 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:14:17.093436 28200 maintenance_manager.cc:643] P aae860d7ae2b4002bf7794c278f9cc56: FlushMRSOp(099e37089b2e42b4839e681f8f37365d) complete. Timing: real 0.066s user 0.056s sys 0.002s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":72,"dirs.run_cpu_time_us":262,"dirs.run_wall_time_us":1468,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1369,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":257,"threads_started":1,"wal-append.queue_time_us":358}
I20240503 08:14:17.094823 31159 raft_consensus.cc:1261] T e30557ffd4594f7d9cda13025e4c5e60 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:17.095074 31026 raft_consensus.cc:1261] T e30557ffd4594f7d9cda13025e4c5e60 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:17.095477 10052 consensus_queue.cc:1035] T e30557ffd4594f7d9cda13025e4c5e60 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:17.095755 9752 consensus_queue.cc:1035] T e30557ffd4594f7d9cda13025e4c5e60 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:17.096700 28309 maintenance_manager.cc:419] P aae860d7ae2b4002bf7794c278f9cc56: Scheduling UndoDeltaBlockGCOp(bab5c25c31a64b578489b1006f631cc0): 4923 bytes on disk
I20240503 08:14:17.098120 28200 maintenance_manager.cc:643] P aae860d7ae2b4002bf7794c278f9cc56: UndoDeltaBlockGCOp(bab5c25c31a64b578489b1006f631cc0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
I20240503 08:14:17.098557 9752 raft_consensus.cc:2932] T e30557ffd4594f7d9cda13025e4c5e60 P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:14:17.100327 31026 raft_consensus.cc:2932] T e30557ffd4594f7d9cda13025e4c5e60 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:14:17.101384 28309 maintenance_manager.cc:419] P aae860d7ae2b4002bf7794c278f9cc56: Scheduling UndoDeltaBlockGCOp(099e37089b2e42b4839e681f8f37365d): 4838 bytes on disk
I20240503 08:14:17.103909 28200 maintenance_manager.cc:643] P aae860d7ae2b4002bf7794c278f9cc56: UndoDeltaBlockGCOp(099e37089b2e42b4839e681f8f37365d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
I20240503 08:14:17.106776 28309 maintenance_manager.cc:419] P aae860d7ae2b4002bf7794c278f9cc56: Scheduling UndoDeltaBlockGCOp(ba1fc93781714efb8eb7ec67c2fceb8a): 4583 bytes on disk
I20240503 08:14:17.108368 28200 maintenance_manager.cc:643] P aae860d7ae2b4002bf7794c278f9cc56: UndoDeltaBlockGCOp(ba1fc93781714efb8eb7ec67c2fceb8a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20240503 08:14:17.108973 27855 catalog_manager.cc:5484] T e30557ffd4594f7d9cda13025e4c5e60 P ced21ca983b749d0b4d4f6a0e2426bf8 reported cstate change: config changed from index 2 to 3, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) evicted. New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:14:17.111018 31159 raft_consensus.cc:2932] T e30557ffd4594f7d9cda13025e4c5e60 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:14:17.126142 27983 tablet_service.cc:1514] Processing DeleteTablet for tablet e30557ffd4594f7d9cda13025e4c5e60 with delete_type TABLET_DATA_TOMBSTONED (TS 833582ff68c94269be364ebd187a75db not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:55610
I20240503 08:14:17.135056 10170 tablet_replica.cc:330] T e30557ffd4594f7d9cda13025e4c5e60 P 833582ff68c94269be364ebd187a75db: stopping tablet replica
I20240503 08:14:17.135306 10170 raft_consensus.cc:2229] T e30557ffd4594f7d9cda13025e4c5e60 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:17.136891 10170 raft_consensus.cc:2258] T e30557ffd4594f7d9cda13025e4c5e60 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:17.137974 10170 ts_tablet_manager.cc:1875] T e30557ffd4594f7d9cda13025e4c5e60 P 833582ff68c94269be364ebd187a75db: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:17.147214 10170 ts_tablet_manager.cc:1888] T e30557ffd4594f7d9cda13025e4c5e60 P 833582ff68c94269be364ebd187a75db: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:14:17.147426 10170 log.cc:1196] T e30557ffd4594f7d9cda13025e4c5e60 P 833582ff68c94269be364ebd187a75db: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/wals/e30557ffd4594f7d9cda13025e4c5e60
I20240503 08:14:17.148372 27849 catalog_manager.cc:4830] TS 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): tablet e30557ffd4594f7d9cda13025e4c5e60 (table rebalancer_extra_table_62 [id=7adc395c24f24d63aa9f9ceed2c44c01]) successfully deleted
I20240503 08:14:17.434029 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:17.455078 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:17.508389 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:17.521353 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:14:17.537700 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:17.551831 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:14:17.609913 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:17.615304 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:17.763239 28390 consensus_queue.cc:237] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:17.764808 28129 raft_consensus.cc:1261] T 03b02a4425784a30b1007b3a14139d8d P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:17.764863 28262 raft_consensus.cc:1261] T 03b02a4425784a30b1007b3a14139d8d P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:17.765251 9152 consensus_queue.cc:1035] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:17.765511 10227 consensus_queue.cc:1035] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:17.767889 10227 raft_consensus.cc:2932] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:17.768293 28262 raft_consensus.cc:2932] T 03b02a4425784a30b1007b3a14139d8d P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:17.772398 28129 raft_consensus.cc:2932] T 03b02a4425784a30b1007b3a14139d8d P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:17.775074 27861 catalog_manager.cc:5484] T 03b02a4425784a30b1007b3a14139d8d P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:17.796221 31006 tablet_service.cc:1514] Processing DeleteTablet for tablet 03b02a4425784a30b1007b3a14139d8d with delete_type TABLET_DATA_TOMBSTONED (TS ced21ca983b749d0b4d4f6a0e2426bf8 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47020
I20240503 08:14:17.796765 10255 tablet_replica.cc:330] T 03b02a4425784a30b1007b3a14139d8d P ced21ca983b749d0b4d4f6a0e2426bf8: stopping tablet replica
I20240503 08:14:17.796977 10255 raft_consensus.cc:2229] T 03b02a4425784a30b1007b3a14139d8d P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:17.797233 10255 raft_consensus.cc:2258] T 03b02a4425784a30b1007b3a14139d8d P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:17.798172 10255 ts_tablet_manager.cc:1875] T 03b02a4425784a30b1007b3a14139d8d P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:17.803769 10255 ts_tablet_manager.cc:1888] T 03b02a4425784a30b1007b3a14139d8d P ced21ca983b749d0b4d4f6a0e2426bf8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:14:17.803870 10255 log.cc:1196] T 03b02a4425784a30b1007b3a14139d8d P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal/wals/03b02a4425784a30b1007b3a14139d8d
I20240503 08:14:17.804733 27847 catalog_manager.cc:4830] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): tablet 03b02a4425784a30b1007b3a14139d8d (table rebalancer_extra_table_60 [id=2979b3f65625430db5d8039280a09b13]) successfully deleted
I20240503 08:14:18.062597 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:18.067164 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:18.077082 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:18.113827 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:14:18.117022 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:18.140992 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:14:18.146346 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:18.164554 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:18.310428 31159 consensus_queue.cc:237] T 644bb089a4064ecfb88438d8b859bf01 P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:14:18.312922 27999 raft_consensus.cc:1261] T 644bb089a4064ecfb88438d8b859bf01 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:18.313124 31026 raft_consensus.cc:1261] T 644bb089a4064ecfb88438d8b859bf01 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:18.313489 10051 consensus_queue.cc:1035] T 644bb089a4064ecfb88438d8b859bf01 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:18.313804 9564 consensus_queue.cc:1035] T 644bb089a4064ecfb88438d8b859bf01 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:18.316036 10051 raft_consensus.cc:2932] T 644bb089a4064ecfb88438d8b859bf01 P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:14:18.316658 27999 raft_consensus.cc:2932] T 644bb089a4064ecfb88438d8b859bf01 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:14:18.316834 31026 raft_consensus.cc:2932] T 644bb089a4064ecfb88438d8b859bf01 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:14:18.321805 27856 catalog_manager.cc:5484] T 644bb089a4064ecfb88438d8b859bf01 P b00f244c69df401997ea99d492218a98 reported cstate change: config changed from index 2 to 3, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) evicted. New cstate: current_term: 1 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:14:18.331105 28373 tablet_service.cc:1514] Processing DeleteTablet for tablet 644bb089a4064ecfb88438d8b859bf01 with delete_type TABLET_DATA_TOMBSTONED (TS d09a11a03989421b9f28019dc1c14cc2 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:34786
I20240503 08:14:18.331709 10339 tablet_replica.cc:330] T 644bb089a4064ecfb88438d8b859bf01 P d09a11a03989421b9f28019dc1c14cc2: stopping tablet replica
I20240503 08:14:18.332046 10339 raft_consensus.cc:2229] T 644bb089a4064ecfb88438d8b859bf01 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:18.332304 10339 raft_consensus.cc:2258] T 644bb089a4064ecfb88438d8b859bf01 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:18.333052 10339 ts_tablet_manager.cc:1875] T 644bb089a4064ecfb88438d8b859bf01 P d09a11a03989421b9f28019dc1c14cc2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:18.338675 10339 ts_tablet_manager.cc:1888] T 644bb089a4064ecfb88438d8b859bf01 P d09a11a03989421b9f28019dc1c14cc2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:14:18.338794 10339 log.cc:1196] T 644bb089a4064ecfb88438d8b859bf01 P d09a11a03989421b9f28019dc1c14cc2: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/wals/644bb089a4064ecfb88438d8b859bf01
I20240503 08:14:18.339434 27849 catalog_manager.cc:4830] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): tablet 644bb089a4064ecfb88438d8b859bf01 (table rebalancer_extra_table_52 [id=40dc8c7f01804911a18e5c62a4b6be48]) successfully deleted
I20240503 08:14:18.591025 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:18.613368 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:18.653725 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:18.682093 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:18.729683 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:14:18.752440 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:18.763724 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:14:18.803171 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:18.927387 28260 consensus_queue.cc:237] T 8c714f29abb140379b57e08eb927e2b7 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:14:18.929244 30894 raft_consensus.cc:1261] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:18.929327 28129 raft_consensus.cc:1261] T 8c714f29abb140379b57e08eb927e2b7 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer aae860d7ae2b4002bf7794c278f9cc56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:18.929849 10312 consensus_queue.cc:1035] T 8c714f29abb140379b57e08eb927e2b7 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:18.930094 10133 consensus_queue.cc:1035] T 8c714f29abb140379b57e08eb927e2b7 P aae860d7ae2b4002bf7794c278f9cc56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:18.932396 10312 raft_consensus.cc:2932] T 8c714f29abb140379b57e08eb927e2b7 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:14:18.933005 30894 raft_consensus.cc:2932] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:14:18.933955 28129 raft_consensus.cc:2932] T 8c714f29abb140379b57e08eb927e2b7 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:14:18.939191 27860 catalog_manager.cc:5484] T 8c714f29abb140379b57e08eb927e2b7 P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New cstate: current_term: 1 leader_uuid: "aae860d7ae2b4002bf7794c278f9cc56" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:14:18.948529 31141 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c714f29abb140379b57e08eb927e2b7 with delete_type TABLET_DATA_TOMBSTONED (TS b00f244c69df401997ea99d492218a98 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45814
I20240503 08:14:18.958216 10414 tablet_replica.cc:330] T 8c714f29abb140379b57e08eb927e2b7 P b00f244c69df401997ea99d492218a98: stopping tablet replica
I20240503 08:14:18.958429 10414 raft_consensus.cc:2229] T 8c714f29abb140379b57e08eb927e2b7 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:18.958638 10414 raft_consensus.cc:2258] T 8c714f29abb140379b57e08eb927e2b7 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:18.959697 10414 ts_tablet_manager.cc:1875] T 8c714f29abb140379b57e08eb927e2b7 P b00f244c69df401997ea99d492218a98: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:18.965801 10414 ts_tablet_manager.cc:1888] T 8c714f29abb140379b57e08eb927e2b7 P b00f244c69df401997ea99d492218a98: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:14:18.965917 10414 log.cc:1196] T 8c714f29abb140379b57e08eb927e2b7 P b00f244c69df401997ea99d492218a98: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal/wals/8c714f29abb140379b57e08eb927e2b7
I20240503 08:14:18.966872 27846 catalog_manager.cc:4830] TS b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): tablet 8c714f29abb140379b57e08eb927e2b7 (table rebalancer_extra_table_4 [id=1c7e1d7fc1be4aa4883c657bf13f0b9f]) successfully deleted
I20240503 08:14:19.372704 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:19.384657 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:19.410248 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:19.424458 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:19.435169 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:19.438737 31141 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240503 08:14:19.472759 28067 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.67:39919, user_credentials={real_user=slave}} blocked reactor thread for 72464.8us
I20240503 08:14:19.481316 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:14:19.536559 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:19.700016 30893 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2401f8f101f94189aa3c0582f08cef88"
dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37126
I20240503 08:14:19.700210 30893 raft_consensus.cc:595] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Received request to transfer leadership
I20240503 08:14:19.707840 10106 raft_consensus.cc:979] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c: : Instructing follower b00f244c69df401997ea99d492218a98 to start an election
I20240503 08:14:19.707994 10530 raft_consensus.cc:1067] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Signalling peer b00f244c69df401997ea99d492218a98 to start an election
I20240503 08:14:19.708599 31159 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2401f8f101f94189aa3c0582f08cef88"
dest_uuid: "b00f244c69df401997ea99d492218a98"
from {username='slave'} at 127.20.230.70:40787
I20240503 08:14:19.708753 31159 raft_consensus.cc:482] T 2401f8f101f94189aa3c0582f08cef88 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:14:19.708822 31159 raft_consensus.cc:3037] T 2401f8f101f94189aa3c0582f08cef88 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:19.711483 31159 raft_consensus.cc:504] T 2401f8f101f94189aa3c0582f08cef88 P b00f244c69df401997ea99d492218a98 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:14:19.712577 31026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401f8f101f94189aa3c0582f08cef88" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:14:19.712780 31026 raft_consensus.cc:3037] T 2401f8f101f94189aa3c0582f08cef88 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:19.713071 27999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401f8f101f94189aa3c0582f08cef88" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:14:19.713235 27999 raft_consensus.cc:3037] T 2401f8f101f94189aa3c0582f08cef88 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:19.715550 31026 raft_consensus.cc:2454] T 2401f8f101f94189aa3c0582f08cef88 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 2.
I20240503 08:14:19.715859 27999 raft_consensus.cc:2454] T 2401f8f101f94189aa3c0582f08cef88 P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 2.
I20240503 08:14:19.716353 31097 leader_election.cc:302] T 2401f8f101f94189aa3c0582f08cef88 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, b00f244c69df401997ea99d492218a98, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:14:19.716591 9564 raft_consensus.cc:2781] T 2401f8f101f94189aa3c0582f08cef88 P b00f244c69df401997ea99d492218a98 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:14:19.716683 9564 raft_consensus.cc:686] T 2401f8f101f94189aa3c0582f08cef88 P b00f244c69df401997ea99d492218a98 [term 2 LEADER]: Becoming Leader. State: Replica: b00f244c69df401997ea99d492218a98, State: Running, Role: LEADER
I20240503 08:14:19.716893 9564 consensus_queue.cc:237] T 2401f8f101f94189aa3c0582f08cef88 P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:14:19.718858 31159 leader_election.cc:288] T 2401f8f101f94189aa3c0582f08cef88 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 election: Requested vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:14:19.721349 30893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401f8f101f94189aa3c0582f08cef88" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:14:19.721536 30893 raft_consensus.cc:3032] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:14:19.721593 30893 raft_consensus.cc:729] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Running, Role: LEADER
I20240503 08:14:19.721832 30893 consensus_queue.cc:260] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:14:19.722250 30893 raft_consensus.cc:3037] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:19.722424 27854 catalog_manager.cc:5484] T 2401f8f101f94189aa3c0582f08cef88 P b00f244c69df401997ea99d492218a98 reported cstate change: term changed from 1 to 2, leader changed from ce1496a47d084365ad786cd09e472a2c (127.20.230.70) to b00f244c69df401997ea99d492218a98 (127.20.230.72). New cstate: current_term: 2 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:14:19.726769 30893 raft_consensus.cc:2454] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 2.
I20240503 08:14:19.990751 28049 maintenance_manager.cc:419] P 833582ff68c94269be364ebd187a75db: Scheduling FlushMRSOp(df90a0d9d338416b81e253fa23542cca): perf score=0.033334
I20240503 08:14:20.023228 28439 maintenance_manager.cc:419] P d09a11a03989421b9f28019dc1c14cc2: Scheduling FlushMRSOp(f46393257ce9471891698c36a07546ad): perf score=0.033336
I20240503 08:14:20.116098 27940 maintenance_manager.cc:643] P 833582ff68c94269be364ebd187a75db: FlushMRSOp(df90a0d9d338416b81e253fa23542cca) complete. Timing: real 0.125s user 0.041s sys 0.029s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":330,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":1797,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":747,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":29473,"threads_started":2,"wal-append.queue_time_us":29288}
I20240503 08:14:20.119298 28049 maintenance_manager.cc:419] P 833582ff68c94269be364ebd187a75db: Scheduling UndoDeltaBlockGCOp(df90a0d9d338416b81e253fa23542cca): 4923 bytes on disk
I20240503 08:14:20.120105 27940 maintenance_manager.cc:643] P 833582ff68c94269be364ebd187a75db: UndoDeltaBlockGCOp(df90a0d9d338416b81e253fa23542cca) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240503 08:14:20.123095 28049 maintenance_manager.cc:419] P 833582ff68c94269be364ebd187a75db: Scheduling FlushMRSOp(8cc6ae30aa60495d8a4364a11d5bef79): perf score=0.033366
I20240503 08:14:20.132575 31026 raft_consensus.cc:1261] T 2401f8f101f94189aa3c0582f08cef88 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:14:20.133793 10051 consensus_queue.cc:1035] T 2401f8f101f94189aa3c0582f08cef88 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:20.146502 28309 maintenance_manager.cc:419] P aae860d7ae2b4002bf7794c278f9cc56: Scheduling FlushMRSOp(8cc6ae30aa60495d8a4364a11d5bef79): perf score=0.033377
I20240503 08:14:20.212391 27940 maintenance_manager.cc:643] P 833582ff68c94269be364ebd187a75db: FlushMRSOp(8cc6ae30aa60495d8a4364a11d5bef79) complete. Timing: real 0.089s user 0.026s sys 0.016s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":3091,"dirs.run_cpu_time_us":158,"dirs.run_wall_time_us":1299,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":893,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":9540,"threads_started":1,"wal-append.queue_time_us":5550}
I20240503 08:14:20.215306 28049 maintenance_manager.cc:419] P 833582ff68c94269be364ebd187a75db: Scheduling UndoDeltaBlockGCOp(8cc6ae30aa60495d8a4364a11d5bef79): 4583 bytes on disk
I20240503 08:14:20.215934 27940 maintenance_manager.cc:643] P 833582ff68c94269be364ebd187a75db: UndoDeltaBlockGCOp(8cc6ae30aa60495d8a4364a11d5bef79) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
I20240503 08:14:20.249306 28330 maintenance_manager.cc:643] P d09a11a03989421b9f28019dc1c14cc2: FlushMRSOp(f46393257ce9471891698c36a07546ad) complete. Timing: real 0.226s user 0.075s sys 0.047s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":14333,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":2282,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1191,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":109827,"threads_started":2,"wal-append.queue_time_us":101405}
I20240503 08:14:20.249645 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:20.252441 28439 maintenance_manager.cc:419] P d09a11a03989421b9f28019dc1c14cc2: Scheduling UndoDeltaBlockGCOp(f46393257ce9471891698c36a07546ad): 4838 bytes on disk
I20240503 08:14:20.253028 28330 maintenance_manager.cc:643] P d09a11a03989421b9f28019dc1c14cc2: UndoDeltaBlockGCOp(f46393257ce9471891698c36a07546ad) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
I20240503 08:14:20.255201 28179 maintenance_manager.cc:419] P 1b411c6bb25549368041773d7b710d48: Scheduling FlushMRSOp(df90a0d9d338416b81e253fa23542cca): perf score=0.033399
I20240503 08:14:20.279568 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:20.347921 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:20.352578 28003 raft_consensus.cc:1261] T 2401f8f101f94189aa3c0582f08cef88 P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:14:20.357231 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:20.359289 30894 raft_consensus.cc:1261] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:14:20.359885 10051 consensus_queue.cc:1035] T 2401f8f101f94189aa3c0582f08cef88 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:20.384013 10051 consensus_queue.cc:1035] T 2401f8f101f94189aa3c0582f08cef88 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:20.410290 31141 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:14:20.432366 28200 tablet_replica.cc:1402] Time spent flushing the Log queue took a long time: real 0.212s user 0.040s sys 0.065s
I20240503 08:14:20.436707 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:20.438270 28200 maintenance_manager.cc:643] P aae860d7ae2b4002bf7794c278f9cc56: FlushMRSOp(8cc6ae30aa60495d8a4364a11d5bef79) complete. Timing: real 0.291s user 0.066s sys 0.079s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":7711,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":7858,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":939,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":171392,"thread_start_us":218130,"threads_started":2,"wal-append.queue_time_us":210224}
I20240503 08:14:20.444185 28309 maintenance_manager.cc:419] P aae860d7ae2b4002bf7794c278f9cc56: Scheduling FlushMRSOp(f46393257ce9471891698c36a07546ad): perf score=0.033453
I20240503 08:14:20.450529 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:20.460630 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:14:20.462047 28070 maintenance_manager.cc:643] P 1b411c6bb25549368041773d7b710d48: FlushMRSOp(df90a0d9d338416b81e253fa23542cca) complete. Timing: real 0.207s user 0.068s sys 0.026s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":13768,"dirs.run_cpu_time_us":190,"dirs.run_wall_time_us":4042,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":779,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":3840,"thread_start_us":89946,"threads_started":2,"wal-append.queue_time_us":71018}
I20240503 08:14:20.467017 28179 maintenance_manager.cc:419] P 1b411c6bb25549368041773d7b710d48: Scheduling UndoDeltaBlockGCOp(df90a0d9d338416b81e253fa23542cca): 4923 bytes on disk
I20240503 08:14:20.468659 28070 maintenance_manager.cc:643] P 1b411c6bb25549368041773d7b710d48: UndoDeltaBlockGCOp(df90a0d9d338416b81e253fa23542cca) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20240503 08:14:20.611071 28200 maintenance_manager.cc:643] P aae860d7ae2b4002bf7794c278f9cc56: FlushMRSOp(f46393257ce9471891698c36a07546ad) complete. Timing: real 0.160s user 0.044s sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":37,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":1392,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":752,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":9423,"threads_started":1,"wal-append.queue_time_us":9518}
I20240503 08:14:20.613956 28309 maintenance_manager.cc:419] P aae860d7ae2b4002bf7794c278f9cc56: Scheduling UndoDeltaBlockGCOp(f46393257ce9471891698c36a07546ad): 4838 bytes on disk
I20240503 08:14:20.614565 28200 maintenance_manager.cc:643] P aae860d7ae2b4002bf7794c278f9cc56: UndoDeltaBlockGCOp(f46393257ce9471891698c36a07546ad) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240503 08:14:20.618559 28309 maintenance_manager.cc:419] P aae860d7ae2b4002bf7794c278f9cc56: Scheduling UndoDeltaBlockGCOp(8cc6ae30aa60495d8a4364a11d5bef79): 4583 bytes on disk
I20240503 08:14:20.619053 28200 maintenance_manager.cc:643] P aae860d7ae2b4002bf7794c278f9cc56: UndoDeltaBlockGCOp(8cc6ae30aa60495d8a4364a11d5bef79) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
I20240503 08:14:20.622859 28309 maintenance_manager.cc:419] P aae860d7ae2b4002bf7794c278f9cc56: Scheduling FlushMRSOp(df90a0d9d338416b81e253fa23542cca): perf score=0.033498
I20240503 08:14:20.727882 28200 maintenance_manager.cc:643] P aae860d7ae2b4002bf7794c278f9cc56: FlushMRSOp(df90a0d9d338416b81e253fa23542cca) complete. Timing: real 0.105s user 0.066s sys 0.023s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":41,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":1624,"drs_written":1,"lbm_read_time_us":177,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1141,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":32071,"threads_started":1,"wal-append.queue_time_us":32166}
I20240503 08:14:20.731703 28309 maintenance_manager.cc:419] P aae860d7ae2b4002bf7794c278f9cc56: Scheduling UndoDeltaBlockGCOp(df90a0d9d338416b81e253fa23542cca): 4923 bytes on disk
I20240503 08:14:20.738241 28200 maintenance_manager.cc:643] P aae860d7ae2b4002bf7794c278f9cc56: UndoDeltaBlockGCOp(df90a0d9d338416b81e253fa23542cca) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
I20240503 08:14:20.791486 28003 consensus_queue.cc:237] T c68032ba399547e3bd272f424927fbce P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
W20240503 08:14:20.795073 27938 consensus_peers.cc:485] T c68032ba399547e3bd272f424927fbce P 833582ff68c94269be364ebd187a75db -> Peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): Couldn't send request to peer 6d1d7de19459435fbde1999e3dfd226b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c68032ba399547e3bd272f424927fbce. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:20.795424 28261 raft_consensus.cc:1261] T c68032ba399547e3bd272f424927fbce P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:20.795516 28133 raft_consensus.cc:1261] T c68032ba399547e3bd272f424927fbce P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:20.795996 10388 consensus_queue.cc:1035] T c68032ba399547e3bd272f424927fbce P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:20.796430 10388 consensus_queue.cc:1035] T c68032ba399547e3bd272f424927fbce P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:20.802179 10645 raft_consensus.cc:2932] T c68032ba399547e3bd272f424927fbce P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:14:20.802712 28261 raft_consensus.cc:2932] T c68032ba399547e3bd272f424927fbce P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:14:20.808699 28129 raft_consensus.cc:2932] T c68032ba399547e3bd272f424927fbce P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:14:20.808977 27860 catalog_manager.cc:5484] T c68032ba399547e3bd272f424927fbce P aae860d7ae2b4002bf7794c278f9cc56 reported cstate change: config changed from index -1 to 4, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) added. New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:14:20.852424 30761 consensus_queue.cc:237] T 2e511a3ab0ee4eabb22240de99690603 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:14:20.854709 28129 raft_consensus.cc:1261] T 2e511a3ab0ee4eabb22240de99690603 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:14:20.854884 30697 consensus_peers.cc:485] T 2e511a3ab0ee4eabb22240de99690603 P 6d1d7de19459435fbde1999e3dfd226b -> Peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): Couldn't send request to peer d09a11a03989421b9f28019dc1c14cc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e511a3ab0ee4eabb22240de99690603. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:20.855306 30894 raft_consensus.cc:1261] T 2e511a3ab0ee4eabb22240de99690603 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:20.855301 10052 consensus_queue.cc:1035] T 2e511a3ab0ee4eabb22240de99690603 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:20.855743 10531 consensus_queue.cc:1035] T 2e511a3ab0ee4eabb22240de99690603 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:20.858191 10052 raft_consensus.cc:2932] T 2e511a3ab0ee4eabb22240de99690603 P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:14:20.858775 30894 raft_consensus.cc:2932] T 2e511a3ab0ee4eabb22240de99690603 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:14:20.863029 28129 raft_consensus.cc:2932] T 2e511a3ab0ee4eabb22240de99690603 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:14:20.869099 27857 catalog_manager.cc:5484] T 2e511a3ab0ee4eabb22240de99690603 P ce1496a47d084365ad786cd09e472a2c reported cstate change: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:14:20.905179 28129 consensus_queue.cc:237] T fe4d26ee022b424195e836834bd2f051 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
W20240503 08:14:20.908216 28067 consensus_peers.cc:485] T fe4d26ee022b424195e836834bd2f051 P 1b411c6bb25549368041773d7b710d48 -> Peer ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): Couldn't send request to peer ced21ca983b749d0b4d4f6a0e2426bf8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe4d26ee022b424195e836834bd2f051. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:20.908207 28003 raft_consensus.cc:1261] T fe4d26ee022b424195e836834bd2f051 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:20.908535 28261 raft_consensus.cc:1261] T fe4d26ee022b424195e836834bd2f051 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:20.908972 10230 consensus_queue.cc:1035] T fe4d26ee022b424195e836834bd2f051 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:20.909255 10688 consensus_queue.cc:1035] T fe4d26ee022b424195e836834bd2f051 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:20.925640 10688 raft_consensus.cc:2932] T fe4d26ee022b424195e836834bd2f051 P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:14:20.926599 28003 raft_consensus.cc:2932] T fe4d26ee022b424195e836834bd2f051 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:14:20.931407 28261 raft_consensus.cc:2932] T fe4d26ee022b424195e836834bd2f051 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:14:20.935425 27857 catalog_manager.cc:5484] T fe4d26ee022b424195e836834bd2f051 P 833582ff68c94269be364ebd187a75db reported cstate change: config changed from index -1 to 4, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) added. New cstate: current_term: 1 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:14:20.972705 28129 consensus_queue.cc:237] T 7083414bc16c4d499c97b93ca4e3f631 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
W20240503 08:14:20.974969 28068 consensus_peers.cc:485] T 7083414bc16c4d499c97b93ca4e3f631 P 1b411c6bb25549368041773d7b710d48 -> Peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): Couldn't send request to peer 6d1d7de19459435fbde1999e3dfd226b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7083414bc16c4d499c97b93ca4e3f631. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:20.975067 28391 raft_consensus.cc:1261] T 7083414bc16c4d499c97b93ca4e3f631 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:20.975598 10688 consensus_queue.cc:1035] T 7083414bc16c4d499c97b93ca4e3f631 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:20.976279 28260 raft_consensus.cc:1261] T 7083414bc16c4d499c97b93ca4e3f631 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:20.976799 10688 consensus_queue.cc:1035] T 7083414bc16c4d499c97b93ca4e3f631 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:20.979652 10230 raft_consensus.cc:2932] T 7083414bc16c4d499c97b93ca4e3f631 P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:14:20.980149 28260 raft_consensus.cc:2932] T 7083414bc16c4d499c97b93ca4e3f631 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:14:20.982811 28391 raft_consensus.cc:2932] T 7083414bc16c4d499c97b93ca4e3f631 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:14:20.985121 27860 catalog_manager.cc:5484] T 7083414bc16c4d499c97b93ca4e3f631 P 1b411c6bb25549368041773d7b710d48 reported cstate change: config changed from index -1 to 4, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) added. New cstate: current_term: 1 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:14:21.053687 30761 consensus_queue.cc:237] T 6c2824aa878845e098f600f1fcfea087 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:14:21.056208 30894 raft_consensus.cc:1261] T 6c2824aa878845e098f600f1fcfea087 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:14:21.056262 30696 consensus_peers.cc:485] T 6c2824aa878845e098f600f1fcfea087 P 6d1d7de19459435fbde1999e3dfd226b -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c2824aa878845e098f600f1fcfea087. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:21.056344 31161 raft_consensus.cc:1261] T 6c2824aa878845e098f600f1fcfea087 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:21.056869 10052 consensus_queue.cc:1035] T 6c2824aa878845e098f600f1fcfea087 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:21.057185 10531 consensus_queue.cc:1035] T 6c2824aa878845e098f600f1fcfea087 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:21.062987 10052 raft_consensus.cc:2932] T 6c2824aa878845e098f600f1fcfea087 P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:14:21.066092 31161 raft_consensus.cc:2932] T 6c2824aa878845e098f600f1fcfea087 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:14:21.072851 30894 raft_consensus.cc:2932] T 6c2824aa878845e098f600f1fcfea087 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:14:21.074362 27857 catalog_manager.cc:5484] T 6c2824aa878845e098f600f1fcfea087 P b00f244c69df401997ea99d492218a98 reported cstate change: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:14:21.111713 28391 consensus_queue.cc:237] T 720f36100d794158be13ff02595e6ddf P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:14:21.114188 30761 raft_consensus.cc:1261] T 720f36100d794158be13ff02595e6ddf P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:21.114264 31161 raft_consensus.cc:1261] T 720f36100d794158be13ff02595e6ddf P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:14:21.114492 28328 consensus_peers.cc:485] T 720f36100d794158be13ff02595e6ddf P d09a11a03989421b9f28019dc1c14cc2 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 720f36100d794158be13ff02595e6ddf. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:21.114748 10227 consensus_queue.cc:1035] T 720f36100d794158be13ff02595e6ddf P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:21.116115 10227 consensus_queue.cc:1035] T 720f36100d794158be13ff02595e6ddf P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:21.119812 10573 raft_consensus.cc:2932] T 720f36100d794158be13ff02595e6ddf P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:14:21.122289 31161 raft_consensus.cc:2932] T 720f36100d794158be13ff02595e6ddf P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:14:21.124112 30761 raft_consensus.cc:2932] T 720f36100d794158be13ff02595e6ddf P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:14:21.128005 27853 catalog_manager.cc:5484] T 720f36100d794158be13ff02595e6ddf P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:14:21.178054 28003 consensus_queue.cc:237] T 45f47478f49149b895060791655503ae P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:14:21.181238 28129 raft_consensus.cc:1261] T 45f47478f49149b895060791655503ae P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:14:21.181281 27939 consensus_peers.cc:485] T 45f47478f49149b895060791655503ae P 833582ff68c94269be364ebd187a75db -> Peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): Couldn't send request to peer d09a11a03989421b9f28019dc1c14cc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 45f47478f49149b895060791655503ae. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:21.181706 10645 consensus_queue.cc:1035] T 45f47478f49149b895060791655503ae P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:21.182073 31027 raft_consensus.cc:1261] T 45f47478f49149b895060791655503ae P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:21.183106 10645 consensus_queue.cc:1035] T 45f47478f49149b895060791655503ae P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:21.185348 10646 raft_consensus.cc:2932] T 45f47478f49149b895060791655503ae P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:14:21.185861 31027 raft_consensus.cc:2932] T 45f47478f49149b895060791655503ae P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:14:21.191242 27853 catalog_manager.cc:5484] T 45f47478f49149b895060791655503ae P 833582ff68c94269be364ebd187a75db reported cstate change: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:14:21.195567 28129 raft_consensus.cc:2932] T 45f47478f49149b895060791655503ae P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:14:21.294355 30894 consensus_queue.cc:237] T d61ad2e11c4544bfbdef7df1e92d2d11 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:14:21.296927 28260 raft_consensus.cc:1261] T d61ad2e11c4544bfbdef7df1e92d2d11 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:21.296967 28391 raft_consensus.cc:1261] T d61ad2e11c4544bfbdef7df1e92d2d11 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:14:21.297398 30829 consensus_peers.cc:485] T d61ad2e11c4544bfbdef7df1e92d2d11 P ce1496a47d084365ad786cd09e472a2c -> Peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): Couldn't send request to peer 6d1d7de19459435fbde1999e3dfd226b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d61ad2e11c4544bfbdef7df1e92d2d11. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:21.297583 10787 consensus_queue.cc:1035] T d61ad2e11c4544bfbdef7df1e92d2d11 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:21.297983 10530 consensus_queue.cc:1035] T d61ad2e11c4544bfbdef7df1e92d2d11 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:21.301064 10530 raft_consensus.cc:2932] T d61ad2e11c4544bfbdef7df1e92d2d11 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:14:21.301611 28391 raft_consensus.cc:2932] T d61ad2e11c4544bfbdef7df1e92d2d11 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:14:21.308533 27860 catalog_manager.cc:5484] T d61ad2e11c4544bfbdef7df1e92d2d11 P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: config changed from index -1 to 2, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) added. New cstate: current_term: 1 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:14:21.309573 28260 raft_consensus.cc:2932] T d61ad2e11c4544bfbdef7df1e92d2d11 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:14:21.351135 28391 consensus_queue.cc:237] T 607a503b64d8440a81cf66ab158ae273 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:14:21.353997 28129 raft_consensus.cc:1261] T 607a503b64d8440a81cf66ab158ae273 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:14:21.354053 28329 consensus_peers.cc:485] T 607a503b64d8440a81cf66ab158ae273 P d09a11a03989421b9f28019dc1c14cc2 -> Peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Couldn't send request to peer 833582ff68c94269be364ebd187a75db. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 607a503b64d8440a81cf66ab158ae273. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:21.354440 10573 consensus_queue.cc:1035] T 607a503b64d8440a81cf66ab158ae273 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:21.354516 30761 raft_consensus.cc:1261] T 607a503b64d8440a81cf66ab158ae273 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:21.354657 10805 ts_tablet_manager.cc:902] T fe4d26ee022b424195e836834bd2f051 P ced21ca983b749d0b4d4f6a0e2426bf8: Initiating tablet copy from peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:14:21.355804 10808 ts_tablet_manager.cc:902] T 7083414bc16c4d499c97b93ca4e3f631 P 6d1d7de19459435fbde1999e3dfd226b: Initiating tablet copy from peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:14:21.356220 10573 consensus_queue.cc:1035] T 607a503b64d8440a81cf66ab158ae273 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:21.370676 10808 tablet_copy_client.cc:308] T 7083414bc16c4d499c97b93ca4e3f631 P 6d1d7de19459435fbde1999e3dfd226b: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.66:37143
I20240503 08:14:21.370713 10747 raft_consensus.cc:2932] T 607a503b64d8440a81cf66ab158ae273 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:14:21.371287 28153 tablet_copy_service.cc:140] P 1b411c6bb25549368041773d7b710d48: Received BeginTabletCopySession request for tablet 7083414bc16c4d499c97b93ca4e3f631 from peer 6d1d7de19459435fbde1999e3dfd226b ({username='slave'} at 127.20.230.69:49059)
I20240503 08:14:21.371425 28153 tablet_copy_service.cc:161] P 1b411c6bb25549368041773d7b710d48: Beginning new tablet copy session on tablet 7083414bc16c4d499c97b93ca4e3f631 from peer 6d1d7de19459435fbde1999e3dfd226b at {username='slave'} at 127.20.230.69:49059: session id = 6d1d7de19459435fbde1999e3dfd226b-7083414bc16c4d499c97b93ca4e3f631
I20240503 08:14:21.374325 28153 tablet_copy_source_session.cc:205] T 7083414bc16c4d499c97b93ca4e3f631 P 1b411c6bb25549368041773d7b710d48: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:21.379364 30761 raft_consensus.cc:2932] T 607a503b64d8440a81cf66ab158ae273 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:14:21.374606 28133 raft_consensus.cc:2932] T 607a503b64d8440a81cf66ab158ae273 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:14:21.382658 10808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7083414bc16c4d499c97b93ca4e3f631. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:21.384085 27860 catalog_manager.cc:5484] T 607a503b64d8440a81cf66ab158ae273 P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: config changed from index -1 to 2, VOTER 833582ff68c94269be364ebd187a75db (127.20.230.65) added. New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:14:21.405978 10808 tablet_copy_client.cc:783] T 7083414bc16c4d499c97b93ca4e3f631 P 6d1d7de19459435fbde1999e3dfd226b: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:21.406237 10808 tablet_copy_client.cc:647] T 7083414bc16c4d499c97b93ca4e3f631 P 6d1d7de19459435fbde1999e3dfd226b: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:21.408857 10808 tablet_copy_client.cc:523] T 7083414bc16c4d499c97b93ca4e3f631 P 6d1d7de19459435fbde1999e3dfd226b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:21.415156 10808 tablet_bootstrap.cc:492] T 7083414bc16c4d499c97b93ca4e3f631 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:14:21.416007 10819 ts_tablet_manager.cc:902] T 2e511a3ab0ee4eabb22240de99690603 P d09a11a03989421b9f28019dc1c14cc2: Initiating tablet copy from peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:14:21.424347 10820 ts_tablet_manager.cc:902] T c68032ba399547e3bd272f424927fbce P 6d1d7de19459435fbde1999e3dfd226b: Initiating tablet copy from peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:14:21.433183 10805 tablet_copy_client.cc:308] T fe4d26ee022b424195e836834bd2f051 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.66:37143
I20240503 08:14:21.433952 28153 tablet_copy_service.cc:140] P 1b411c6bb25549368041773d7b710d48: Received BeginTabletCopySession request for tablet fe4d26ee022b424195e836834bd2f051 from peer ced21ca983b749d0b4d4f6a0e2426bf8 ({username='slave'} at 127.20.230.71:43205)
I20240503 08:14:21.434070 28153 tablet_copy_service.cc:161] P 1b411c6bb25549368041773d7b710d48: Beginning new tablet copy session on tablet fe4d26ee022b424195e836834bd2f051 from peer ced21ca983b749d0b4d4f6a0e2426bf8 at {username='slave'} at 127.20.230.71:43205: session id = ced21ca983b749d0b4d4f6a0e2426bf8-fe4d26ee022b424195e836834bd2f051
I20240503 08:14:21.436271 28153 tablet_copy_source_session.cc:205] T fe4d26ee022b424195e836834bd2f051 P 1b411c6bb25549368041773d7b710d48: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:21.437508 10805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe4d26ee022b424195e836834bd2f051. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:21.440277 10820 tablet_copy_client.cc:308] T c68032ba399547e3bd272f424927fbce P 6d1d7de19459435fbde1999e3dfd226b: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.65:39081
I20240503 08:14:21.440919 28023 tablet_copy_service.cc:140] P 833582ff68c94269be364ebd187a75db: Received BeginTabletCopySession request for tablet c68032ba399547e3bd272f424927fbce from peer 6d1d7de19459435fbde1999e3dfd226b ({username='slave'} at 127.20.230.69:32907)
I20240503 08:14:21.441087 28023 tablet_copy_service.cc:161] P 833582ff68c94269be364ebd187a75db: Beginning new tablet copy session on tablet c68032ba399547e3bd272f424927fbce from peer 6d1d7de19459435fbde1999e3dfd226b at {username='slave'} at 127.20.230.69:32907: session id = 6d1d7de19459435fbde1999e3dfd226b-c68032ba399547e3bd272f424927fbce
I20240503 08:14:21.444099 28023 tablet_copy_source_session.cc:205] T c68032ba399547e3bd272f424927fbce P 833582ff68c94269be364ebd187a75db: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:21.445382 10820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c68032ba399547e3bd272f424927fbce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:21.455845 10820 tablet_copy_client.cc:783] T c68032ba399547e3bd272f424927fbce P 6d1d7de19459435fbde1999e3dfd226b: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:21.456131 10820 tablet_copy_client.cc:647] T c68032ba399547e3bd272f424927fbce P 6d1d7de19459435fbde1999e3dfd226b: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:21.456939 10805 tablet_copy_client.cc:783] T fe4d26ee022b424195e836834bd2f051 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:21.457221 10805 tablet_copy_client.cc:647] T fe4d26ee022b424195e836834bd2f051 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:21.459439 10805 tablet_copy_client.cc:523] T fe4d26ee022b424195e836834bd2f051 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:21.460899 10820 tablet_copy_client.cc:523] T c68032ba399547e3bd272f424927fbce P 6d1d7de19459435fbde1999e3dfd226b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:21.469702 10805 tablet_bootstrap.cc:492] T fe4d26ee022b424195e836834bd2f051 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:14:21.474825 10820 tablet_bootstrap.cc:492] T c68032ba399547e3bd272f424927fbce P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:14:21.511498 10819 tablet_copy_client.cc:308] T 2e511a3ab0ee4eabb22240de99690603 P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.69:34725
I20240503 08:14:21.511837 10836 ts_tablet_manager.cc:902] T 6c2824aa878845e098f600f1fcfea087 P 1b411c6bb25549368041773d7b710d48: Initiating tablet copy from peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:14:21.512212 30783 tablet_copy_service.cc:140] P 6d1d7de19459435fbde1999e3dfd226b: Received BeginTabletCopySession request for tablet 2e511a3ab0ee4eabb22240de99690603 from peer d09a11a03989421b9f28019dc1c14cc2 ({username='slave'} at 127.20.230.68:50161)
I20240503 08:14:21.512356 30783 tablet_copy_service.cc:161] P 6d1d7de19459435fbde1999e3dfd226b: Beginning new tablet copy session on tablet 2e511a3ab0ee4eabb22240de99690603 from peer d09a11a03989421b9f28019dc1c14cc2 at {username='slave'} at 127.20.230.68:50161: session id = d09a11a03989421b9f28019dc1c14cc2-2e511a3ab0ee4eabb22240de99690603
I20240503 08:14:21.514958 30783 tablet_copy_source_session.cc:205] T 2e511a3ab0ee4eabb22240de99690603 P 6d1d7de19459435fbde1999e3dfd226b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:21.516371 10819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e511a3ab0ee4eabb22240de99690603. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:21.522645 10819 tablet_copy_client.cc:783] T 2e511a3ab0ee4eabb22240de99690603 P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:21.522927 10819 tablet_copy_client.cc:647] T 2e511a3ab0ee4eabb22240de99690603 P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:21.524981 10819 tablet_copy_client.cc:523] T 2e511a3ab0ee4eabb22240de99690603 P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:21.530052 10819 tablet_bootstrap.cc:492] T 2e511a3ab0ee4eabb22240de99690603 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:14:21.538671 10836 tablet_copy_client.cc:308] T 6c2824aa878845e098f600f1fcfea087 P 1b411c6bb25549368041773d7b710d48: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.69:34725
I20240503 08:14:21.539364 30783 tablet_copy_service.cc:140] P 6d1d7de19459435fbde1999e3dfd226b: Received BeginTabletCopySession request for tablet 6c2824aa878845e098f600f1fcfea087 from peer 1b411c6bb25549368041773d7b710d48 ({username='slave'} at 127.20.230.66:42445)
I20240503 08:14:21.539517 30783 tablet_copy_service.cc:161] P 6d1d7de19459435fbde1999e3dfd226b: Beginning new tablet copy session on tablet 6c2824aa878845e098f600f1fcfea087 from peer 1b411c6bb25549368041773d7b710d48 at {username='slave'} at 127.20.230.66:42445: session id = 1b411c6bb25549368041773d7b710d48-6c2824aa878845e098f600f1fcfea087
I20240503 08:14:21.542764 30783 tablet_copy_source_session.cc:205] T 6c2824aa878845e098f600f1fcfea087 P 6d1d7de19459435fbde1999e3dfd226b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:21.544029 10836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c2824aa878845e098f600f1fcfea087. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:21.554114 10836 tablet_copy_client.cc:783] T 6c2824aa878845e098f600f1fcfea087 P 1b411c6bb25549368041773d7b710d48: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:21.554644 10836 tablet_copy_client.cc:647] T 6c2824aa878845e098f600f1fcfea087 P 1b411c6bb25549368041773d7b710d48: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:21.559288 10836 tablet_copy_client.cc:523] T 6c2824aa878845e098f600f1fcfea087 P 1b411c6bb25549368041773d7b710d48: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:21.569089 10836 tablet_bootstrap.cc:492] T 6c2824aa878845e098f600f1fcfea087 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:14:21.576058 10819 tablet_bootstrap.cc:492] T 2e511a3ab0ee4eabb22240de99690603 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:21.576489 10819 tablet_bootstrap.cc:492] T 2e511a3ab0ee4eabb22240de99690603 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap complete.
I20240503 08:14:21.576776 10819 ts_tablet_manager.cc:1367] T 2e511a3ab0ee4eabb22240de99690603 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.047s user 0.010s sys 0.019s
I20240503 08:14:21.577636 10819 raft_consensus.cc:348] T 2e511a3ab0ee4eabb22240de99690603 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:14:21.578043 10819 raft_consensus.cc:729] T 2e511a3ab0ee4eabb22240de99690603 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:14:21.578393 10819 consensus_queue.cc:260] T 2e511a3ab0ee4eabb22240de99690603 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:14:21.580597 10819 ts_tablet_manager.cc:1398] T 2e511a3ab0ee4eabb22240de99690603 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 08:14:21.582603 30783 tablet_copy_service.cc:342] P 6d1d7de19459435fbde1999e3dfd226b: Request end of tablet copy session d09a11a03989421b9f28019dc1c14cc2-2e511a3ab0ee4eabb22240de99690603 received from {username='slave'} at 127.20.230.68:50161
I20240503 08:14:21.582698 30783 tablet_copy_service.cc:433] P 6d1d7de19459435fbde1999e3dfd226b: ending tablet copy session d09a11a03989421b9f28019dc1c14cc2-2e511a3ab0ee4eabb22240de99690603 on tablet 2e511a3ab0ee4eabb22240de99690603 with peer d09a11a03989421b9f28019dc1c14cc2
I20240503 08:14:21.587880 10808 tablet_bootstrap.cc:492] T 7083414bc16c4d499c97b93ca4e3f631 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:21.595788 10808 tablet_bootstrap.cc:492] T 7083414bc16c4d499c97b93ca4e3f631 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap complete.
I20240503 08:14:21.597869 10808 ts_tablet_manager.cc:1367] T 7083414bc16c4d499c97b93ca4e3f631 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.183s user 0.048s sys 0.000s
I20240503 08:14:21.600287 10808 raft_consensus.cc:348] T 7083414bc16c4d499c97b93ca4e3f631 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:14:21.600888 10808 raft_consensus.cc:729] T 7083414bc16c4d499c97b93ca4e3f631 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:14:21.601202 10808 consensus_queue.cc:260] T 7083414bc16c4d499c97b93ca4e3f631 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:14:21.604104 10808 ts_tablet_manager.cc:1398] T 7083414bc16c4d499c97b93ca4e3f631 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:14:21.604835 28153 tablet_copy_service.cc:342] P 1b411c6bb25549368041773d7b710d48: Request end of tablet copy session 6d1d7de19459435fbde1999e3dfd226b-7083414bc16c4d499c97b93ca4e3f631 received from {username='slave'} at 127.20.230.69:49059
I20240503 08:14:21.604955 28153 tablet_copy_service.cc:433] P 1b411c6bb25549368041773d7b710d48: ending tablet copy session 6d1d7de19459435fbde1999e3dfd226b-7083414bc16c4d499c97b93ca4e3f631 on tablet 7083414bc16c4d499c97b93ca4e3f631 with peer 6d1d7de19459435fbde1999e3dfd226b
I20240503 08:14:21.609475 10836 tablet_bootstrap.cc:492] T 6c2824aa878845e098f600f1fcfea087 P 1b411c6bb25549368041773d7b710d48: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:21.609884 10836 tablet_bootstrap.cc:492] T 6c2824aa878845e098f600f1fcfea087 P 1b411c6bb25549368041773d7b710d48: Bootstrap complete.
I20240503 08:14:21.610111 10836 ts_tablet_manager.cc:1367] T 6c2824aa878845e098f600f1fcfea087 P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.004s
I20240503 08:14:21.610764 10836 raft_consensus.cc:348] T 6c2824aa878845e098f600f1fcfea087 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:14:21.610970 10836 raft_consensus.cc:729] T 6c2824aa878845e098f600f1fcfea087 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:14:21.611184 10836 consensus_queue.cc:260] T 6c2824aa878845e098f600f1fcfea087 P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:14:21.622356 10836 ts_tablet_manager.cc:1398] T 6c2824aa878845e098f600f1fcfea087 P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240503 08:14:21.629272 30783 tablet_copy_service.cc:342] P 6d1d7de19459435fbde1999e3dfd226b: Request end of tablet copy session 1b411c6bb25549368041773d7b710d48-6c2824aa878845e098f600f1fcfea087 received from {username='slave'} at 127.20.230.66:42445
I20240503 08:14:21.629390 30783 tablet_copy_service.cc:433] P 6d1d7de19459435fbde1999e3dfd226b: ending tablet copy session 1b411c6bb25549368041773d7b710d48-6c2824aa878845e098f600f1fcfea087 on tablet 6c2824aa878845e098f600f1fcfea087 with peer 1b411c6bb25549368041773d7b710d48
I20240503 08:14:21.631700 10820 tablet_bootstrap.cc:492] T c68032ba399547e3bd272f424927fbce P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:21.632156 10820 tablet_bootstrap.cc:492] T c68032ba399547e3bd272f424927fbce P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap complete.
I20240503 08:14:21.632412 10820 ts_tablet_manager.cc:1367] T c68032ba399547e3bd272f424927fbce P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.161s user 0.037s sys 0.016s
I20240503 08:14:21.633172 10820 raft_consensus.cc:348] T c68032ba399547e3bd272f424927fbce P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:14:21.634546 10820 raft_consensus.cc:729] T c68032ba399547e3bd272f424927fbce P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:14:21.634820 10820 consensus_queue.cc:260] T c68032ba399547e3bd272f424927fbce P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:14:21.640005 10820 ts_tablet_manager.cc:1398] T c68032ba399547e3bd272f424927fbce P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20240503 08:14:21.640661 28023 tablet_copy_service.cc:342] P 833582ff68c94269be364ebd187a75db: Request end of tablet copy session 6d1d7de19459435fbde1999e3dfd226b-c68032ba399547e3bd272f424927fbce received from {username='slave'} at 127.20.230.69:32907
I20240503 08:14:21.640761 28023 tablet_copy_service.cc:433] P 833582ff68c94269be364ebd187a75db: ending tablet copy session 6d1d7de19459435fbde1999e3dfd226b-c68032ba399547e3bd272f424927fbce on tablet c68032ba399547e3bd272f424927fbce with peer 6d1d7de19459435fbde1999e3dfd226b
I20240503 08:14:21.652910 10805 tablet_bootstrap.cc:492] T fe4d26ee022b424195e836834bd2f051 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:21.653462 10805 tablet_bootstrap.cc:492] T fe4d26ee022b424195e836834bd2f051 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap complete.
I20240503 08:14:21.653705 10805 ts_tablet_manager.cc:1367] T fe4d26ee022b424195e836834bd2f051 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.184s user 0.057s sys 0.000s
I20240503 08:14:21.654299 10805 raft_consensus.cc:348] T fe4d26ee022b424195e836834bd2f051 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:21.654544 10805 raft_consensus.cc:729] T fe4d26ee022b424195e836834bd2f051 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:14:21.654771 10805 consensus_queue.cc:260] T fe4d26ee022b424195e836834bd2f051 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:21.656607 10805 ts_tablet_manager.cc:1398] T fe4d26ee022b424195e836834bd2f051 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:14:21.656642 10836 ts_tablet_manager.cc:902] T 720f36100d794158be13ff02595e6ddf P 1b411c6bb25549368041773d7b710d48: Initiating tablet copy from peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:14:21.657215 28153 tablet_copy_service.cc:342] P 1b411c6bb25549368041773d7b710d48: Request end of tablet copy session ced21ca983b749d0b4d4f6a0e2426bf8-fe4d26ee022b424195e836834bd2f051 received from {username='slave'} at 127.20.230.71:43205
I20240503 08:14:21.657317 28153 tablet_copy_service.cc:433] P 1b411c6bb25549368041773d7b710d48: ending tablet copy session ced21ca983b749d0b4d4f6a0e2426bf8-fe4d26ee022b424195e836834bd2f051 on tablet fe4d26ee022b424195e836834bd2f051 with peer ced21ca983b749d0b4d4f6a0e2426bf8
I20240503 08:14:21.712543 10819 ts_tablet_manager.cc:902] T 45f47478f49149b895060791655503ae P d09a11a03989421b9f28019dc1c14cc2: Initiating tablet copy from peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:14:21.715030 28389 consensus_queue.cc:237] T ba1fc93781714efb8eb7ec67c2fceb8a P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:14:21.715329 10836 tablet_copy_client.cc:308] T 720f36100d794158be13ff02595e6ddf P 1b411c6bb25549368041773d7b710d48: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.68:46211
I20240503 08:14:21.716151 28413 tablet_copy_service.cc:140] P d09a11a03989421b9f28019dc1c14cc2: Received BeginTabletCopySession request for tablet 720f36100d794158be13ff02595e6ddf from peer 1b411c6bb25549368041773d7b710d48 ({username='slave'} at 127.20.230.66:43721)
I20240503 08:14:21.716290 28413 tablet_copy_service.cc:161] P d09a11a03989421b9f28019dc1c14cc2: Beginning new tablet copy session on tablet 720f36100d794158be13ff02595e6ddf from peer 1b411c6bb25549368041773d7b710d48 at {username='slave'} at 127.20.230.66:43721: session id = 1b411c6bb25549368041773d7b710d48-720f36100d794158be13ff02595e6ddf
I20240503 08:14:21.718542 28129 raft_consensus.cc:1261] T ba1fc93781714efb8eb7ec67c2fceb8a P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:21.719082 28413 tablet_copy_source_session.cc:205] T 720f36100d794158be13ff02595e6ddf P d09a11a03989421b9f28019dc1c14cc2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:21.719058 10747 consensus_queue.cc:1035] T ba1fc93781714efb8eb7ec67c2fceb8a P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:14:21.719600 28326 consensus_peers.cc:485] T ba1fc93781714efb8eb7ec67c2fceb8a P d09a11a03989421b9f28019dc1c14cc2 -> Peer ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): Couldn't send request to peer ce1496a47d084365ad786cd09e472a2c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba1fc93781714efb8eb7ec67c2fceb8a. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:21.720860 10836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 720f36100d794158be13ff02595e6ddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:21.727504 10836 tablet_copy_client.cc:783] T 720f36100d794158be13ff02595e6ddf P 1b411c6bb25549368041773d7b710d48: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:21.727696 10836 tablet_copy_client.cc:647] T 720f36100d794158be13ff02595e6ddf P 1b411c6bb25549368041773d7b710d48: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:21.739681 10819 tablet_copy_client.cc:308] T 45f47478f49149b895060791655503ae P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.65:39081
I20240503 08:14:21.740257 28023 tablet_copy_service.cc:140] P 833582ff68c94269be364ebd187a75db: Received BeginTabletCopySession request for tablet 45f47478f49149b895060791655503ae from peer d09a11a03989421b9f28019dc1c14cc2 ({username='slave'} at 127.20.230.68:42231)
I20240503 08:14:21.740381 28023 tablet_copy_service.cc:161] P 833582ff68c94269be364ebd187a75db: Beginning new tablet copy session on tablet 45f47478f49149b895060791655503ae from peer d09a11a03989421b9f28019dc1c14cc2 at {username='slave'} at 127.20.230.68:42231: session id = d09a11a03989421b9f28019dc1c14cc2-45f47478f49149b895060791655503ae
I20240503 08:14:21.743234 28023 tablet_copy_source_session.cc:205] T 45f47478f49149b895060791655503ae P 833582ff68c94269be364ebd187a75db: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:21.744832 10819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45f47478f49149b895060791655503ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:21.747166 28262 raft_consensus.cc:1261] T ba1fc93781714efb8eb7ec67c2fceb8a P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:21.747674 10747 consensus_queue.cc:1035] T ba1fc93781714efb8eb7ec67c2fceb8a P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:21.751534 10819 tablet_copy_client.cc:783] T 45f47478f49149b895060791655503ae P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:21.751791 10819 tablet_copy_client.cc:647] T 45f47478f49149b895060791655503ae P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:21.761180 10819 tablet_copy_client.cc:523] T 45f47478f49149b895060791655503ae P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:21.762290 10836 tablet_copy_client.cc:523] T 720f36100d794158be13ff02595e6ddf P 1b411c6bb25549368041773d7b710d48: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:21.767930 10747 raft_consensus.cc:2932] T ba1fc93781714efb8eb7ec67c2fceb8a P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:14:21.768932 10819 tablet_bootstrap.cc:492] T 45f47478f49149b895060791655503ae P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:14:21.773919 10836 tablet_bootstrap.cc:492] T 720f36100d794158be13ff02595e6ddf P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
I20240503 08:14:21.774762 31027 raft_consensus.cc:1203] T fe4d26ee022b424195e836834bd2f051 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:14:21.775781 10819 tablet_bootstrap.cc:492] T 45f47478f49149b895060791655503ae P d09a11a03989421b9f28019dc1c14cc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:21.776137 10819 tablet_bootstrap.cc:492] T 45f47478f49149b895060791655503ae P d09a11a03989421b9f28019dc1c14cc2: Bootstrap complete.
I20240503 08:14:21.776348 10819 ts_tablet_manager.cc:1367] T 45f47478f49149b895060791655503ae P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.002s
I20240503 08:14:21.778851 28129 raft_consensus.cc:2932] T ba1fc93781714efb8eb7ec67c2fceb8a P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:14:21.779134 10819 raft_consensus.cc:348] T 45f47478f49149b895060791655503ae P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:14:21.779336 10819 raft_consensus.cc:729] T 45f47478f49149b895060791655503ae P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:14:21.779537 10819 consensus_queue.cc:260] T 45f47478f49149b895060791655503ae P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:14:21.780105 10819 ts_tablet_manager.cc:1398] T 45f47478f49149b895060791655503ae P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20240503 08:14:21.786939 27860 catalog_manager.cc:5484] T ba1fc93781714efb8eb7ec67c2fceb8a P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: config changed from index -1 to 4, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:14:21.790575 28023 tablet_copy_service.cc:342] P 833582ff68c94269be364ebd187a75db: Request end of tablet copy session d09a11a03989421b9f28019dc1c14cc2-45f47478f49149b895060791655503ae received from {username='slave'} at 127.20.230.68:42231
I20240503 08:14:21.790692 28023 tablet_copy_service.cc:433] P 833582ff68c94269be364ebd187a75db: ending tablet copy session d09a11a03989421b9f28019dc1c14cc2-45f47478f49149b895060791655503ae on tablet 45f47478f49149b895060791655503ae with peer d09a11a03989421b9f28019dc1c14cc2
I20240503 08:14:21.792402 28260 raft_consensus.cc:2932] T ba1fc93781714efb8eb7ec67c2fceb8a P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:14:21.794728 10836 tablet_bootstrap.cc:492] T 720f36100d794158be13ff02595e6ddf P 1b411c6bb25549368041773d7b710d48: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:21.795083 10836 tablet_bootstrap.cc:492] T 720f36100d794158be13ff02595e6ddf P 1b411c6bb25549368041773d7b710d48: Bootstrap complete.
I20240503 08:14:21.795317 10836 ts_tablet_manager.cc:1367] T 720f36100d794158be13ff02595e6ddf P 1b411c6bb25549368041773d7b710d48: Time spent bootstrapping tablet: real 0.030s user 0.012s sys 0.006s
I20240503 08:14:21.795943 10836 raft_consensus.cc:348] T 720f36100d794158be13ff02595e6ddf P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:14:21.796358 10836 raft_consensus.cc:729] T 720f36100d794158be13ff02595e6ddf P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Initialized, Role: FOLLOWER
I20240503 08:14:21.796582 10836 consensus_queue.cc:260] T 720f36100d794158be13ff02595e6ddf P 1b411c6bb25549368041773d7b710d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:14:21.797075 10836 ts_tablet_manager.cc:1398] T 720f36100d794158be13ff02595e6ddf P 1b411c6bb25549368041773d7b710d48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:21.799845 28413 tablet_copy_service.cc:342] P d09a11a03989421b9f28019dc1c14cc2: Request end of tablet copy session 1b411c6bb25549368041773d7b710d48-720f36100d794158be13ff02595e6ddf received from {username='slave'} at 127.20.230.66:43721
I20240503 08:14:21.799964 28413 tablet_copy_service.cc:433] P d09a11a03989421b9f28019dc1c14cc2: ending tablet copy session 1b411c6bb25549368041773d7b710d48-720f36100d794158be13ff02595e6ddf on tablet 720f36100d794158be13ff02595e6ddf with peer 1b411c6bb25549368041773d7b710d48
I20240503 08:14:21.822600 10820 ts_tablet_manager.cc:902] T d61ad2e11c4544bfbdef7df1e92d2d11 P 6d1d7de19459435fbde1999e3dfd226b: Initiating tablet copy from peer ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:14:21.824602 10820 tablet_copy_client.cc:308] T d61ad2e11c4544bfbdef7df1e92d2d11 P 6d1d7de19459435fbde1999e3dfd226b: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.70:33747
I20240503 08:14:21.825317 30915 tablet_copy_service.cc:140] P ce1496a47d084365ad786cd09e472a2c: Received BeginTabletCopySession request for tablet d61ad2e11c4544bfbdef7df1e92d2d11 from peer 6d1d7de19459435fbde1999e3dfd226b ({username='slave'} at 127.20.230.69:51843)
I20240503 08:14:21.825465 30915 tablet_copy_service.cc:161] P ce1496a47d084365ad786cd09e472a2c: Beginning new tablet copy session on tablet d61ad2e11c4544bfbdef7df1e92d2d11 from peer 6d1d7de19459435fbde1999e3dfd226b at {username='slave'} at 127.20.230.69:51843: session id = 6d1d7de19459435fbde1999e3dfd226b-d61ad2e11c4544bfbdef7df1e92d2d11
I20240503 08:14:21.827701 30915 tablet_copy_source_session.cc:205] T d61ad2e11c4544bfbdef7df1e92d2d11 P ce1496a47d084365ad786cd09e472a2c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:21.828799 10820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d61ad2e11c4544bfbdef7df1e92d2d11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:21.833472 10820 tablet_copy_client.cc:783] T d61ad2e11c4544bfbdef7df1e92d2d11 P 6d1d7de19459435fbde1999e3dfd226b: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:21.833671 10820 tablet_copy_client.cc:647] T d61ad2e11c4544bfbdef7df1e92d2d11 P 6d1d7de19459435fbde1999e3dfd226b: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:21.834982 10820 tablet_copy_client.cc:523] T d61ad2e11c4544bfbdef7df1e92d2d11 P 6d1d7de19459435fbde1999e3dfd226b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:21.836490 10884 ts_tablet_manager.cc:902] T 607a503b64d8440a81cf66ab158ae273 P 833582ff68c94269be364ebd187a75db: Initiating tablet copy from peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:14:21.838249 10820 tablet_bootstrap.cc:492] T d61ad2e11c4544bfbdef7df1e92d2d11 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:14:21.847710 10820 tablet_bootstrap.cc:492] T d61ad2e11c4544bfbdef7df1e92d2d11 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:21.848095 10820 tablet_bootstrap.cc:492] T d61ad2e11c4544bfbdef7df1e92d2d11 P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap complete.
I20240503 08:14:21.848340 10820 ts_tablet_manager.cc:1367] T d61ad2e11c4544bfbdef7df1e92d2d11 P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20240503 08:14:21.848999 10820 raft_consensus.cc:348] T d61ad2e11c4544bfbdef7df1e92d2d11 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:14:21.849243 10820 raft_consensus.cc:729] T d61ad2e11c4544bfbdef7df1e92d2d11 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:14:21.849480 10820 consensus_queue.cc:260] T d61ad2e11c4544bfbdef7df1e92d2d11 P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:14:21.850219 10820 ts_tablet_manager.cc:1398] T d61ad2e11c4544bfbdef7df1e92d2d11 P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:21.859611 30915 tablet_copy_service.cc:342] P ce1496a47d084365ad786cd09e472a2c: Request end of tablet copy session 6d1d7de19459435fbde1999e3dfd226b-d61ad2e11c4544bfbdef7df1e92d2d11 received from {username='slave'} at 127.20.230.69:51843
I20240503 08:14:21.859742 30915 tablet_copy_service.cc:433] P ce1496a47d084365ad786cd09e472a2c: ending tablet copy session 6d1d7de19459435fbde1999e3dfd226b-d61ad2e11c4544bfbdef7df1e92d2d11 on tablet d61ad2e11c4544bfbdef7df1e92d2d11 with peer 6d1d7de19459435fbde1999e3dfd226b
I20240503 08:14:21.869236 10884 tablet_copy_client.cc:308] T 607a503b64d8440a81cf66ab158ae273 P 833582ff68c94269be364ebd187a75db: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.68:46211
I20240503 08:14:21.869895 28413 tablet_copy_service.cc:140] P d09a11a03989421b9f28019dc1c14cc2: Received BeginTabletCopySession request for tablet 607a503b64d8440a81cf66ab158ae273 from peer 833582ff68c94269be364ebd187a75db ({username='slave'} at 127.20.230.65:36529)
I20240503 08:14:21.870033 28413 tablet_copy_service.cc:161] P d09a11a03989421b9f28019dc1c14cc2: Beginning new tablet copy session on tablet 607a503b64d8440a81cf66ab158ae273 from peer 833582ff68c94269be364ebd187a75db at {username='slave'} at 127.20.230.65:36529: session id = 833582ff68c94269be364ebd187a75db-607a503b64d8440a81cf66ab158ae273
I20240503 08:14:21.871997 28413 tablet_copy_source_session.cc:205] T 607a503b64d8440a81cf66ab158ae273 P d09a11a03989421b9f28019dc1c14cc2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:21.873425 10884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 607a503b64d8440a81cf66ab158ae273. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:21.880259 10884 tablet_copy_client.cc:783] T 607a503b64d8440a81cf66ab158ae273 P 833582ff68c94269be364ebd187a75db: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:21.880488 10884 tablet_copy_client.cc:647] T 607a503b64d8440a81cf66ab158ae273 P 833582ff68c94269be364ebd187a75db: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:21.881915 10884 tablet_copy_client.cc:523] T 607a503b64d8440a81cf66ab158ae273 P 833582ff68c94269be364ebd187a75db: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:21.884662 10884 tablet_bootstrap.cc:492] T 607a503b64d8440a81cf66ab158ae273 P 833582ff68c94269be364ebd187a75db: Bootstrap starting.
I20240503 08:14:21.890556 10884 tablet_bootstrap.cc:492] T 607a503b64d8440a81cf66ab158ae273 P 833582ff68c94269be364ebd187a75db: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:21.890952 10884 tablet_bootstrap.cc:492] T 607a503b64d8440a81cf66ab158ae273 P 833582ff68c94269be364ebd187a75db: Bootstrap complete.
I20240503 08:14:21.891218 10884 ts_tablet_manager.cc:1367] T 607a503b64d8440a81cf66ab158ae273 P 833582ff68c94269be364ebd187a75db: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.006s
I20240503 08:14:21.892107 10884 raft_consensus.cc:348] T 607a503b64d8440a81cf66ab158ae273 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:14:21.892354 10884 raft_consensus.cc:729] T 607a503b64d8440a81cf66ab158ae273 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833582ff68c94269be364ebd187a75db, State: Initialized, Role: FOLLOWER
I20240503 08:14:21.892585 10884 consensus_queue.cc:260] T 607a503b64d8440a81cf66ab158ae273 P 833582ff68c94269be364ebd187a75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:14:21.893079 10884 ts_tablet_manager.cc:1398] T 607a503b64d8440a81cf66ab158ae273 P 833582ff68c94269be364ebd187a75db: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:14:21.893862 28413 tablet_copy_service.cc:342] P d09a11a03989421b9f28019dc1c14cc2: Request end of tablet copy session 833582ff68c94269be364ebd187a75db-607a503b64d8440a81cf66ab158ae273 received from {username='slave'} at 127.20.230.65:36529
I20240503 08:14:21.893980 28413 tablet_copy_service.cc:433] P d09a11a03989421b9f28019dc1c14cc2: ending tablet copy session 833582ff68c94269be364ebd187a75db-607a503b64d8440a81cf66ab158ae273 on tablet 607a503b64d8440a81cf66ab158ae273 with peer 833582ff68c94269be364ebd187a75db
I20240503 08:14:21.936525 30761 raft_consensus.cc:1203] T 7083414bc16c4d499c97b93ca4e3f631 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:14:21.947762 30761 raft_consensus.cc:1203] T c68032ba399547e3bd272f424927fbce P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:14:21.999315 28390 raft_consensus.cc:1203] T 2e511a3ab0ee4eabb22240de99690603 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:14:22.033509 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:22.058655 28129 raft_consensus.cc:1203] T 6c2824aa878845e098f600f1fcfea087 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:14:22.063261 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:22.086266 28129 raft_consensus.cc:1203] T 720f36100d794158be13ff02595e6ddf P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:14:22.102152 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:22.149724 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:14:22.158041 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:22.176621 31141 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:22.195852 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:22.233032 30762 raft_consensus.cc:1203] T d61ad2e11c4544bfbdef7df1e92d2d11 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:14:22.239863 28389 raft_consensus.cc:1203] T 45f47478f49149b895060791655503ae P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:14:22.266297 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:22.280730 27999 raft_consensus.cc:1203] T 607a503b64d8440a81cf66ab158ae273 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:14:22.293701 10939 ts_tablet_manager.cc:902] T ba1fc93781714efb8eb7ec67c2fceb8a P ce1496a47d084365ad786cd09e472a2c: Initiating tablet copy from peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:14:22.347447 10939 tablet_copy_client.cc:308] T ba1fc93781714efb8eb7ec67c2fceb8a P ce1496a47d084365ad786cd09e472a2c: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.68:46211
I20240503 08:14:22.348120 28413 tablet_copy_service.cc:140] P d09a11a03989421b9f28019dc1c14cc2: Received BeginTabletCopySession request for tablet ba1fc93781714efb8eb7ec67c2fceb8a from peer ce1496a47d084365ad786cd09e472a2c ({username='slave'} at 127.20.230.70:51715)
I20240503 08:14:22.348284 28413 tablet_copy_service.cc:161] P d09a11a03989421b9f28019dc1c14cc2: Beginning new tablet copy session on tablet ba1fc93781714efb8eb7ec67c2fceb8a from peer ce1496a47d084365ad786cd09e472a2c at {username='slave'} at 127.20.230.70:51715: session id = ce1496a47d084365ad786cd09e472a2c-ba1fc93781714efb8eb7ec67c2fceb8a
I20240503 08:14:22.351173 28413 tablet_copy_source_session.cc:205] T ba1fc93781714efb8eb7ec67c2fceb8a P d09a11a03989421b9f28019dc1c14cc2: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:14:22.352855 10939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba1fc93781714efb8eb7ec67c2fceb8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:22.362427 10939 tablet_copy_client.cc:783] T ba1fc93781714efb8eb7ec67c2fceb8a P ce1496a47d084365ad786cd09e472a2c: tablet copy: Starting download of 5 data blocks...
I20240503 08:14:22.374382 10939 tablet_copy_client.cc:647] T ba1fc93781714efb8eb7ec67c2fceb8a P ce1496a47d084365ad786cd09e472a2c: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:22.377880 10939 tablet_copy_client.cc:523] T ba1fc93781714efb8eb7ec67c2fceb8a P ce1496a47d084365ad786cd09e472a2c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:22.381654 10939 tablet_bootstrap.cc:492] T ba1fc93781714efb8eb7ec67c2fceb8a P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:14:22.413441 10939 tablet_bootstrap.cc:492] T ba1fc93781714efb8eb7ec67c2fceb8a P ce1496a47d084365ad786cd09e472a2c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:22.413970 10939 tablet_bootstrap.cc:492] T ba1fc93781714efb8eb7ec67c2fceb8a P ce1496a47d084365ad786cd09e472a2c: Bootstrap complete.
I20240503 08:14:22.414530 10939 ts_tablet_manager.cc:1367] T ba1fc93781714efb8eb7ec67c2fceb8a P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.033s user 0.025s sys 0.000s
I20240503 08:14:22.415444 10939 raft_consensus.cc:348] T ba1fc93781714efb8eb7ec67c2fceb8a P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:14:22.415760 10939 raft_consensus.cc:729] T ba1fc93781714efb8eb7ec67c2fceb8a P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:14:22.416074 10939 consensus_queue.cc:260] T ba1fc93781714efb8eb7ec67c2fceb8a P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:14:22.424890 10939 ts_tablet_manager.cc:1398] T ba1fc93781714efb8eb7ec67c2fceb8a P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240503 08:14:22.427112 28413 tablet_copy_service.cc:342] P d09a11a03989421b9f28019dc1c14cc2: Request end of tablet copy session ce1496a47d084365ad786cd09e472a2c-ba1fc93781714efb8eb7ec67c2fceb8a received from {username='slave'} at 127.20.230.70:51715
I20240503 08:14:22.427248 28413 tablet_copy_service.cc:433] P d09a11a03989421b9f28019dc1c14cc2: ending tablet copy session ce1496a47d084365ad786cd09e472a2c-ba1fc93781714efb8eb7ec67c2fceb8a on tablet ba1fc93781714efb8eb7ec67c2fceb8a with peer ce1496a47d084365ad786cd09e472a2c
I20240503 08:14:22.440096 30941 maintenance_manager.cc:419] P ce1496a47d084365ad786cd09e472a2c: Scheduling UndoDeltaBlockGCOp(ba1fc93781714efb8eb7ec67c2fceb8a): 4583 bytes on disk
I20240503 08:14:22.441249 30831 maintenance_manager.cc:643] P ce1496a47d084365ad786cd09e472a2c: UndoDeltaBlockGCOp(ba1fc93781714efb8eb7ec67c2fceb8a) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20240503 08:14:22.716522 30895 raft_consensus.cc:1203] T ba1fc93781714efb8eb7ec67c2fceb8a P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:14:22.752056 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:22.823645 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:22.845925 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:14:22.875437 31141 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:22.890628 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 08:14:22.921999 28327 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.67:39919, user_credentials={real_user=slave}} blocked reactor thread for 52794.9us
I20240503 08:14:22.922801 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:22.949244 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:22.974026 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:23.250768 28179 maintenance_manager.cc:419] P 1b411c6bb25549368041773d7b710d48: Scheduling FlushMRSOp(07866de372e541ec9dca8a3d42f8b0ab): perf score=0.033363
I20240503 08:14:23.258806 28049 maintenance_manager.cc:419] P 833582ff68c94269be364ebd187a75db: Scheduling FlushMRSOp(c68032ba399547e3bd272f424927fbce): perf score=0.033365
I20240503 08:14:23.275339 28309 maintenance_manager.cc:419] P aae860d7ae2b4002bf7794c278f9cc56: Scheduling FlushMRSOp(07866de372e541ec9dca8a3d42f8b0ab): perf score=0.033372
I20240503 08:14:23.294687 28439 maintenance_manager.cc:419] P d09a11a03989421b9f28019dc1c14cc2: Scheduling FlushMRSOp(599cb18c58a74e93a3e0334cc43a73d4): perf score=0.033375
I20240503 08:14:23.402653 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:23.425472 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:23.467536 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:23.478945 28070 maintenance_manager.cc:643] P 1b411c6bb25549368041773d7b710d48: FlushMRSOp(07866de372e541ec9dca8a3d42f8b0ab) complete. Timing: real 0.228s user 0.047s sys 0.006s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":21691,"dirs.run_cpu_time_us":220,"dirs.run_wall_time_us":2978,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":972,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":36725,"threads_started":2,"wal-append.queue_time_us":14014}
I20240503 08:14:23.489338 28200 maintenance_manager.cc:643] P aae860d7ae2b4002bf7794c278f9cc56: FlushMRSOp(07866de372e541ec9dca8a3d42f8b0ab) complete. Timing: real 0.213s user 0.032s sys 0.019s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":21893,"dirs.run_cpu_time_us":240,"dirs.run_wall_time_us":2916,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":882,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":80853888,"thread_start_us":45959,"threads_started":2,"wal-append.queue_time_us":20088}
I20240503 08:14:23.492712 28179 maintenance_manager.cc:419] P 1b411c6bb25549368041773d7b710d48: Scheduling FlushMRSOp(c68032ba399547e3bd272f424927fbce): perf score=0.033422
I20240503 08:14:23.492985 28309 maintenance_manager.cc:419] P aae860d7ae2b4002bf7794c278f9cc56: Scheduling FlushMRSOp(599cb18c58a74e93a3e0334cc43a73d4): perf score=0.033429
I20240503 08:14:23.512753 27940 maintenance_manager.cc:643] P 833582ff68c94269be364ebd187a75db: FlushMRSOp(c68032ba399547e3bd272f424927fbce) complete. Timing: real 0.253s user 0.053s sys 0.062s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":295,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":4779,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1725,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":145441,"threads_started":2,"wal-append.queue_time_us":139669}
I20240503 08:14:23.513289 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:23.515880 28049 maintenance_manager.cc:419] P 833582ff68c94269be364ebd187a75db: Scheduling UndoDeltaBlockGCOp(c68032ba399547e3bd272f424927fbce): 4839 bytes on disk
I20240503 08:14:23.522233 27940 maintenance_manager.cc:643] P 833582ff68c94269be364ebd187a75db: UndoDeltaBlockGCOp(c68032ba399547e3bd272f424927fbce) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
I20240503 08:14:23.574165 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:14:23.575453 31141 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:23.642151 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:23.687739 28330 maintenance_manager.cc:643] P d09a11a03989421b9f28019dc1c14cc2: FlushMRSOp(599cb18c58a74e93a3e0334cc43a73d4) complete. Timing: real 0.393s user 0.052s sys 0.036s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":122409,"dirs.run_cpu_time_us":252,"dirs.run_wall_time_us":7900,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2046,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":134946,"threads_started":2,"wal-append.queue_time_us":12547}
I20240503 08:14:23.693096 28439 maintenance_manager.cc:419] P d09a11a03989421b9f28019dc1c14cc2: Scheduling FlushMRSOp(07866de372e541ec9dca8a3d42f8b0ab): perf score=0.033477
I20240503 08:14:23.722177 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:23.765723 28200 maintenance_manager.cc:643] P aae860d7ae2b4002bf7794c278f9cc56: FlushMRSOp(599cb18c58a74e93a3e0334cc43a73d4) complete. Timing: real 0.264s user 0.035s sys 0.015s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":158,"dirs.run_wall_time_us":1284,"drs_written":1,"lbm_read_time_us":1808,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3,"lbm_write_time_us":768,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":56868,"threads_started":1,"wal-append.queue_time_us":55280}
I20240503 08:14:23.769624 28309 maintenance_manager.cc:419] P aae860d7ae2b4002bf7794c278f9cc56: Scheduling UndoDeltaBlockGCOp(599cb18c58a74e93a3e0334cc43a73d4): 4923 bytes on disk
I20240503 08:14:23.773628 28070 maintenance_manager.cc:643] P 1b411c6bb25549368041773d7b710d48: FlushMRSOp(c68032ba399547e3bd272f424927fbce) complete. Timing: real 0.281s user 0.036s sys 0.018s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":43,"dirs.run_cpu_time_us":235,"dirs.run_wall_time_us":1505,"drs_written":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4,"lbm_write_time_us":746,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":53020,"threads_started":1,"wal-append.queue_time_us":53856}
I20240503 08:14:23.777284 28179 maintenance_manager.cc:419] P 1b411c6bb25549368041773d7b710d48: Scheduling UndoDeltaBlockGCOp(c68032ba399547e3bd272f424927fbce): 4839 bytes on disk
I20240503 08:14:23.778071 28200 maintenance_manager.cc:643] P aae860d7ae2b4002bf7794c278f9cc56: UndoDeltaBlockGCOp(599cb18c58a74e93a3e0334cc43a73d4) complete. Timing: real 0.004s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":126,"lbm_reads_lt_1ms":4}
I20240503 08:14:23.779044 28070 maintenance_manager.cc:643] P 1b411c6bb25549368041773d7b710d48: UndoDeltaBlockGCOp(c68032ba399547e3bd272f424927fbce) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":177,"lbm_reads_lt_1ms":4}
I20240503 08:14:23.782991 28179 maintenance_manager.cc:419] P 1b411c6bb25549368041773d7b710d48: Scheduling UndoDeltaBlockGCOp(07866de372e541ec9dca8a3d42f8b0ab): 4583 bytes on disk
I20240503 08:14:23.783612 28070 maintenance_manager.cc:643] P 1b411c6bb25549368041773d7b710d48: UndoDeltaBlockGCOp(07866de372e541ec9dca8a3d42f8b0ab) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240503 08:14:23.790139 28309 maintenance_manager.cc:419] P aae860d7ae2b4002bf7794c278f9cc56: Scheduling UndoDeltaBlockGCOp(07866de372e541ec9dca8a3d42f8b0ab): 4583 bytes on disk
I20240503 08:14:23.790797 28200 maintenance_manager.cc:643] P aae860d7ae2b4002bf7794c278f9cc56: UndoDeltaBlockGCOp(07866de372e541ec9dca8a3d42f8b0ab) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240503 08:14:23.794986 28309 maintenance_manager.cc:419] P aae860d7ae2b4002bf7794c278f9cc56: Scheduling FlushMRSOp(c68032ba399547e3bd272f424927fbce): perf score=0.033502
I20240503 08:14:23.871094 28200 maintenance_manager.cc:643] P aae860d7ae2b4002bf7794c278f9cc56: FlushMRSOp(c68032ba399547e3bd272f424927fbce) complete. Timing: real 0.076s user 0.037s sys 0.014s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":37,"dirs.run_cpu_time_us":174,"dirs.run_wall_time_us":1519,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":936,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":17381,"threads_started":1,"wal-append.queue_time_us":17266}
I20240503 08:14:23.874724 28309 maintenance_manager.cc:419] P aae860d7ae2b4002bf7794c278f9cc56: Scheduling UndoDeltaBlockGCOp(c68032ba399547e3bd272f424927fbce): 4839 bytes on disk
I20240503 08:14:23.875888 28200 maintenance_manager.cc:643] P aae860d7ae2b4002bf7794c278f9cc56: UndoDeltaBlockGCOp(c68032ba399547e3bd272f424927fbce) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20240503 08:14:23.894496 28330 maintenance_manager.cc:643] P d09a11a03989421b9f28019dc1c14cc2: FlushMRSOp(07866de372e541ec9dca8a3d42f8b0ab) complete. Timing: real 0.200s user 0.054s sys 0.033s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":71,"dirs.run_cpu_time_us":224,"dirs.run_wall_time_us":1886,"drs_written":1,"lbm_read_time_us":507,"lbm_reads_lt_1ms":4,"lbm_write_time_us":840,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":35699,"threads_started":1,"wal-append.queue_time_us":35799}
I20240503 08:14:23.897390 28439 maintenance_manager.cc:419] P d09a11a03989421b9f28019dc1c14cc2: Scheduling UndoDeltaBlockGCOp(599cb18c58a74e93a3e0334cc43a73d4): 4923 bytes on disk
I20240503 08:14:23.898672 28330 maintenance_manager.cc:643] P d09a11a03989421b9f28019dc1c14cc2: UndoDeltaBlockGCOp(599cb18c58a74e93a3e0334cc43a73d4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
I20240503 08:14:23.902670 28439 maintenance_manager.cc:419] P d09a11a03989421b9f28019dc1c14cc2: Scheduling UndoDeltaBlockGCOp(07866de372e541ec9dca8a3d42f8b0ab): 4583 bytes on disk
I20240503 08:14:23.903332 28330 maintenance_manager.cc:643] P d09a11a03989421b9f28019dc1c14cc2: UndoDeltaBlockGCOp(07866de372e541ec9dca8a3d42f8b0ab) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240503 08:14:23.943728 30895 consensus_queue.cc:237] T d61ad2e11c4544bfbdef7df1e92d2d11 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:14:23.945736 28389 raft_consensus.cc:1261] T d61ad2e11c4544bfbdef7df1e92d2d11 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:23.945758 30761 raft_consensus.cc:1261] T d61ad2e11c4544bfbdef7df1e92d2d11 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:23.946324 11059 consensus_queue.cc:1035] T d61ad2e11c4544bfbdef7df1e92d2d11 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:23.947736 11059 consensus_queue.cc:1035] T d61ad2e11c4544bfbdef7df1e92d2d11 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:23.949250 11059 raft_consensus.cc:2932] T d61ad2e11c4544bfbdef7df1e92d2d11 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:14:23.950713 30761 raft_consensus.cc:2932] T d61ad2e11c4544bfbdef7df1e92d2d11 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:14:23.950750 28389 raft_consensus.cc:2932] T d61ad2e11c4544bfbdef7df1e92d2d11 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:14:23.957973 27853 catalog_manager.cc:5484] T d61ad2e11c4544bfbdef7df1e92d2d11 P ce1496a47d084365ad786cd09e472a2c reported cstate change: config changed from index 2 to 3, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) evicted. New cstate: current_term: 1 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:14:23.965373 28243 tablet_service.cc:1514] Processing DeleteTablet for tablet d61ad2e11c4544bfbdef7df1e92d2d11 with delete_type TABLET_DATA_TOMBSTONED (TS aae860d7ae2b4002bf7794c278f9cc56 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:41960
I20240503 08:14:23.966029 11114 tablet_replica.cc:330] T d61ad2e11c4544bfbdef7df1e92d2d11 P aae860d7ae2b4002bf7794c278f9cc56: stopping tablet replica
I20240503 08:14:23.966233 11114 raft_consensus.cc:2229] T d61ad2e11c4544bfbdef7df1e92d2d11 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:23.966784 11114 raft_consensus.cc:2258] T d61ad2e11c4544bfbdef7df1e92d2d11 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:23.967454 11114 ts_tablet_manager.cc:1875] T d61ad2e11c4544bfbdef7df1e92d2d11 P aae860d7ae2b4002bf7794c278f9cc56: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:23.975914 11114 ts_tablet_manager.cc:1888] T d61ad2e11c4544bfbdef7df1e92d2d11 P aae860d7ae2b4002bf7794c278f9cc56: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:14:23.976133 11114 log.cc:1196] T d61ad2e11c4544bfbdef7df1e92d2d11 P aae860d7ae2b4002bf7794c278f9cc56: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal/wals/d61ad2e11c4544bfbdef7df1e92d2d11
I20240503 08:14:23.976910 27847 catalog_manager.cc:4830] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): tablet d61ad2e11c4544bfbdef7df1e92d2d11 (table rebalancer_extra_table_56 [id=0d91b0f1ea854f178cc99c7e43f0195f]) successfully deleted
I20240503 08:14:24.323182 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:24.356846 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:24.360467 31141 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:24.362234 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:24.387450 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:24.416685 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:24.421238 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:24.436403 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:14:24.620934 27999 consensus_queue.cc:237] T 45f47478f49149b895060791655503ae P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:14:24.636348 28389 raft_consensus.cc:1261] T 45f47478f49149b895060791655503ae P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:24.636449 28133 raft_consensus.cc:1261] T 45f47478f49149b895060791655503ae P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:24.637001 10645 consensus_queue.cc:1035] T 45f47478f49149b895060791655503ae P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:24.637579 10645 consensus_queue.cc:1035] T 45f47478f49149b895060791655503ae P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:24.658316 10645 raft_consensus.cc:2932] T 45f47478f49149b895060791655503ae P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:14:24.663069 28133 raft_consensus.cc:2932] T 45f47478f49149b895060791655503ae P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:14:24.664012 28389 raft_consensus.cc:2932] T 45f47478f49149b895060791655503ae P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:14:24.670930 27861 catalog_manager.cc:5484] T 45f47478f49149b895060791655503ae P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: config changed from index 2 to 3, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) evicted. New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:14:24.703464 31006 tablet_service.cc:1514] Processing DeleteTablet for tablet 45f47478f49149b895060791655503ae with delete_type TABLET_DATA_TOMBSTONED (TS ced21ca983b749d0b4d4f6a0e2426bf8 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47020
I20240503 08:14:24.714237 11200 tablet_replica.cc:330] T 45f47478f49149b895060791655503ae P ced21ca983b749d0b4d4f6a0e2426bf8: stopping tablet replica
I20240503 08:14:24.714464 11200 raft_consensus.cc:2229] T 45f47478f49149b895060791655503ae P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:24.714910 11200 raft_consensus.cc:2258] T 45f47478f49149b895060791655503ae P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:24.716717 11200 ts_tablet_manager.cc:1875] T 45f47478f49149b895060791655503ae P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:24.723798 11200 ts_tablet_manager.cc:1888] T 45f47478f49149b895060791655503ae P ced21ca983b749d0b4d4f6a0e2426bf8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:14:24.724000 11200 log.cc:1196] T 45f47478f49149b895060791655503ae P ced21ca983b749d0b4d4f6a0e2426bf8: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal/wals/45f47478f49149b895060791655503ae
I20240503 08:14:24.724889 27847 catalog_manager.cc:4830] TS ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): tablet 45f47478f49149b895060791655503ae (table rebalancer_extra_table_16 [id=c740702474bd4c328043a94e39c2db1b]) successfully deleted
I20240503 08:14:25.016238 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:25.061270 31141 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:25.072216 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:14:25.091146 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:25.113937 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:25.147452 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:25.148404 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:25.156935 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:25.327683 28390 consensus_queue.cc:237] T 720f36100d794158be13ff02595e6ddf P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:14:25.329885 28133 raft_consensus.cc:1261] T 720f36100d794158be13ff02595e6ddf P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:25.329931 30762 raft_consensus.cc:1261] T 720f36100d794158be13ff02595e6ddf P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:25.330472 10747 consensus_queue.cc:1035] T 720f36100d794158be13ff02595e6ddf P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:25.330735 10573 consensus_queue.cc:1035] T 720f36100d794158be13ff02595e6ddf P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:25.333438 10747 raft_consensus.cc:2932] T 720f36100d794158be13ff02595e6ddf P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:14:25.334163 28133 raft_consensus.cc:2932] T 720f36100d794158be13ff02595e6ddf P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:14:25.337278 30762 raft_consensus.cc:2932] T 720f36100d794158be13ff02595e6ddf P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:14:25.341697 27861 catalog_manager.cc:5484] T 720f36100d794158be13ff02595e6ddf P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: config changed from index 2 to 3, VOTER b00f244c69df401997ea99d492218a98 (127.20.230.72) evicted. New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:14:25.348385 31141 tablet_service.cc:1514] Processing DeleteTablet for tablet 720f36100d794158be13ff02595e6ddf with delete_type TABLET_DATA_TOMBSTONED (TS b00f244c69df401997ea99d492218a98 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45814
I20240503 08:14:25.348932 11288 tablet_replica.cc:330] T 720f36100d794158be13ff02595e6ddf P b00f244c69df401997ea99d492218a98: stopping tablet replica
I20240503 08:14:25.349139 11288 raft_consensus.cc:2229] T 720f36100d794158be13ff02595e6ddf P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:25.349360 11288 raft_consensus.cc:2258] T 720f36100d794158be13ff02595e6ddf P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:25.350239 11288 ts_tablet_manager.cc:1875] T 720f36100d794158be13ff02595e6ddf P b00f244c69df401997ea99d492218a98: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:25.359426 11288 ts_tablet_manager.cc:1888] T 720f36100d794158be13ff02595e6ddf P b00f244c69df401997ea99d492218a98: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:14:25.359561 11288 log.cc:1196] T 720f36100d794158be13ff02595e6ddf P b00f244c69df401997ea99d492218a98: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal/wals/720f36100d794158be13ff02595e6ddf
I20240503 08:14:25.360294 27846 catalog_manager.cc:4830] TS b00f244c69df401997ea99d492218a98 (127.20.230.72:40893): tablet 720f36100d794158be13ff02595e6ddf (table rebalancer_extra_table_14 [id=6db67fd7274948c0858a325398f6acbc]) successfully deleted
I20240503 08:14:25.661538 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:25.685909 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:25.713264 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:25.758713 31141 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:25.774469 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:25.857535 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 08:14:25.861096 30962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.68:46211, user_credentials={real_user=slave}} blocked reactor thread for 47959.3us
I20240503 08:14:25.869540 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:14:25.871555 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:25.978169 30762 consensus_queue.cc:237] T 6c2824aa878845e098f600f1fcfea087 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:14:25.980137 28133 raft_consensus.cc:1261] T 6c2824aa878845e098f600f1fcfea087 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:25.980274 31160 raft_consensus.cc:1261] T 6c2824aa878845e098f600f1fcfea087 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:25.980650 10995 consensus_queue.cc:1035] T 6c2824aa878845e098f600f1fcfea087 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:25.980908 10052 consensus_queue.cc:1035] T 6c2824aa878845e098f600f1fcfea087 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:25.983633 10995 raft_consensus.cc:2932] T 6c2824aa878845e098f600f1fcfea087 P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:14:25.984153 31160 raft_consensus.cc:2932] T 6c2824aa878845e098f600f1fcfea087 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:14:25.986379 28133 raft_consensus.cc:2932] T 6c2824aa878845e098f600f1fcfea087 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:14:25.989295 27854 catalog_manager.cc:5484] T 6c2824aa878845e098f600f1fcfea087 P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: config changed from index 2 to 3, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:14:26.019945 30874 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c2824aa878845e098f600f1fcfea087 with delete_type TABLET_DATA_TOMBSTONED (TS ce1496a47d084365ad786cd09e472a2c not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:34426
I20240503 08:14:26.033965 11381 tablet_replica.cc:330] T 6c2824aa878845e098f600f1fcfea087 P ce1496a47d084365ad786cd09e472a2c: stopping tablet replica
I20240503 08:14:26.034168 11381 raft_consensus.cc:2229] T 6c2824aa878845e098f600f1fcfea087 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:26.034346 11381 raft_consensus.cc:2258] T 6c2824aa878845e098f600f1fcfea087 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:26.035305 11381 ts_tablet_manager.cc:1875] T 6c2824aa878845e098f600f1fcfea087 P ce1496a47d084365ad786cd09e472a2c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:26.043445 11381 ts_tablet_manager.cc:1888] T 6c2824aa878845e098f600f1fcfea087 P ce1496a47d084365ad786cd09e472a2c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:14:26.043610 11381 log.cc:1196] T 6c2824aa878845e098f600f1fcfea087 P ce1496a47d084365ad786cd09e472a2c: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal/wals/6c2824aa878845e098f600f1fcfea087
I20240503 08:14:26.044298 27846 catalog_manager.cc:4830] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): tablet 6c2824aa878845e098f600f1fcfea087 (table rebalancer_extra_table_2 [id=7358500a74314a5a8795ddb89337f666]) successfully deleted
I20240503 08:14:26.158663 28309 maintenance_manager.cc:419] P aae860d7ae2b4002bf7794c278f9cc56: Scheduling FlushMRSOp(3d0498256b5f4b8fb1681759429f10ed): perf score=0.033345
I20240503 08:14:26.191265 28439 maintenance_manager.cc:419] P d09a11a03989421b9f28019dc1c14cc2: Scheduling FlushMRSOp(3042212758bf4aef8be5c5d12d10113c): perf score=0.033356
I20240503 08:14:26.240286 28200 maintenance_manager.cc:643] P aae860d7ae2b4002bf7794c278f9cc56: FlushMRSOp(3d0498256b5f4b8fb1681759429f10ed) complete. Timing: real 0.081s user 0.035s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":9294,"dirs.run_cpu_time_us":253,"dirs.run_wall_time_us":1783,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":916,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":14433,"threads_started":2,"wal-append.queue_time_us":5406}
I20240503 08:14:26.243882 28309 maintenance_manager.cc:419] P aae860d7ae2b4002bf7794c278f9cc56: Scheduling FlushMRSOp(8fc87f7f3c724cd4a270583ddebd87aa): perf score=0.033366
I20240503 08:14:26.288424 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:26.310523 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:26.316126 28179 maintenance_manager.cc:419] P 1b411c6bb25549368041773d7b710d48: Scheduling FlushMRSOp(3d0498256b5f4b8fb1681759429f10ed): perf score=0.033391
I20240503 08:14:26.316469 28049 maintenance_manager.cc:419] P 833582ff68c94269be364ebd187a75db: Scheduling FlushMRSOp(3d0498256b5f4b8fb1681759429f10ed): perf score=0.033393
I20240503 08:14:26.326252 28200 maintenance_manager.cc:643] P aae860d7ae2b4002bf7794c278f9cc56: FlushMRSOp(8fc87f7f3c724cd4a270583ddebd87aa) complete. Timing: real 0.082s user 0.047s sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":69,"dirs.run_cpu_time_us":211,"dirs.run_wall_time_us":2769,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":768,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":2531,"threads_started":1,"wal-append.queue_time_us":2603}
I20240503 08:14:26.330271 28309 maintenance_manager.cc:419] P aae860d7ae2b4002bf7794c278f9cc56: Scheduling UndoDeltaBlockGCOp(3d0498256b5f4b8fb1681759429f10ed): 4923 bytes on disk
I20240503 08:14:26.332088 28200 maintenance_manager.cc:643] P aae860d7ae2b4002bf7794c278f9cc56: UndoDeltaBlockGCOp(3d0498256b5f4b8fb1681759429f10ed) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
I20240503 08:14:26.335719 28309 maintenance_manager.cc:419] P aae860d7ae2b4002bf7794c278f9cc56: Scheduling UndoDeltaBlockGCOp(8fc87f7f3c724cd4a270583ddebd87aa): 4838 bytes on disk
I20240503 08:14:26.336212 28200 maintenance_manager.cc:643] P aae860d7ae2b4002bf7794c278f9cc56: UndoDeltaBlockGCOp(8fc87f7f3c724cd4a270583ddebd87aa) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20240503 08:14:26.342155 31141 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:26.359151 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:26.385844 27940 maintenance_manager.cc:643] P 833582ff68c94269be364ebd187a75db: FlushMRSOp(3d0498256b5f4b8fb1681759429f10ed) complete. Timing: real 0.068s user 0.025s sys 0.032s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":10195,"dirs.run_cpu_time_us":245,"dirs.run_wall_time_us":1487,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":630,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":20041,"threads_started":2,"wal-append.queue_time_us":9838}
I20240503 08:14:26.388665 28049 maintenance_manager.cc:419] P 833582ff68c94269be364ebd187a75db: Scheduling FlushMRSOp(3042212758bf4aef8be5c5d12d10113c): perf score=0.033410
I20240503 08:14:26.401170 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:14:26.413313 28330 maintenance_manager.cc:643] P d09a11a03989421b9f28019dc1c14cc2: FlushMRSOp(3042212758bf4aef8be5c5d12d10113c) complete. Timing: real 0.221s user 0.055s sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":1144,"dirs.run_cpu_time_us":161,"dirs.run_wall_time_us":1899,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":815,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":1872,"threads_started":2,"wal-append.queue_time_us":273}
I20240503 08:14:26.416117 28439 maintenance_manager.cc:419] P d09a11a03989421b9f28019dc1c14cc2: Scheduling UndoDeltaBlockGCOp(3042212758bf4aef8be5c5d12d10113c): 4581 bytes on disk
I20240503 08:14:26.417415 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:26.429972 28330 maintenance_manager.cc:643] P d09a11a03989421b9f28019dc1c14cc2: UndoDeltaBlockGCOp(3042212758bf4aef8be5c5d12d10113c) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":1456,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3}
I20240503 08:14:26.449378 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:26.458293 27940 maintenance_manager.cc:643] P 833582ff68c94269be364ebd187a75db: FlushMRSOp(3042212758bf4aef8be5c5d12d10113c) complete. Timing: real 0.069s user 0.055s sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":4918,"dirs.run_cpu_time_us":223,"dirs.run_wall_time_us":1845,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":824,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":6486,"threads_started":1,"wal-append.queue_time_us":6388}
I20240503 08:14:26.462007 28049 maintenance_manager.cc:419] P 833582ff68c94269be364ebd187a75db: Scheduling UndoDeltaBlockGCOp(3d0498256b5f4b8fb1681759429f10ed): 4923 bytes on disk
I20240503 08:14:26.462596 27940 maintenance_manager.cc:643] P 833582ff68c94269be364ebd187a75db: UndoDeltaBlockGCOp(3d0498256b5f4b8fb1681759429f10ed) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
I20240503 08:14:26.464954 28049 maintenance_manager.cc:419] P 833582ff68c94269be364ebd187a75db: Scheduling UndoDeltaBlockGCOp(3042212758bf4aef8be5c5d12d10113c): 4581 bytes on disk
I20240503 08:14:26.465446 27940 maintenance_manager.cc:643] P 833582ff68c94269be364ebd187a75db: UndoDeltaBlockGCOp(3042212758bf4aef8be5c5d12d10113c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
I20240503 08:14:26.488257 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:26.522045 28070 maintenance_manager.cc:643] P 1b411c6bb25549368041773d7b710d48: FlushMRSOp(3d0498256b5f4b8fb1681759429f10ed) complete. Timing: real 0.206s user 0.037s sys 0.017s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":16162,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":4957,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":813,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":40861,"threads_started":2,"wal-append.queue_time_us":24465}
I20240503 08:14:26.524838 28179 maintenance_manager.cc:419] P 1b411c6bb25549368041773d7b710d48: Scheduling FlushMRSOp(8fc87f7f3c724cd4a270583ddebd87aa): perf score=0.033445
I20240503 08:14:26.633579 28070 maintenance_manager.cc:643] P 1b411c6bb25549368041773d7b710d48: FlushMRSOp(8fc87f7f3c724cd4a270583ddebd87aa) complete. Timing: real 0.108s user 0.035s sys 0.005s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":1443,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":690,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":2747,"threads_started":1,"wal-append.queue_time_us":2838}
I20240503 08:14:26.636525 28179 maintenance_manager.cc:419] P 1b411c6bb25549368041773d7b710d48: Scheduling UndoDeltaBlockGCOp(3d0498256b5f4b8fb1681759429f10ed): 4923 bytes on disk
I20240503 08:14:26.637224 28070 maintenance_manager.cc:643] P 1b411c6bb25549368041773d7b710d48: UndoDeltaBlockGCOp(3d0498256b5f4b8fb1681759429f10ed) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20240503 08:14:26.645980 28179 maintenance_manager.cc:419] P 1b411c6bb25549368041773d7b710d48: Scheduling UndoDeltaBlockGCOp(8fc87f7f3c724cd4a270583ddebd87aa): 4838 bytes on disk
I20240503 08:14:26.650184 28070 maintenance_manager.cc:643] P 1b411c6bb25549368041773d7b710d48: UndoDeltaBlockGCOp(8fc87f7f3c724cd4a270583ddebd87aa) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
I20240503 08:14:26.706156 28133 consensus_queue.cc:237] T 7083414bc16c4d499c97b93ca4e3f631 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:14:26.707767 30762 raft_consensus.cc:1261] T 7083414bc16c4d499c97b93ca4e3f631 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:26.707881 28390 raft_consensus.cc:1261] T 7083414bc16c4d499c97b93ca4e3f631 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:26.708281 10938 consensus_queue.cc:1035] T 7083414bc16c4d499c97b93ca4e3f631 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:26.708539 11441 consensus_queue.cc:1035] T 7083414bc16c4d499c97b93ca4e3f631 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:26.712908 10688 raft_consensus.cc:2932] T 7083414bc16c4d499c97b93ca4e3f631 P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:14:26.713529 30762 raft_consensus.cc:2932] T 7083414bc16c4d499c97b93ca4e3f631 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:14:26.713529 28390 raft_consensus.cc:2932] T 7083414bc16c4d499c97b93ca4e3f631 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:14:26.724531 27862 catalog_manager.cc:5484] T 7083414bc16c4d499c97b93ca4e3f631 P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: config changed from index 4 to 5, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) evicted. New cstate: current_term: 1 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:14:26.738544 28243 tablet_service.cc:1514] Processing DeleteTablet for tablet 7083414bc16c4d499c97b93ca4e3f631 with delete_type TABLET_DATA_TOMBSTONED (TS aae860d7ae2b4002bf7794c278f9cc56 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41960
I20240503 08:14:26.744669 11469 tablet_replica.cc:330] T 7083414bc16c4d499c97b93ca4e3f631 P aae860d7ae2b4002bf7794c278f9cc56: stopping tablet replica
I20240503 08:14:26.744887 11469 raft_consensus.cc:2229] T 7083414bc16c4d499c97b93ca4e3f631 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:26.745150 11469 raft_consensus.cc:2258] T 7083414bc16c4d499c97b93ca4e3f631 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:26.745857 11469 ts_tablet_manager.cc:1875] T 7083414bc16c4d499c97b93ca4e3f631 P aae860d7ae2b4002bf7794c278f9cc56: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:26.766409 11469 ts_tablet_manager.cc:1888] T 7083414bc16c4d499c97b93ca4e3f631 P aae860d7ae2b4002bf7794c278f9cc56: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:14:26.766546 11469 log.cc:1196] T 7083414bc16c4d499c97b93ca4e3f631 P aae860d7ae2b4002bf7794c278f9cc56: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal/wals/7083414bc16c4d499c97b93ca4e3f631
I20240503 08:14:26.767186 27847 catalog_manager.cc:4830] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): tablet 7083414bc16c4d499c97b93ca4e3f631 (table rebalance_test_table_13 [id=6e8538506685489f8a1e931a2a781746]) successfully deleted
I20240503 08:14:26.961428 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:26.994411 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:27.009862 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:27.052161 31141 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:27.063056 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:27.086026 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:14:27.179634 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:27.247625 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:27.331568 28133 consensus_queue.cc:237] T fe4d26ee022b424195e836834bd2f051 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:27.334101 31025 raft_consensus.cc:1261] T fe4d26ee022b424195e836834bd2f051 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:27.334407 27999 raft_consensus.cc:1261] T fe4d26ee022b424195e836834bd2f051 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:27.334846 10938 consensus_queue.cc:1035] T fe4d26ee022b424195e836834bd2f051 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:27.335449 11539 consensus_queue.cc:1035] T fe4d26ee022b424195e836834bd2f051 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:27.337746 10938 raft_consensus.cc:2932] T fe4d26ee022b424195e836834bd2f051 P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:14:27.338429 27999 raft_consensus.cc:2932] T fe4d26ee022b424195e836834bd2f051 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:14:27.338647 31025 raft_consensus.cc:2932] T fe4d26ee022b424195e836834bd2f051 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:14:27.345456 27856 catalog_manager.cc:5484] T fe4d26ee022b424195e836834bd2f051 P 833582ff68c94269be364ebd187a75db reported cstate change: config changed from index 4 to 5, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) evicted. New cstate: current_term: 1 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:14:27.350831 28243 tablet_service.cc:1514] Processing DeleteTablet for tablet fe4d26ee022b424195e836834bd2f051 with delete_type TABLET_DATA_TOMBSTONED (TS aae860d7ae2b4002bf7794c278f9cc56 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41960
I20240503 08:14:27.354169 11545 tablet_replica.cc:330] T fe4d26ee022b424195e836834bd2f051 P aae860d7ae2b4002bf7794c278f9cc56: stopping tablet replica
I20240503 08:14:27.354382 11545 raft_consensus.cc:2229] T fe4d26ee022b424195e836834bd2f051 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:27.354682 11545 raft_consensus.cc:2258] T fe4d26ee022b424195e836834bd2f051 P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:27.355391 11545 ts_tablet_manager.cc:1875] T fe4d26ee022b424195e836834bd2f051 P aae860d7ae2b4002bf7794c278f9cc56: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:27.361440 11545 ts_tablet_manager.cc:1888] T fe4d26ee022b424195e836834bd2f051 P aae860d7ae2b4002bf7794c278f9cc56: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:14:27.361582 11545 log.cc:1196] T fe4d26ee022b424195e836834bd2f051 P aae860d7ae2b4002bf7794c278f9cc56: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal/wals/fe4d26ee022b424195e836834bd2f051
I20240503 08:14:27.362260 27847 catalog_manager.cc:4830] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): tablet fe4d26ee022b424195e836834bd2f051 (table rebalance_test_table_15 [id=261c96b04fc141ddb92650c24ce76fa8]) successfully deleted
I20240503 08:14:27.606274 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:27.618854 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:27.655966 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:27.721697 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:27.722729 31141 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:27.723873 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:27.743363 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:27.818439 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:14:27.936925 30761 consensus_queue.cc:237] T 2e511a3ab0ee4eabb22240de99690603 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:14:27.938779 28390 raft_consensus.cc:1261] T 2e511a3ab0ee4eabb22240de99690603 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:27.939291 30895 raft_consensus.cc:1261] T 2e511a3ab0ee4eabb22240de99690603 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:27.939278 10995 consensus_queue.cc:1035] T 2e511a3ab0ee4eabb22240de99690603 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:27.939859 10995 consensus_queue.cc:1035] T 2e511a3ab0ee4eabb22240de99690603 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:27.941757 11366 raft_consensus.cc:2932] T 2e511a3ab0ee4eabb22240de99690603 P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:14:27.942395 28390 raft_consensus.cc:2932] T 2e511a3ab0ee4eabb22240de99690603 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:14:27.942835 30893 raft_consensus.cc:2932] T 2e511a3ab0ee4eabb22240de99690603 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:14:27.949349 27859 catalog_manager.cc:5484] T 2e511a3ab0ee4eabb22240de99690603 P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: config changed from index 2 to 3, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:14:27.956327 28111 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e511a3ab0ee4eabb22240de99690603 with delete_type TABLET_DATA_TOMBSTONED (TS 1b411c6bb25549368041773d7b710d48 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:57912
I20240503 08:14:27.962213 11631 tablet_replica.cc:330] T 2e511a3ab0ee4eabb22240de99690603 P 1b411c6bb25549368041773d7b710d48: stopping tablet replica
I20240503 08:14:27.962406 11631 raft_consensus.cc:2229] T 2e511a3ab0ee4eabb22240de99690603 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:27.962594 11631 raft_consensus.cc:2258] T 2e511a3ab0ee4eabb22240de99690603 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:27.963164 11631 ts_tablet_manager.cc:1875] T 2e511a3ab0ee4eabb22240de99690603 P 1b411c6bb25549368041773d7b710d48: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:27.968693 11631 ts_tablet_manager.cc:1888] T 2e511a3ab0ee4eabb22240de99690603 P 1b411c6bb25549368041773d7b710d48: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:14:27.968811 11631 log.cc:1196] T 2e511a3ab0ee4eabb22240de99690603 P 1b411c6bb25549368041773d7b710d48: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/2e511a3ab0ee4eabb22240de99690603
I20240503 08:14:27.969457 27848 catalog_manager.cc:4830] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): tablet 2e511a3ab0ee4eabb22240de99690603 (table rebalancer_extra_table_46 [id=b8de6da1757347c7b752a17ffa18b098]) successfully deleted
I20240503 08:14:28.261675 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:28.266101 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:14:28.275378 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:28.287320 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:28.298918 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:28.299419 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:28.300225 31141 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:28.329622 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:28.368425 28309 maintenance_manager.cc:419] P aae860d7ae2b4002bf7794c278f9cc56: Scheduling FlushMRSOp(ca38671a760d4534ab903579e0487772): perf score=0.033337
I20240503 08:14:28.421811 28179 maintenance_manager.cc:419] P 1b411c6bb25549368041773d7b710d48: Scheduling FlushMRSOp(7083414bc16c4d499c97b93ca4e3f631): perf score=0.033354
I20240503 08:14:28.466702 28439 maintenance_manager.cc:419] P d09a11a03989421b9f28019dc1c14cc2: Scheduling FlushMRSOp(7083414bc16c4d499c97b93ca4e3f631): perf score=0.033363
I20240503 08:14:28.496091 28049 maintenance_manager.cc:419] P 833582ff68c94269be364ebd187a75db: Scheduling FlushMRSOp(ca38671a760d4534ab903579e0487772): perf score=0.033373
I20240503 08:14:28.501792 28200 maintenance_manager.cc:643] P aae860d7ae2b4002bf7794c278f9cc56: FlushMRSOp(ca38671a760d4534ab903579e0487772) complete. Timing: real 0.133s user 0.043s sys 0.015s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":11842,"dirs.run_cpu_time_us":211,"dirs.run_wall_time_us":32793,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":787,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":38473,"threads_started":2,"wal-append.queue_time_us":26162}
I20240503 08:14:28.537065 28309 maintenance_manager.cc:419] P aae860d7ae2b4002bf7794c278f9cc56: Scheduling UndoDeltaBlockGCOp(ca38671a760d4534ab903579e0487772): 4838 bytes on disk
I20240503 08:14:28.546479 28200 maintenance_manager.cc:643] P aae860d7ae2b4002bf7794c278f9cc56: UndoDeltaBlockGCOp(ca38671a760d4534ab903579e0487772) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
I20240503 08:14:28.561048 28070 maintenance_manager.cc:643] P 1b411c6bb25549368041773d7b710d48: FlushMRSOp(7083414bc16c4d499c97b93ca4e3f631) complete. Timing: real 0.139s user 0.056s sys 0.009s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":7756,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":2830,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":829,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":19040,"threads_started":2,"wal-append.queue_time_us":11225}
I20240503 08:14:28.564783 28179 maintenance_manager.cc:419] P 1b411c6bb25549368041773d7b710d48: Scheduling UndoDeltaBlockGCOp(7083414bc16c4d499c97b93ca4e3f631): 4923 bytes on disk
I20240503 08:14:28.565387 28070 maintenance_manager.cc:643] P 1b411c6bb25549368041773d7b710d48: UndoDeltaBlockGCOp(7083414bc16c4d499c97b93ca4e3f631) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
I20240503 08:14:28.568042 28179 maintenance_manager.cc:419] P 1b411c6bb25549368041773d7b710d48: Scheduling FlushMRSOp(7a101bee3af34166848ee7fa7c733892): perf score=0.033384
I20240503 08:14:28.594477 27940 maintenance_manager.cc:643] P 833582ff68c94269be364ebd187a75db: FlushMRSOp(ca38671a760d4534ab903579e0487772) complete. Timing: real 0.098s user 0.050s sys 0.025s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":4949,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":8490,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":827,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":13031,"threads_started":2,"wal-append.queue_time_us":7914}
I20240503 08:14:28.597582 28049 maintenance_manager.cc:419] P 833582ff68c94269be364ebd187a75db: Scheduling UndoDeltaBlockGCOp(ca38671a760d4534ab903579e0487772): 4838 bytes on disk
I20240503 08:14:28.598150 27940 maintenance_manager.cc:643] P 833582ff68c94269be364ebd187a75db: UndoDeltaBlockGCOp(ca38671a760d4534ab903579e0487772) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
I20240503 08:14:28.603242 28070 maintenance_manager.cc:643] P 1b411c6bb25549368041773d7b710d48: FlushMRSOp(7a101bee3af34166848ee7fa7c733892) complete. Timing: real 0.035s user 0.028s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":35,"dirs.run_cpu_time_us":216,"dirs.run_wall_time_us":1354,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":822,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":230,"threads_started":1,"wal-append.queue_time_us":310}
I20240503 08:14:28.605490 28179 maintenance_manager.cc:419] P 1b411c6bb25549368041773d7b710d48: Scheduling UndoDeltaBlockGCOp(7a101bee3af34166848ee7fa7c733892): 4583 bytes on disk
I20240503 08:14:28.606019 28070 maintenance_manager.cc:643] P 1b411c6bb25549368041773d7b710d48: UndoDeltaBlockGCOp(7a101bee3af34166848ee7fa7c733892) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4}
I20240503 08:14:28.610533 28330 maintenance_manager.cc:643] P d09a11a03989421b9f28019dc1c14cc2: FlushMRSOp(7083414bc16c4d499c97b93ca4e3f631) complete. Timing: real 0.143s user 0.036s sys 0.013s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":3301,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":16976,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":948,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":11849,"threads_started":2,"wal-append.queue_time_us":992}
I20240503 08:14:28.613428 28439 maintenance_manager.cc:419] P d09a11a03989421b9f28019dc1c14cc2: Scheduling UndoDeltaBlockGCOp(7083414bc16c4d499c97b93ca4e3f631): 4923 bytes on disk
I20240503 08:14:28.615186 28330 maintenance_manager.cc:643] P d09a11a03989421b9f28019dc1c14cc2: UndoDeltaBlockGCOp(7083414bc16c4d499c97b93ca4e3f631) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
I20240503 08:14:28.619215 28439 maintenance_manager.cc:419] P d09a11a03989421b9f28019dc1c14cc2: Scheduling FlushMRSOp(7a101bee3af34166848ee7fa7c733892): perf score=0.033401
I20240503 08:14:28.675133 28330 maintenance_manager.cc:643] P d09a11a03989421b9f28019dc1c14cc2: FlushMRSOp(7a101bee3af34166848ee7fa7c733892) complete. Timing: real 0.054s user 0.038s sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":64,"dirs.run_cpu_time_us":166,"dirs.run_wall_time_us":1823,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":735,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":6357,"threads_started":1,"wal-append.queue_time_us":6439}
I20240503 08:14:28.678201 28439 maintenance_manager.cc:419] P d09a11a03989421b9f28019dc1c14cc2: Scheduling UndoDeltaBlockGCOp(7a101bee3af34166848ee7fa7c733892): 4583 bytes on disk
I20240503 08:14:28.678782 28330 maintenance_manager.cc:643] P d09a11a03989421b9f28019dc1c14cc2: UndoDeltaBlockGCOp(7a101bee3af34166848ee7fa7c733892) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20240503 08:14:28.716077 27999 consensus_queue.cc:237] T c68032ba399547e3bd272f424927fbce P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:14:28.717942 28133 raft_consensus.cc:1261] T c68032ba399547e3bd272f424927fbce P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:28.718247 30761 raft_consensus.cc:1261] T c68032ba399547e3bd272f424927fbce P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:28.718786 11522 consensus_queue.cc:1035] T c68032ba399547e3bd272f424927fbce P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:28.719328 11522 consensus_queue.cc:1035] T c68032ba399547e3bd272f424927fbce P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:28.722786 10936 raft_consensus.cc:2932] T c68032ba399547e3bd272f424927fbce P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:14:28.723632 28133 raft_consensus.cc:2932] T c68032ba399547e3bd272f424927fbce P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:14:28.727453 30761 raft_consensus.cc:2932] T c68032ba399547e3bd272f424927fbce P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:14:28.731492 27860 catalog_manager.cc:5484] T c68032ba399547e3bd272f424927fbce P 1b411c6bb25549368041773d7b710d48 reported cstate change: config changed from index 4 to 5, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) evicted. New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:14:28.776829 28243 tablet_service.cc:1514] Processing DeleteTablet for tablet c68032ba399547e3bd272f424927fbce with delete_type TABLET_DATA_TOMBSTONED (TS aae860d7ae2b4002bf7794c278f9cc56 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41960
I20240503 08:14:28.777726 11722 tablet_replica.cc:330] T c68032ba399547e3bd272f424927fbce P aae860d7ae2b4002bf7794c278f9cc56: stopping tablet replica
I20240503 08:14:28.778039 11722 raft_consensus.cc:2229] T c68032ba399547e3bd272f424927fbce P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:28.778359 11722 raft_consensus.cc:2258] T c68032ba399547e3bd272f424927fbce P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:28.779434 11722 ts_tablet_manager.cc:1875] T c68032ba399547e3bd272f424927fbce P aae860d7ae2b4002bf7794c278f9cc56: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:28.795437 11722 ts_tablet_manager.cc:1888] T c68032ba399547e3bd272f424927fbce P aae860d7ae2b4002bf7794c278f9cc56: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:14:28.795614 11722 log.cc:1196] T c68032ba399547e3bd272f424927fbce P aae860d7ae2b4002bf7794c278f9cc56: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal/wals/c68032ba399547e3bd272f424927fbce
I20240503 08:14:28.796582 27847 catalog_manager.cc:4830] TS aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): tablet c68032ba399547e3bd272f424927fbce (table rebalance_test_table_9 [id=eeeb4ab7272f4686b013f5d8f74fc917]) successfully deleted
I20240503 08:14:29.084633 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:29.085911 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:29.086939 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:29.114351 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:29.151495 31141 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:29.164477 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:29.212987 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:14:29.230779 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:29.377137 30762 consensus_queue.cc:237] T 71fa62206f744de0b97e3aa9ea753e37 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:14:29.382902 30895 raft_consensus.cc:1261] T 71fa62206f744de0b97e3aa9ea753e37 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:29.383436 11623 consensus_queue.cc:1035] T 71fa62206f744de0b97e3aa9ea753e37 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:29.385737 31161 raft_consensus.cc:1261] T 71fa62206f744de0b97e3aa9ea753e37 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:14:29.385792 11366 raft_consensus.cc:2932] T 71fa62206f744de0b97e3aa9ea753e37 P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:14:29.389353 11623 consensus_queue.cc:1035] T 71fa62206f744de0b97e3aa9ea753e37 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:29.389492 30895 raft_consensus.cc:2932] T 71fa62206f744de0b97e3aa9ea753e37 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:14:29.391587 31161 raft_consensus.cc:2932] T 71fa62206f744de0b97e3aa9ea753e37 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:14:29.395861 27860 catalog_manager.cc:5484] T 71fa62206f744de0b97e3aa9ea753e37 P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: config changed from index 2 to 3, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) evicted. New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:14:29.402076 28373 tablet_service.cc:1514] Processing DeleteTablet for tablet 71fa62206f744de0b97e3aa9ea753e37 with delete_type TABLET_DATA_TOMBSTONED (TS d09a11a03989421b9f28019dc1c14cc2 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:34786
I20240503 08:14:29.402617 11810 tablet_replica.cc:330] T 71fa62206f744de0b97e3aa9ea753e37 P d09a11a03989421b9f28019dc1c14cc2: stopping tablet replica
I20240503 08:14:29.402810 11810 raft_consensus.cc:2229] T 71fa62206f744de0b97e3aa9ea753e37 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:29.402987 11810 raft_consensus.cc:2258] T 71fa62206f744de0b97e3aa9ea753e37 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:29.403641 11810 ts_tablet_manager.cc:1875] T 71fa62206f744de0b97e3aa9ea753e37 P d09a11a03989421b9f28019dc1c14cc2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:29.409355 11810 ts_tablet_manager.cc:1888] T 71fa62206f744de0b97e3aa9ea753e37 P d09a11a03989421b9f28019dc1c14cc2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:14:29.409485 11810 log.cc:1196] T 71fa62206f744de0b97e3aa9ea753e37 P d09a11a03989421b9f28019dc1c14cc2: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/wals/71fa62206f744de0b97e3aa9ea753e37
I20240503 08:14:29.410537 27849 catalog_manager.cc:4830] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): tablet 71fa62206f744de0b97e3aa9ea753e37 (table rebalancer_extra_table_14 [id=6db67fd7274948c0858a325398f6acbc]) successfully deleted
I20240503 08:14:29.781486 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:29.792191 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:29.834971 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:29.883499 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:29.899704 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:29.911908 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:29.912401 31141 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:29.948832 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:14:30.065009 28389 consensus_queue.cc:237] T ba1fc93781714efb8eb7ec67c2fceb8a P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:14:30.066992 28261 raft_consensus.cc:1261] T ba1fc93781714efb8eb7ec67c2fceb8a P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:30.067065 30894 raft_consensus.cc:1261] T ba1fc93781714efb8eb7ec67c2fceb8a P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:30.067566 11614 consensus_queue.cc:1035] T ba1fc93781714efb8eb7ec67c2fceb8a P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:30.067835 11827 consensus_queue.cc:1035] T ba1fc93781714efb8eb7ec67c2fceb8a P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:30.070785 11827 raft_consensus.cc:2932] T ba1fc93781714efb8eb7ec67c2fceb8a P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:14:30.071533 28261 raft_consensus.cc:2932] T ba1fc93781714efb8eb7ec67c2fceb8a P aae860d7ae2b4002bf7794c278f9cc56 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:14:30.074120 30893 raft_consensus.cc:2932] T ba1fc93781714efb8eb7ec67c2fceb8a P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:14:30.081126 27861 catalog_manager.cc:5484] T ba1fc93781714efb8eb7ec67c2fceb8a P ce1496a47d084365ad786cd09e472a2c reported cstate change: config changed from index 4 to 5, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) evicted. New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:14:30.087512 28111 tablet_service.cc:1514] Processing DeleteTablet for tablet ba1fc93781714efb8eb7ec67c2fceb8a with delete_type TABLET_DATA_TOMBSTONED (TS 1b411c6bb25549368041773d7b710d48 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57912
I20240503 08:14:30.088104 11906 tablet_replica.cc:330] T ba1fc93781714efb8eb7ec67c2fceb8a P 1b411c6bb25549368041773d7b710d48: stopping tablet replica
I20240503 08:14:30.088480 11906 raft_consensus.cc:2229] T ba1fc93781714efb8eb7ec67c2fceb8a P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:30.088771 11906 raft_consensus.cc:2258] T ba1fc93781714efb8eb7ec67c2fceb8a P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:30.089565 11906 ts_tablet_manager.cc:1875] T ba1fc93781714efb8eb7ec67c2fceb8a P 1b411c6bb25549368041773d7b710d48: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:30.097405 11906 ts_tablet_manager.cc:1888] T ba1fc93781714efb8eb7ec67c2fceb8a P 1b411c6bb25549368041773d7b710d48: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:14:30.097553 11906 log.cc:1196] T ba1fc93781714efb8eb7ec67c2fceb8a P 1b411c6bb25549368041773d7b710d48: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/wals/ba1fc93781714efb8eb7ec67c2fceb8a
I20240503 08:14:30.099222 27848 catalog_manager.cc:4830] TS 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): tablet ba1fc93781714efb8eb7ec67c2fceb8a (table rebalance_test_table_5 [id=490e37ef740f4e96b915339369c454ff]) successfully deleted
I20240503 08:14:30.440490 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:30.457237 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:30.466486 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:30.471699 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:30.535903 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:30.551113 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:14:30.621755 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:30.633428 31141 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:30.762446 31160 consensus_queue.cc:237] T 2401f8f101f94189aa3c0582f08cef88 P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:14:30.764451 28000 raft_consensus.cc:1261] T 2401f8f101f94189aa3c0582f08cef88 P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:14:30.764752 31026 raft_consensus.cc:1261] T 2401f8f101f94189aa3c0582f08cef88 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:14:30.765254 10051 consensus_queue.cc:1035] T 2401f8f101f94189aa3c0582f08cef88 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:30.765635 11144 consensus_queue.cc:1035] T 2401f8f101f94189aa3c0582f08cef88 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:30.767781 10051 raft_consensus.cc:2932] T 2401f8f101f94189aa3c0582f08cef88 P b00f244c69df401997ea99d492218a98 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:14:30.770040 28000 raft_consensus.cc:2932] T 2401f8f101f94189aa3c0582f08cef88 P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:14:30.770426 31026 raft_consensus.cc:2932] T 2401f8f101f94189aa3c0582f08cef88 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:14:30.774029 27853 catalog_manager.cc:5484] T 2401f8f101f94189aa3c0582f08cef88 P b00f244c69df401997ea99d492218a98 reported cstate change: config changed from index 2 to 4, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) evicted. New cstate: current_term: 2 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:14:30.790230 30874 tablet_service.cc:1514] Processing DeleteTablet for tablet 2401f8f101f94189aa3c0582f08cef88 with delete_type TABLET_DATA_TOMBSTONED (TS ce1496a47d084365ad786cd09e472a2c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34426
I20240503 08:14:30.790766 12017 tablet_replica.cc:330] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c: stopping tablet replica
I20240503 08:14:30.791759 12017 raft_consensus.cc:2229] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:30.791939 12017 raft_consensus.cc:2258] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:30.792562 12017 ts_tablet_manager.cc:1875] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:30.799981 12017 ts_tablet_manager.cc:1888] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240503 08:14:30.800128 12017 log.cc:1196] T 2401f8f101f94189aa3c0582f08cef88 P ce1496a47d084365ad786cd09e472a2c: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal/wals/2401f8f101f94189aa3c0582f08cef88
I20240503 08:14:30.800854 27846 catalog_manager.cc:4830] TS ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): tablet 2401f8f101f94189aa3c0582f08cef88 (table rebalancer_extra_table_56 [id=0d91b0f1ea854f178cc99c7e43f0195f]) successfully deleted
I20240503 08:14:30.810554 30894 consensus_queue.cc:237] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:30.812539 30761 raft_consensus.cc:1261] T 97149ad1d03847a085b2142519135a17 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:30.812851 31160 raft_consensus.cc:1261] T 97149ad1d03847a085b2142519135a17 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:14:30.812875 30828 consensus_peers.cc:485] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 97149ad1d03847a085b2142519135a17. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:30.813235 11227 consensus_queue.cc:1035] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:30.813459 10945 consensus_queue.cc:1035] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:30.816012 11989 raft_consensus.cc:2932] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:30.816946 30761 raft_consensus.cc:2932] T 97149ad1d03847a085b2142519135a17 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:30.822427 27853 catalog_manager.cc:5484] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c reported cstate change: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New cstate: current_term: 1 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:30.824771 31160 raft_consensus.cc:2932] T 97149ad1d03847a085b2142519135a17 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:30.856246 28389 consensus_queue.cc:237] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:14:30.858278 30761 raft_consensus.cc:1261] T e3a6b72060804beb81a177988d287f63 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:30.858655 28000 raft_consensus.cc:1261] T e3a6b72060804beb81a177988d287f63 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:30.858875 11614 consensus_queue.cc:1035] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 08:14:30.859127 28326 consensus_peers.cc:485] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2 -> Peer ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): Couldn't send request to peer ce1496a47d084365ad786cd09e472a2c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e3a6b72060804beb81a177988d287f63. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:30.859258 11827 consensus_queue.cc:1035] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:30.876973 11827 raft_consensus.cc:2932] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:14:30.877787 28000 raft_consensus.cc:2932] T e3a6b72060804beb81a177988d287f63 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:14:30.880967 30761 raft_consensus.cc:2932] T e3a6b72060804beb81a177988d287f63 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:14:30.884078 27853 catalog_manager.cc:5484] T e3a6b72060804beb81a177988d287f63 P 833582ff68c94269be364ebd187a75db reported cstate change: config changed from index -1 to 2, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New cstate: current_term: 1 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:14:31.141448 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:31.172286 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:31.177042 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:31.191931 31141 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:31.193907 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:14:31.210026 28439 maintenance_manager.cc:419] P d09a11a03989421b9f28019dc1c14cc2: Scheduling FlushMRSOp(33f0f4bc5722485a8f978a02367ca80e): perf score=0.033342
I20240503 08:14:31.229219 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:31.234699 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:31.258800 28243 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:31.330561 28309 maintenance_manager.cc:419] P aae860d7ae2b4002bf7794c278f9cc56: Scheduling FlushMRSOp(33f0f4bc5722485a8f978a02367ca80e): perf score=0.033375
I20240503 08:14:31.383999 28330 maintenance_manager.cc:643] P d09a11a03989421b9f28019dc1c14cc2: FlushMRSOp(33f0f4bc5722485a8f978a02367ca80e) complete. Timing: real 0.174s user 0.055s sys 0.036s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":42995,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":1689,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1327,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":51913,"threads_started":2,"wal-append.queue_time_us":8912}
I20240503 08:14:31.384356 28049 maintenance_manager.cc:419] P 833582ff68c94269be364ebd187a75db: Scheduling FlushMRSOp(fe4d26ee022b424195e836834bd2f051): perf score=0.033396
I20240503 08:14:31.386978 28439 maintenance_manager.cc:419] P d09a11a03989421b9f28019dc1c14cc2: Scheduling UndoDeltaBlockGCOp(33f0f4bc5722485a8f978a02367ca80e): 4838 bytes on disk
I20240503 08:14:31.388314 28179 maintenance_manager.cc:419] P 1b411c6bb25549368041773d7b710d48: Scheduling FlushMRSOp(fe4d26ee022b424195e836834bd2f051): perf score=0.033395
I20240503 08:14:31.402740 28330 maintenance_manager.cc:643] P d09a11a03989421b9f28019dc1c14cc2: UndoDeltaBlockGCOp(33f0f4bc5722485a8f978a02367ca80e) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":676,"lbm_reads_lt_1ms":4}
I20240503 08:14:31.440588 12115 ts_tablet_manager.cc:902] T 97149ad1d03847a085b2142519135a17 P aae860d7ae2b4002bf7794c278f9cc56: Initiating tablet copy from peer ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
F20240503 08:14:31.440897 12115 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-03b02a4425784a30b1007b3a14139d8d.queue_time_us, tablet-download-pool-03b02a4425784a30b1007b3a14139d8d.queue_time_us) (tablet-download-pool-03b02a4425784a30b1007b3a14139d8d.run_cpu_time_us, tablet-download-pool-03b02a4425784a30b1007b3a14139d8d.run_cpu_time_us) (tablet-download-pool-03b02a4425784a30b1007b3a14139d8d.run_wall_time_us, tablet-download-pool-03b02a4425784a30b1007b3a14139d8d.run_wall_time_us) (tablet-download-pool-17c6e3293aad4591a9c1395ebffff2e6.queue_time_us, tablet-download-pool-17c6e3293aad4591a9c1395ebffff2e6.queue_time_us) (tablet-download-pool-17c6e3293aad4591a9c1395ebffff2e6.run_cpu_time_us, tablet-download-pool-17c6e3293aad4591a9c1395ebffff2e6.run_cpu_time_us) (tablet-download-pool-17c6e3293aad4591a9c1395ebffff2e6.run_wall_time_us, tablet-download-pool-17c6e3293aad4591a9c1395ebffff2e6.run_wall_time_us) (tablet-download-pool-362590d3d0834d0b8a672ab552f8c209.queue_time_us, tablet-download-pool-362590d3d0834d0b8a672ab552f8c209.queue_time_us) (tablet-download-pool-362590d3d0834d0b8a672ab552f8c209.run_cpu_time_us, tablet-download-pool-362590d3d0834d0b8a672ab552f8c209.run_cpu_time_us) (tablet-download-pool-362590d3d0834d0b8a672ab552f8c209.run_wall_time_us, tablet-download-pool-362590d3d0834d0b8a672ab552f8c209.run_wall_time_us) (tablet-download-pool-393abed63f334ac4b71986573c86c4d4.queue_time_us, tablet-download-pool-393abed63f334ac4b71986573c86c4d4.queue_time_us) (tablet-download-pool-393abed63f334ac4b71986573c86c4d4.run_cpu_time_us, tablet-download-pool-393abed63f334ac4b71986573c86c4d4.run_cpu_time_us) (tablet-download-pool-393abed63f334ac4b71986573c86c4d4.run_wall_time_us, tablet-download-pool-393abed63f334ac4b71986573c86c4d4.run_wall_time_us) (tablet-download-pool-3bc8f4968b9743baab5d25c8c7237c9c.queue_time_us, tablet-download-pool-3bc8f4968b9743baab5d25c8c7237c9c.queue_time_us) (tablet-download-pool-3bc8f4968b9743baab5d25c8c7237c9c.run_cpu_time_us, tablet-download-pool-3bc8f4968b9743baab5d25c8c7237c9c.run_cpu_time_us) (tablet-download-pool-3bc8f4968b9743baab5d25c8c7237c9c.run_wall_time_us, tablet-download-pool-3bc8f4968b9743baab5d25c8c7237c9c.run_wall_time_us) (tablet-download-pool-45e7a03f838d4beaae2080d523319d58.queue_time_us, tablet-download-pool-45e7a03f838d4beaae2080d523319d58.queue_time_us) (tablet-download-pool-45e7a03f838d4beaae2080d523319d58.run_cpu_time_us, tablet-download-pool-45e7a03f838d4beaae2080d523319d58.run_cpu_time_us) (tablet-download-pool-45e7a03f838d4beaae2080d523319d58.run_wall_time_us, tablet-download-pool-45e7a03f838d4beaae2080d523319d58.run_wall_time_us) (tablet-download-pool-486e9843fd784148b9b49dfe27996dff.queue_time_us, tablet-download-pool-486e9843fd784148b9b49dfe27996dff.queue_time_us) (tablet-download-pool-486e9843fd784148b9b49dfe27996dff.run_cpu_time_us, tablet-download-pool-486e9843fd784148b9b49dfe27996dff.run_cpu_time_us) (tablet-download-pool-486e9843fd784148b9b49dfe27996dff.run_wall_time_us, tablet-download-pool-486e9843fd784148b9b49dfe27996dff.run_wall_time_us) (tablet-download-pool-658865ee3f8d424ca5cd35e12e6d7c77.queue_time_us, tablet-download-pool-658865ee3f8d424ca5cd35e12e6d7c77.queue_time_us) (tablet-download-pool-658865ee3f8d424ca5cd35e12e6d7c77.run_cpu_time_us, tablet-download-pool-658865ee3f8d424ca5cd35e12e6d7c77.run_cpu_time_us) (tablet-download-pool-658865ee3f8d424ca5cd35e12e6d7c77.run_wall_time_us, tablet-download-pool-658865ee3f8d424ca5cd35e12e6d7c77.run_wall_time_us) (tablet-download-pool-8425cfe5c6134a60852e12f9af55227b.queue_time_us, tablet-download-pool-8425cfe5c6134a60852e12f9af55227b.queue_time_us) (tablet-download-pool-8425cfe5c6134a60852e12f9af55227b.run_cpu_time_us, tablet-download-pool-8425cfe5c6134a60852e12f9af55227b.run_cpu_time_us) (tablet-download-pool-8425cfe5c6134a60852e12f9af55227b.run_wall_time_us, tablet-download-pool-8425cfe5c6134a60852e12f9af55227b.run_wall_time_us) (tablet-download-pool-97149ad1d03847a085b2142519135a17.queue_time_us, tablet-download-pool-97149ad1d03847a085b2142519135a17.queue_time_us) (tablet-download-pool-9fbe2eaba5344dfaa29e78a0d73f224a.queue_time_us, tablet-download-pool-9fbe2eaba5344dfaa29e78a0d73f224a.queue_time_us) (tablet-download-pool-9fbe2eaba5344dfaa29e78a0d73f224a.run_cpu_time_us, tablet-download-pool-9fbe2eaba5344dfaa29e78a0d73f224a.run_cpu_time_us) (tablet-download-pool-9fbe2eaba5344dfaa29e78a0d73f224a.run_wall_time_us, tablet-download-pool-9fbe2eaba5344dfaa29e78a0d73f224a.run_wall_time_us) (tablet-download-pool-e7b54127ad4245838c4cc05071378472.queue_time_us, tablet-download-pool-e7b54127ad4245838c4cc05071378472.queue_time_us) (tablet-download-pool-e7b54127ad4245838c4cc05071378472.run_cpu_time_us, tablet-download-pool-e7b54127ad4245838c4cc05071378472.run_cpu_time_us) (tablet-download-pool-e7b54127ad4245838c4cc05071378472.run_wall_time_us, tablet-download-pool-e7b54127ad4245838c4cc05071378472.run_wall_time_us) (tablet-download-pool-f53acf725b62447382e9bae95e6e74d5.queue_time_us, tablet-download-pool-f53acf725b62447382e9bae95e6e74d5.queue_time_us) (tablet-download-pool-f53acf725b62447382e9bae95e6e74d5.run_cpu_time_us, tablet-download-pool-f53acf725b62447382e9bae95e6e74d5.run_cpu_time_us) (tablet-download-pool-f53acf725b62447382e9bae95e6e74d5.run_wall_time_us, tablet-download-pool-f53acf725b62447382e9bae95e6e74d5.run_wall_time_us) (tablet-download-pool-f83a4778ffe640ee907e0a73e58abb73.queue_time_us, tablet-download-pool-f83a4778ffe640ee907e0a73e58abb73.queue_time_us) (tablet-download-pool-f83a4778ffe640ee907e0a73e58abb73.run_cpu_time_us, tablet-download-pool-f83a4778ffe640ee907e0a73e58abb73.run_cpu_time_us) (tablet-download-pool-f83a4778ffe640ee907e0a73e58abb73.run_wall_time_us, tablet-download-pool-f83a4778ffe640ee907e0a73e58abb73.run_wall_time_us) (tablet-download-pool-fa7e18041c294a239eb7b8f735b66c53.queue_time_us, tablet-download-pool-fa7e18041c294a239eb7b8f735b66c53.queue_time_us) (tablet-download-pool-fa7e18041c294a239eb7b8f735b66c53.run_cpu_time_us, tablet-download-pool-fa7e18041c294a239eb7b8f735b66c53.run_cpu_time_us) (tablet-download-pool-fa7e18041c294a239eb7b8f735b66c53.run_wall_time_us, tablet-download-pool-fa7e18041c294a239eb7b8f735b66c53.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240503 08:14:31.445093 28200 maintenance_manager.cc:643] P aae860d7ae2b4002bf7794c278f9cc56: FlushMRSOp(33f0f4bc5722485a8f978a02367ca80e) complete. Timing: real 0.113s user 0.038s sys 0.011s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":5798,"dirs.run_cpu_time_us":211,"dirs.run_wall_time_us":12839,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":785,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":15558,"threads_started":2,"wal-append.queue_time_us":5849}
@ 0x7fb9c7cb6dcd google::LogMessage::Fail() at ??:0
I20240503 08:14:31.448356 28309 maintenance_manager.cc:419] P aae860d7ae2b4002bf7794c278f9cc56: Scheduling FlushMRSOp(46d27fa12990405cae5f94976ac7a014): perf score=0.033406
@ 0x7fb9c7cbab93 google::LogMessage::SendToLog() at ??:0
I20240503 08:14:31.450740 12117 ts_tablet_manager.cc:902] T e3a6b72060804beb81a177988d287f63 P ce1496a47d084365ad786cd09e472a2c: Initiating tablet copy from peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
@ 0x7fb9c7cb67cc google::LogMessage::Flush() at ??:0
@ 0x7fb9c7cb7f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb9ca9f7742 kudu::TraceMetrics::InternName() at ??:0
I20240503 08:14:31.473956 12117 tablet_copy_client.cc:308] T e3a6b72060804beb81a177988d287f63 P ce1496a47d084365ad786cd09e472a2c: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.68:46211
I20240503 08:14:31.474499 28413 tablet_copy_service.cc:140] P d09a11a03989421b9f28019dc1c14cc2: Received BeginTabletCopySession request for tablet e3a6b72060804beb81a177988d287f63 from peer ce1496a47d084365ad786cd09e472a2c ({username='slave'} at 127.20.230.70:51715)
I20240503 08:14:31.474624 28413 tablet_copy_service.cc:161] P d09a11a03989421b9f28019dc1c14cc2: Beginning new tablet copy session on tablet e3a6b72060804beb81a177988d287f63 from peer ce1496a47d084365ad786cd09e472a2c at {username='slave'} at 127.20.230.70:51715: session id = ce1496a47d084365ad786cd09e472a2c-e3a6b72060804beb81a177988d287f63
I20240503 08:14:31.477362 28413 tablet_copy_source_session.cc:205] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:31.478621 12117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3a6b72060804beb81a177988d287f63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:31.484763 12117 tablet_copy_client.cc:783] T e3a6b72060804beb81a177988d287f63 P ce1496a47d084365ad786cd09e472a2c: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:31.485009 12117 tablet_copy_client.cc:647] T e3a6b72060804beb81a177988d287f63 P ce1496a47d084365ad786cd09e472a2c: tablet copy: Starting download of 1 WAL segments...
@ 0x7fb9ca9c3f8b kudu::ThreadPool::ThreadPool() at ??:0
I20240503 08:14:31.490084 12117 tablet_copy_client.cc:523] T e3a6b72060804beb81a177988d287f63 P ce1496a47d084365ad786cd09e472a2c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:31.493412 12117 tablet_bootstrap.cc:492] T e3a6b72060804beb81a177988d287f63 P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:14:31.498227 28200 maintenance_manager.cc:643] P aae860d7ae2b4002bf7794c278f9cc56: FlushMRSOp(46d27fa12990405cae5f94976ac7a014) complete. Timing: real 0.049s user 0.031s sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":51,"dirs.run_cpu_time_us":154,"dirs.run_wall_time_us":1442,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":865,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":1971,"threads_started":1,"wal-append.queue_time_us":2036}
@ 0x7fb9ca9bdcd6 kudu::ThreadPoolBuilder::Build() at ??:0
I20240503 08:14:31.501555 28309 maintenance_manager.cc:419] P aae860d7ae2b4002bf7794c278f9cc56: Scheduling UndoDeltaBlockGCOp(33f0f4bc5722485a8f978a02367ca80e): 4838 bytes on disk
@ 0x7fb9d9e118de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240503 08:14:31.505383 28200 maintenance_manager.cc:643] P aae860d7ae2b4002bf7794c278f9cc56: UndoDeltaBlockGCOp(33f0f4bc5722485a8f978a02367ca80e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
@ 0x7fb9d9e29daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240503 08:14:31.509569 28309 maintenance_manager.cc:419] P aae860d7ae2b4002bf7794c278f9cc56: Scheduling UndoDeltaBlockGCOp(46d27fa12990405cae5f94976ac7a014): 4581 bytes on disk
I20240503 08:14:31.510097 28200 maintenance_manager.cc:643] P aae860d7ae2b4002bf7794c278f9cc56: UndoDeltaBlockGCOp(46d27fa12990405cae5f94976ac7a014) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
I20240503 08:14:31.511260 28070 maintenance_manager.cc:643] P 1b411c6bb25549368041773d7b710d48: FlushMRSOp(fe4d26ee022b424195e836834bd2f051) complete. Timing: real 0.123s user 0.050s sys 0.051s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":17366,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":2108,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":926,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":1280,"thread_start_us":71565,"threads_started":2,"wal-append.queue_time_us":47344}
I20240503 08:14:31.514156 28179 maintenance_manager.cc:419] P 1b411c6bb25549368041773d7b710d48: Scheduling FlushMRSOp(33f0f4bc5722485a8f978a02367ca80e): perf score=0.033427
@ 0x7fb9d9f54480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240503 08:14:31.521252 12117 tablet_bootstrap.cc:492] T e3a6b72060804beb81a177988d287f63 P ce1496a47d084365ad786cd09e472a2c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:31.521624 12117 tablet_bootstrap.cc:492] T e3a6b72060804beb81a177988d287f63 P ce1496a47d084365ad786cd09e472a2c: Bootstrap complete.
I20240503 08:14:31.521883 12117 ts_tablet_manager.cc:1367] T e3a6b72060804beb81a177988d287f63 P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.007s
I20240503 08:14:31.522660 12117 raft_consensus.cc:348] T e3a6b72060804beb81a177988d287f63 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:14:31.522878 12117 raft_consensus.cc:729] T e3a6b72060804beb81a177988d287f63 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:14:31.523176 12117 consensus_queue.cc:260] T e3a6b72060804beb81a177988d287f63 P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:14:31.523758 12117 ts_tablet_manager.cc:1398] T e3a6b72060804beb81a177988d287f63 P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:14:31.524710 28413 tablet_copy_service.cc:342] P d09a11a03989421b9f28019dc1c14cc2: Request end of tablet copy session ce1496a47d084365ad786cd09e472a2c-e3a6b72060804beb81a177988d287f63 received from {username='slave'} at 127.20.230.70:51715
I20240503 08:14:31.524804 28413 tablet_copy_service.cc:433] P d09a11a03989421b9f28019dc1c14cc2: ending tablet copy session ce1496a47d084365ad786cd09e472a2c-e3a6b72060804beb81a177988d287f63 on tablet e3a6b72060804beb81a177988d287f63 with peer ce1496a47d084365ad786cd09e472a2c
@ 0x7fb9d9f8ca8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7fb9d9f6f981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7fb9d9f6f70e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fb9ca9c7b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb9ca9cb19c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fb9ca9cb019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fb9ca9a601e kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb9cdd2b6db start_thread at ??:0
@ 0x7fb9c6b6971f clone at ??:0
I20240503 08:14:31.605033 28070 maintenance_manager.cc:643] P 1b411c6bb25549368041773d7b710d48: FlushMRSOp(33f0f4bc5722485a8f978a02367ca80e) complete. Timing: real 0.083s user 0.039s sys 0.010s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":46,"dirs.run_cpu_time_us":163,"dirs.run_wall_time_us":1404,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":779,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":9147,"threads_started":1,"wal-append.queue_time_us":9048}
I20240503 08:14:31.607856 28179 maintenance_manager.cc:419] P 1b411c6bb25549368041773d7b710d48: Scheduling UndoDeltaBlockGCOp(fe4d26ee022b424195e836834bd2f051): 4923 bytes on disk
I20240503 08:14:31.608438 28070 maintenance_manager.cc:643] P 1b411c6bb25549368041773d7b710d48: UndoDeltaBlockGCOp(fe4d26ee022b424195e836834bd2f051) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20240503 08:14:31.610926 28179 maintenance_manager.cc:419] P 1b411c6bb25549368041773d7b710d48: Scheduling UndoDeltaBlockGCOp(33f0f4bc5722485a8f978a02367ca80e): 4838 bytes on disk
I20240503 08:14:31.611445 28070 maintenance_manager.cc:643] P 1b411c6bb25549368041773d7b710d48: UndoDeltaBlockGCOp(33f0f4bc5722485a8f978a02367ca80e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20240503 08:14:31.627163 27940 maintenance_manager.cc:643] P 833582ff68c94269be364ebd187a75db: FlushMRSOp(fe4d26ee022b424195e836834bd2f051) complete. Timing: real 0.241s user 0.061s sys 0.010s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":6519,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":7981,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":851,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":11947,"threads_started":2,"wal-append.queue_time_us":12900}
I20240503 08:14:31.630255 28049 maintenance_manager.cc:419] P 833582ff68c94269be364ebd187a75db: Scheduling FlushMRSOp(46d27fa12990405cae5f94976ac7a014): perf score=0.033460
I20240503 08:14:31.759949 27940 maintenance_manager.cc:643] P 833582ff68c94269be364ebd187a75db: FlushMRSOp(46d27fa12990405cae5f94976ac7a014) complete. Timing: real 0.129s user 0.042s sys 0.005s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":36,"dirs.run_cpu_time_us":182,"dirs.run_wall_time_us":1488,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":816,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":27582,"threads_started":1,"wal-append.queue_time_us":27672}
I20240503 08:14:31.762722 28049 maintenance_manager.cc:419] P 833582ff68c94269be364ebd187a75db: Scheduling UndoDeltaBlockGCOp(fe4d26ee022b424195e836834bd2f051): 4923 bytes on disk
I20240503 08:14:31.763307 27940 maintenance_manager.cc:643] P 833582ff68c94269be364ebd187a75db: UndoDeltaBlockGCOp(fe4d26ee022b424195e836834bd2f051) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
I20240503 08:14:31.767496 28049 maintenance_manager.cc:419] P 833582ff68c94269be364ebd187a75db: Scheduling UndoDeltaBlockGCOp(46d27fa12990405cae5f94976ac7a014): 4581 bytes on disk
I20240503 08:14:31.768014 27940 maintenance_manager.cc:643] P 833582ff68c94269be364ebd187a75db: UndoDeltaBlockGCOp(46d27fa12990405cae5f94976ac7a014) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
W20240503 08:14:31.946903 30697 connection.cc:535] server connection from 127.20.230.67:45903 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:14:31.947247 30695 connection.cc:535] client connection to 127.20.230.67:39919 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:14:31.947309 28327 connection.cc:535] client connection to 127.20.230.67:39919 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:14:31.947491 31095 connection.cc:535] client connection to 127.20.230.67:39919 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:14:31.947527 28327 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:14:31.947711 31095 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:14:31.947916 28067 connection.cc:535] client connection to 127.20.230.67:39919 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:14:31.948119 28067 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:14:31.948385 27937 connection.cc:535] client connection to 127.20.230.67:39919 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:14:31.948573 27937 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:14:31.949196 30695 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:14:31.949690 30828 connection.cc:535] client connection to 127.20.230.67:39919 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:14:31.949929 30828 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:14:31.950352 30960 connection.cc:535] client connection to 127.20.230.67:39919 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:14:31.950553 30960 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:14:31.951088 30894 raft_consensus.cc:1203] T e3a6b72060804beb81a177988d287f63 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240503 08:14:31.962967 28327 consensus_peers.cc:485] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:31.967263 28327 consensus_peers.cc:485] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:31.967401 28327 consensus_peers.cc:485] T eeee6a265fc94e91bf9daa8301662e5f P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:31.967511 28327 consensus_peers.cc:485] T bdfe68a127f043f08f04749843a38375 P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:31.967612 28327 consensus_peers.cc:485] T 8bcbc022caa94bb08a2db4de9b060557 P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:32.010370 30828 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.67:39919, user_credentials={real_user=slave}} blocked reactor thread for 60484.1us
W20240503 08:14:32.014736 31095 consensus_peers.cc:485] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:32.020113 28327 consensus_peers.cc:485] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:32.021282 27937 consensus_peers.cc:485] T 3d0498256b5f4b8fb1681759429f10ed P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:32.021441 27937 consensus_peers.cc:485] T e0b904461c1c44ed93e1a227d93498eb P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:32.021529 27937 consensus_peers.cc:485] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:32.021615 27937 consensus_peers.cc:485] T 7c8c7659181040cabf20ae116fae78c8 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:32.021696 27937 consensus_peers.cc:485] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:32.024598 30960 consensus_peers.cc:485] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:32.024741 30960 consensus_peers.cc:485] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:32.024829 30960 consensus_peers.cc:485] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:32.025784 30828 consensus_peers.cc:462] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:32.025915 30828 consensus_peers.cc:485] T 0f25688be4e74d31aaab85b7a4329a34 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:32.026019 30828 consensus_peers.cc:485] T 045b3ce4b1d44f94a802b41a12e2704d P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:32.026108 30828 consensus_peers.cc:485] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:32.026192 30828 consensus_peers.cc:485] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:32.026904 31095 consensus_peers.cc:485] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:32.027036 31095 consensus_peers.cc:485] T 2c595805729d48fd974d761ed8a60158 P b00f244c69df401997ea99d492218a98 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:32.027137 31095 consensus_peers.cc:485] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:32.027729 30695 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.67:39919, user_credentials={real_user=slave}} blocked reactor thread for 78562.8us
W20240503 08:14:32.030656 28067 consensus_peers.cc:485] T 658865ee3f8d424ca5cd35e12e6d7c77 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:32.030736 30695 consensus_peers.cc:485] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:32.030790 28067 consensus_peers.cc:485] T 2dce5b2aaea045ccb7424cbbd44576c2 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:32.030867 28067 consensus_peers.cc:485] T df90a0d9d338416b81e253fa23542cca P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:32.030937 28067 consensus_peers.cc:485] T 099e37089b2e42b4839e681f8f37365d P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:32.031122 28067 consensus_peers.cc:485] T fa7e18041c294a239eb7b8f735b66c53 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:32.033649 30828 consensus_peers.cc:485] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:32.034909 28067 consensus_peers.cc:485] T bab5c25c31a64b578489b1006f631cc0 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:32.049746 27937 consensus_peers.cc:485] T 0ceedfb43869465bb63931c39c28583a P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:32.050987 28067 consensus_peers.cc:485] T d94f5824a71f454f8a1bfa13dd032684 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:32.077244 30828 consensus_peers.cc:485] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:32.078743 27937 consensus_peers.cc:485] T c8ca21ffe16e41ab80f5693dc4dbb682 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:32.109212 28327 consensus_peers.cc:485] T 8425cfe5c6134a60852e12f9af55227b P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:32.134977 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:32.136933 27983 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:32.139465 30742 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 08:14:32.148953 28327 consensus_peers.cc:485] T ba1fc93781714efb8eb7ec67c2fceb8a P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:32.157655 28327 consensus_peers.cc:485] T 45e7a03f838d4beaae2080d523319d58 P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:32.164309 31141 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:32.242815 31006 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:32.252928 28373 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:32.281919 30874 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:14:32.361956 28389 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "607a503b64d8440a81cf66ab158ae273"
dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47618
I20240503 08:14:32.362169 28389 raft_consensus.cc:595] T 607a503b64d8440a81cf66ab158ae273 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Received request to transfer leadership
W20240503 08:14:32.414649 30828 consensus_peers.cc:462] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.468245 11714 raft_consensus.cc:482] T 6a96965691644db8b9461d146536a026 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.468469 11714 raft_consensus.cc:504] T 6a96965691644db8b9461d146536a026 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:32.469372 11714 leader_election.cc:288] T 6a96965691644db8b9461d146536a026 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:14:32.470234 31161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a96965691644db8b9461d146536a026" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
W20240503 08:14:32.478785 27937 leader_election.cc:334] T 6a96965691644db8b9461d146536a026 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.478964 27937 leader_election.cc:302] T 6a96965691644db8b9461d146536a026 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 833582ff68c94269be364ebd187a75db; no voters: aae860d7ae2b4002bf7794c278f9cc56, b00f244c69df401997ea99d492218a98
I20240503 08:14:32.479540 12243 raft_consensus.cc:2726] T 6a96965691644db8b9461d146536a026 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:14:32.495885 12048 raft_consensus.cc:979] T 607a503b64d8440a81cf66ab158ae273 P d09a11a03989421b9f28019dc1c14cc2: : Instructing follower 6d1d7de19459435fbde1999e3dfd226b to start an election
I20240503 08:14:32.496063 12048 raft_consensus.cc:1067] T 607a503b64d8440a81cf66ab158ae273 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Signalling peer 6d1d7de19459435fbde1999e3dfd226b to start an election
I20240503 08:14:32.496981 30761 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "607a503b64d8440a81cf66ab158ae273"
dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
from {username='slave'} at 127.20.230.68:50161
I20240503 08:14:32.497222 30761 raft_consensus.cc:482] T 607a503b64d8440a81cf66ab158ae273 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:14:32.497308 30761 raft_consensus.cc:3037] T 607a503b64d8440a81cf66ab158ae273 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:32.500339 30761 raft_consensus.cc:504] T 607a503b64d8440a81cf66ab158ae273 P 6d1d7de19459435fbde1999e3dfd226b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:14:32.502722 30761 leader_election.cc:288] T 607a503b64d8440a81cf66ab158ae273 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 2 election: Requested vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:14:32.503166 28389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "607a503b64d8440a81cf66ab158ae273" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:14:32.503394 28389 raft_consensus.cc:3032] T 607a503b64d8440a81cf66ab158ae273 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:14:32.503386 28129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "607a503b64d8440a81cf66ab158ae273" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:14:32.503485 28389 raft_consensus.cc:729] T 607a503b64d8440a81cf66ab158ae273 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Running, Role: LEADER
I20240503 08:14:32.503566 28129 raft_consensus.cc:3037] T 607a503b64d8440a81cf66ab158ae273 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:32.503738 28389 consensus_queue.cc:260] T 607a503b64d8440a81cf66ab158ae273 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:14:32.504158 28389 raft_consensus.cc:3037] T 607a503b64d8440a81cf66ab158ae273 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:32.507297 28000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "607a503b64d8440a81cf66ab158ae273" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:14:32.507536 28000 raft_consensus.cc:3037] T 607a503b64d8440a81cf66ab158ae273 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:32.510133 28389 raft_consensus.cc:2454] T 607a503b64d8440a81cf66ab158ae273 P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 2.
I20240503 08:14:32.510493 28000 raft_consensus.cc:2454] T 607a503b64d8440a81cf66ab158ae273 P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 2.
I20240503 08:14:32.511219 30697 leader_election.cc:302] T 607a503b64d8440a81cf66ab158ae273 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, 833582ff68c94269be364ebd187a75db, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:14:32.511880 11366 raft_consensus.cc:2781] T 607a503b64d8440a81cf66ab158ae273 P 6d1d7de19459435fbde1999e3dfd226b [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:14:32.512029 11366 raft_consensus.cc:686] T 607a503b64d8440a81cf66ab158ae273 P 6d1d7de19459435fbde1999e3dfd226b [term 2 LEADER]: Becoming Leader. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Running, Role: LEADER
I20240503 08:14:32.512287 11366 consensus_queue.cc:237] T 607a503b64d8440a81cf66ab158ae273 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:14:32.519070 28129 raft_consensus.cc:2454] T 607a503b64d8440a81cf66ab158ae273 P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 2.
I20240503 08:14:32.532325 27862 catalog_manager.cc:5484] T 607a503b64d8440a81cf66ab158ae273 P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: term changed from 1 to 2, leader changed from d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) to 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69). New cstate: current_term: 2 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
W20240503 08:14:32.581624 30694 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.70:33747, user_credentials={real_user=slave}} blocked reactor thread for 69420.7us
I20240503 08:14:32.595305 11965 raft_consensus.cc:482] T 33f0f4bc5722485a8f978a02367ca80e P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.595491 11965 raft_consensus.cc:504] T 33f0f4bc5722485a8f978a02367ca80e P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:14:32.596143 11965 leader_election.cc:288] T 33f0f4bc5722485a8f978a02367ca80e P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:14:32.597362 28390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33f0f4bc5722485a8f978a02367ca80e" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
W20240503 08:14:32.600423 28067 leader_election.cc:334] T 33f0f4bc5722485a8f978a02367ca80e P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.600693 28067 leader_election.cc:302] T 33f0f4bc5722485a8f978a02367ca80e P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b411c6bb25549368041773d7b710d48; no voters: aae860d7ae2b4002bf7794c278f9cc56, d09a11a03989421b9f28019dc1c14cc2
I20240503 08:14:32.601006 11965 raft_consensus.cc:2726] T 33f0f4bc5722485a8f978a02367ca80e P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:14:32.608001 12094 raft_consensus.cc:482] T 08fb64977309479b831635344969c662 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.608177 12094 raft_consensus.cc:504] T 08fb64977309479b831635344969c662 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:14:32.608825 12094 leader_election.cc:288] T 08fb64977309479b831635344969c662 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:14:32.609450 27999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08fb64977309479b831635344969c662" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
W20240503 08:14:32.611641 30695 leader_election.cc:334] T 08fb64977309479b831635344969c662 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.611805 30695 leader_election.cc:302] T 08fb64977309479b831635344969c662 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b; no voters: 833582ff68c94269be364ebd187a75db, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:32.612213 12094 raft_consensus.cc:2726] T 08fb64977309479b831635344969c662 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:14:32.612483 11539 raft_consensus.cc:482] T 2222ff7fb5184d6c86010e1475d6c1f7 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.612663 11539 raft_consensus.cc:504] T 2222ff7fb5184d6c86010e1475d6c1f7 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:14:32.613312 11539 leader_election.cc:288] T 2222ff7fb5184d6c86010e1475d6c1f7 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:14:32.618374 30760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2222ff7fb5184d6c86010e1475d6c1f7" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
W20240503 08:14:32.620419 28067 leader_election.cc:334] T 2222ff7fb5184d6c86010e1475d6c1f7 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.620687 28067 leader_election.cc:302] T 2222ff7fb5184d6c86010e1475d6c1f7 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b411c6bb25549368041773d7b710d48; no voters: 6d1d7de19459435fbde1999e3dfd226b, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:32.626701 11539 raft_consensus.cc:2726] T 2222ff7fb5184d6c86010e1475d6c1f7 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:14:32.631310 12286 raft_consensus.cc:482] T 2222ff7fb5184d6c86010e1475d6c1f7 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.631462 12286 raft_consensus.cc:504] T 2222ff7fb5184d6c86010e1475d6c1f7 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:14:32.635354 12286 leader_election.cc:288] T 2222ff7fb5184d6c86010e1475d6c1f7 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:14:32.636152 28129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2222ff7fb5184d6c86010e1475d6c1f7" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:14:32.636404 28129 raft_consensus.cc:2454] T 2222ff7fb5184d6c86010e1475d6c1f7 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:14:32.636799 30696 leader_election.cc:302] T 2222ff7fb5184d6c86010e1475d6c1f7 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 6d1d7de19459435fbde1999e3dfd226b; no voters:
I20240503 08:14:32.637037 12286 raft_consensus.cc:2781] T 2222ff7fb5184d6c86010e1475d6c1f7 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:14:32.637101 12286 raft_consensus.cc:482] T 2222ff7fb5184d6c86010e1475d6c1f7 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting leader election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.637143 12286 raft_consensus.cc:3037] T 2222ff7fb5184d6c86010e1475d6c1f7 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:32.639489 12286 raft_consensus.cc:504] T 2222ff7fb5184d6c86010e1475d6c1f7 P 6d1d7de19459435fbde1999e3dfd226b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:14:32.640064 12286 leader_election.cc:288] T 2222ff7fb5184d6c86010e1475d6c1f7 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 2 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:14:32.640470 28129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2222ff7fb5184d6c86010e1475d6c1f7" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:14:32.640642 28129 raft_consensus.cc:3037] T 2222ff7fb5184d6c86010e1475d6c1f7 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:14:32.641296 30695 leader_election.cc:334] T 2222ff7fb5184d6c86010e1475d6c1f7 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.643376 28129 raft_consensus.cc:2454] T 2222ff7fb5184d6c86010e1475d6c1f7 P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 2.
W20240503 08:14:32.643682 30695 leader_election.cc:334] T 2222ff7fb5184d6c86010e1475d6c1f7 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.643906 30696 leader_election.cc:302] T 2222ff7fb5184d6c86010e1475d6c1f7 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 6d1d7de19459435fbde1999e3dfd226b; no voters: aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:32.644248 11539 raft_consensus.cc:482] T 07866de372e541ec9dca8a3d42f8b0ab P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.644380 11539 raft_consensus.cc:504] T 07866de372e541ec9dca8a3d42f8b0ab P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:14:32.644968 11539 leader_election.cc:288] T 07866de372e541ec9dca8a3d42f8b0ab P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:14:32.645505 12286 raft_consensus.cc:2781] T 2222ff7fb5184d6c86010e1475d6c1f7 P 6d1d7de19459435fbde1999e3dfd226b [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:14:32.645587 28390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07866de372e541ec9dca8a3d42f8b0ab" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:14:32.645720 12286 raft_consensus.cc:686] T 2222ff7fb5184d6c86010e1475d6c1f7 P 6d1d7de19459435fbde1999e3dfd226b [term 2 LEADER]: Becoming Leader. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Running, Role: LEADER
I20240503 08:14:32.645975 12286 consensus_queue.cc:237] T 2222ff7fb5184d6c86010e1475d6c1f7 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
W20240503 08:14:32.648309 28067 leader_election.cc:334] T 07866de372e541ec9dca8a3d42f8b0ab P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.648536 28067 leader_election.cc:302] T 07866de372e541ec9dca8a3d42f8b0ab P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b411c6bb25549368041773d7b710d48; no voters: aae860d7ae2b4002bf7794c278f9cc56, d09a11a03989421b9f28019dc1c14cc2
I20240503 08:14:32.649513 11539 raft_consensus.cc:2726] T 07866de372e541ec9dca8a3d42f8b0ab P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:14:32.659765 27858 catalog_manager.cc:5484] T 2222ff7fb5184d6c86010e1475d6c1f7 P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: term changed from 1 to 2, leader changed from aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) to 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69). New cstate: current_term: 2 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:14:32.689100 11603 raft_consensus.cc:482] T f46393257ce9471891698c36a07546ad P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.689335 11603 raft_consensus.cc:504] T f46393257ce9471891698c36a07546ad P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:32.690238 11603 leader_election.cc:288] T f46393257ce9471891698c36a07546ad P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:14:32.691877 28390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46393257ce9471891698c36a07546ad" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
W20240503 08:14:32.693060 30960 leader_election.cc:334] T f46393257ce9471891698c36a07546ad P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.693197 30960 leader_election.cc:302] T f46393257ce9471891698c36a07546ad P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8; no voters: aae860d7ae2b4002bf7794c278f9cc56, d09a11a03989421b9f28019dc1c14cc2
I20240503 08:14:32.693794 11603 raft_consensus.cc:2726] T f46393257ce9471891698c36a07546ad P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:14:32.711658 10051 raft_consensus.cc:482] T d9098f3d36b24d578a3f8d7d3ccad052 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.711869 10051 raft_consensus.cc:504] T d9098f3d36b24d578a3f8d7d3ccad052 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:14:32.712527 10051 leader_election.cc:288] T d9098f3d36b24d578a3f8d7d3ccad052 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:14:32.714866 27999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9098f3d36b24d578a3f8d7d3ccad052" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
W20240503 08:14:32.719233 31095 leader_election.cc:334] T d9098f3d36b24d578a3f8d7d3ccad052 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.719385 31095 leader_election.cc:302] T d9098f3d36b24d578a3f8d7d3ccad052 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b00f244c69df401997ea99d492218a98; no voters: 833582ff68c94269be364ebd187a75db, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:32.719686 10051 raft_consensus.cc:2726] T d9098f3d36b24d578a3f8d7d3ccad052 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:14:32.726063 11714 raft_consensus.cc:482] T d9098f3d36b24d578a3f8d7d3ccad052 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.726267 11714 raft_consensus.cc:504] T d9098f3d36b24d578a3f8d7d3ccad052 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:14:32.726922 11714 leader_election.cc:288] T d9098f3d36b24d578a3f8d7d3ccad052 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:14:32.727535 31161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9098f3d36b24d578a3f8d7d3ccad052" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:14:32.727809 31161 raft_consensus.cc:2454] T d9098f3d36b24d578a3f8d7d3ccad052 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:14:32.728439 27936 leader_election.cc:302] T d9098f3d36b24d578a3f8d7d3ccad052 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, b00f244c69df401997ea99d492218a98; no voters:
I20240503 08:14:32.729792 12243 raft_consensus.cc:2781] T d9098f3d36b24d578a3f8d7d3ccad052 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:14:32.729900 12243 raft_consensus.cc:482] T d9098f3d36b24d578a3f8d7d3ccad052 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting leader election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.729993 12243 raft_consensus.cc:3037] T d9098f3d36b24d578a3f8d7d3ccad052 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:32.737881 12243 raft_consensus.cc:504] T d9098f3d36b24d578a3f8d7d3ccad052 P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
W20240503 08:14:32.738754 27937 leader_election.cc:334] T d9098f3d36b24d578a3f8d7d3ccad052 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.739259 31161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9098f3d36b24d578a3f8d7d3ccad052" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:14:32.739486 31161 raft_consensus.cc:3037] T d9098f3d36b24d578a3f8d7d3ccad052 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:14:32.740547 27937 leader_election.cc:334] T d9098f3d36b24d578a3f8d7d3ccad052 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.742298 31161 raft_consensus.cc:2454] T d9098f3d36b24d578a3f8d7d3ccad052 P b00f244c69df401997ea99d492218a98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 2.
I20240503 08:14:32.742705 27936 leader_election.cc:302] T d9098f3d36b24d578a3f8d7d3ccad052 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 833582ff68c94269be364ebd187a75db, b00f244c69df401997ea99d492218a98; no voters: aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:32.743038 11714 raft_consensus.cc:2781] T d9098f3d36b24d578a3f8d7d3ccad052 P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:14:32.743054 12243 leader_election.cc:288] T d9098f3d36b24d578a3f8d7d3ccad052 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:14:32.743218 11714 raft_consensus.cc:686] T d9098f3d36b24d578a3f8d7d3ccad052 P 833582ff68c94269be364ebd187a75db [term 2 LEADER]: Becoming Leader. State: Replica: 833582ff68c94269be364ebd187a75db, State: Running, Role: LEADER
I20240503 08:14:32.743492 11714 consensus_queue.cc:237] T d9098f3d36b24d578a3f8d7d3ccad052 P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:14:32.762496 27858 catalog_manager.cc:5484] T d9098f3d36b24d578a3f8d7d3ccad052 P 833582ff68c94269be364ebd187a75db reported cstate change: term changed from 1 to 2, leader changed from aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) to 833582ff68c94269be364ebd187a75db (127.20.230.65). New cstate: current_term: 2 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:14:32.763571 11603 raft_consensus.cc:482] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.763746 11603 raft_consensus.cc:504] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:32.764463 11603 leader_election.cc:288] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:14:32.765223 28129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8450e62352a74d2ea1471166d4cb1132" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:14:32.766136 10051 raft_consensus.cc:482] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.766330 10051 raft_consensus.cc:504] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:14:32.767050 10051 leader_election.cc:288] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
W20240503 08:14:32.768483 30960 leader_election.cc:334] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.768617 30960 leader_election.cc:302] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:32.768759 28129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc87f7f3c724cd4a270583ddebd87aa" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:14:32.769418 11603 raft_consensus.cc:2726] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:14:32.770464 11603 raft_consensus.cc:482] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.770617 11603 raft_consensus.cc:504] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:32.771144 11603 leader_election.cc:288] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:14:32.773231 28129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de5dfccb0359462dbb7da776293fa679" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
W20240503 08:14:32.773981 30960 leader_election.cc:334] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.774106 30960 leader_election.cc:302] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:32.774803 11603 raft_consensus.cc:2726] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 08:14:32.775327 31095 leader_election.cc:334] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.775466 31095 leader_election.cc:302] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b00f244c69df401997ea99d492218a98; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:32.778312 10051 raft_consensus.cc:2726] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:14:32.779254 11144 raft_consensus.cc:482] T ca38671a760d4534ab903579e0487772 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.779448 11144 raft_consensus.cc:504] T ca38671a760d4534ab903579e0487772 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:14:32.780128 11144 leader_election.cc:288] T ca38671a760d4534ab903579e0487772 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:14:32.780946 27999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca38671a760d4534ab903579e0487772" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
W20240503 08:14:32.782411 31095 leader_election.cc:334] T ca38671a760d4534ab903579e0487772 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.782598 31095 leader_election.cc:302] T ca38671a760d4534ab903579e0487772 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b00f244c69df401997ea99d492218a98; no voters: 833582ff68c94269be364ebd187a75db, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:32.783038 10051 raft_consensus.cc:2726] T ca38671a760d4534ab903579e0487772 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:14:32.783388 12051 raft_consensus.cc:482] T 41aa0e288ca4469ba34a07ff159ace91 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.783556 12051 raft_consensus.cc:504] T 41aa0e288ca4469ba34a07ff159ace91 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:32.784375 12051 leader_election.cc:288] T 41aa0e288ca4469ba34a07ff159ace91 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:14:32.789734 31025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41aa0e288ca4469ba34a07ff159ace91" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:14:32.790766 28111 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:14:32.791071 12048 raft_consensus.cc:482] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.791237 12048 raft_consensus.cc:504] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:14:32.791853 12048 leader_election.cc:288] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:14:32.795835 28129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07866de372e541ec9dca8a3d42f8b0ab" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:14:32.796149 28129 raft_consensus.cc:2454] T 07866de372e541ec9dca8a3d42f8b0ab P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
W20240503 08:14:32.796940 28327 leader_election.cc:334] T 41aa0e288ca4469ba34a07ff159ace91 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.797080 28327 leader_election.cc:302] T 41aa0e288ca4469ba34a07ff159ace91 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d09a11a03989421b9f28019dc1c14cc2; no voters: aae860d7ae2b4002bf7794c278f9cc56, ced21ca983b749d0b4d4f6a0e2426bf8
I20240503 08:14:32.797367 12048 raft_consensus.cc:2726] T 41aa0e288ca4469ba34a07ff159ace91 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:14:32.798027 11603 raft_consensus.cc:482] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.798192 11603 raft_consensus.cc:504] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:32.798456 28328 leader_election.cc:302] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:14:32.798694 12048 raft_consensus.cc:2781] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:14:32.798784 12048 raft_consensus.cc:482] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.798844 11603 leader_election.cc:288] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:14:32.798873 12048 raft_consensus.cc:3037] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:14:32.799367 28327 leader_election.cc:334] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.801606 12048 raft_consensus.cc:504] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:14:32.803491 28129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17c6e3293aad4591a9c1395ebffff2e6" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:14:32.803876 12048 leader_election.cc:288] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
W20240503 08:14:32.804160 30960 leader_election.cc:334] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.804421 30961 leader_election.cc:302] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:32.805055 11603 raft_consensus.cc:2726] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 08:14:32.805810 28129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07866de372e541ec9dca8a3d42f8b0ab" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:14:32.805999 28129 raft_consensus.cc:3037] T 07866de372e541ec9dca8a3d42f8b0ab P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:14:32.812316 28327 leader_election.cc:334] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.812718 28129 raft_consensus.cc:2454] T 07866de372e541ec9dca8a3d42f8b0ab P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 2.
I20240503 08:14:32.813127 28328 leader_election.cc:302] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, d09a11a03989421b9f28019dc1c14cc2; no voters: aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:32.813531 12051 raft_consensus.cc:2781] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:14:32.813570 30742 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:32.813666 12051 raft_consensus.cc:686] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2 [term 2 LEADER]: Becoming Leader. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Running, Role: LEADER
I20240503 08:14:32.813954 12051 consensus_queue.cc:237] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:14:32.816731 12286 raft_consensus.cc:482] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.816869 12286 raft_consensus.cc:504] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:14:32.817453 12286 leader_election.cc:288] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:14:32.818167 28129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cac39d5831d4b6887250f7a9d63dcd5" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
W20240503 08:14:32.819705 30695 leader_election.cc:334] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.820215 27858 catalog_manager.cc:5484] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: term changed from 1 to 2, leader changed from aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) to d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68). New cstate: current_term: 2 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:14:32.821487 30696 leader_election.cc:302] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:32.823964 12286 raft_consensus.cc:2726] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:14:32.836024 11603 raft_consensus.cc:482] T 41aa0e288ca4469ba34a07ff159ace91 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.836207 11603 raft_consensus.cc:504] T 41aa0e288ca4469ba34a07ff159ace91 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:32.836894 11603 leader_election.cc:288] T 41aa0e288ca4469ba34a07ff159ace91 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:14:32.838366 28390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41aa0e288ca4469ba34a07ff159ace91" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:14:32.838621 28390 raft_consensus.cc:2454] T 41aa0e288ca4469ba34a07ff159ace91 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 1.
I20240503 08:14:32.839277 11965 raft_consensus.cc:482] T 8c714f29abb140379b57e08eb927e2b7 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.839421 11965 raft_consensus.cc:504] T 8c714f29abb140379b57e08eb927e2b7 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:14:32.839599 30962 leader_election.cc:302] T 41aa0e288ca4469ba34a07ff159ace91 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:14:32.839982 11603 raft_consensus.cc:2781] T 41aa0e288ca4469ba34a07ff159ace91 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:14:32.840077 11603 raft_consensus.cc:482] T 41aa0e288ca4469ba34a07ff159ace91 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.840184 11603 raft_consensus.cc:3037] T 41aa0e288ca4469ba34a07ff159ace91 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:32.840236 11965 leader_election.cc:288] T 8c714f29abb140379b57e08eb927e2b7 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:14:32.840528 12051 raft_consensus.cc:482] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.840694 12051 raft_consensus.cc:504] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
W20240503 08:14:32.841270 30960 leader_election.cc:334] T 41aa0e288ca4469ba34a07ff159ace91 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.841356 12051 leader_election.cc:288] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
W20240503 08:14:32.842902 28067 leader_election.cc:334] T 8c714f29abb140379b57e08eb927e2b7 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.843145 28129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "362590d3d0834d0b8a672ab552f8c209" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:14:32.843166 30892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c714f29abb140379b57e08eb927e2b7" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:14:32.843981 28066 leader_election.cc:302] T 8c714f29abb140379b57e08eb927e2b7 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b411c6bb25549368041773d7b710d48; no voters: aae860d7ae2b4002bf7794c278f9cc56, ce1496a47d084365ad786cd09e472a2c
I20240503 08:14:32.844406 11965 raft_consensus.cc:2726] T 8c714f29abb140379b57e08eb927e2b7 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:14:32.851917 12243 raft_consensus.cc:482] T 08fb64977309479b831635344969c662 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.852255 12243 raft_consensus.cc:504] T 08fb64977309479b831635344969c662 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:14:32.855499 12243 leader_election.cc:288] T 08fb64977309479b831635344969c662 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:14:32.857663 11539 raft_consensus.cc:482] T 1cac39d5831d4b6887250f7a9d63dcd5 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.857844 11539 raft_consensus.cc:504] T 1cac39d5831d4b6887250f7a9d63dcd5 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:14:32.858448 11539 leader_election.cc:288] T 1cac39d5831d4b6887250f7a9d63dcd5 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:14:32.858525 11989 raft_consensus.cc:482] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.858695 11989 raft_consensus.cc:504] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:14:32.859656 11989 leader_election.cc:288] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
W20240503 08:14:32.862542 30828 leader_election.cc:334] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:32.862828 28327 leader_election.cc:334] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.862977 28327 leader_election.cc:302] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d09a11a03989421b9f28019dc1c14cc2; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:32.862958 28129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c714f29abb140379b57e08eb927e2b7" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:14:32.863188 28129 raft_consensus.cc:2454] T 8c714f29abb140379b57e08eb927e2b7 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:14:32.863919 12048 raft_consensus.cc:2726] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 08:14:32.864233 30759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cac39d5831d4b6887250f7a9d63dcd5" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:14:32.864468 30759 raft_consensus.cc:2454] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:14:32.864486 30829 leader_election.cc:302] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, ce1496a47d084365ad786cd09e472a2c; no voters: aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:32.864804 28068 leader_election.cc:302] T 1cac39d5831d4b6887250f7a9d63dcd5 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 6d1d7de19459435fbde1999e3dfd226b; no voters:
I20240503 08:14:32.865104 11539 raft_consensus.cc:2781] T 1cac39d5831d4b6887250f7a9d63dcd5 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:14:32.865193 11539 raft_consensus.cc:482] T 1cac39d5831d4b6887250f7a9d63dcd5 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting leader election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.865275 11539 raft_consensus.cc:3037] T 1cac39d5831d4b6887250f7a9d63dcd5 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:32.866814 11989 raft_consensus.cc:2781] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:14:32.866902 11989 raft_consensus.cc:482] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting leader election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.866986 11989 raft_consensus.cc:3037] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:32.868865 11539 raft_consensus.cc:504] T 1cac39d5831d4b6887250f7a9d63dcd5 P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:14:32.872314 12292 raft_consensus.cc:482] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.872462 12292 raft_consensus.cc:504] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:32.873135 12292 leader_election.cc:288] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:14:32.873657 28129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7a8c95ac81942dabef98a07caa84082" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
W20240503 08:14:32.875669 27937 leader_election.cc:334] T 08fb64977309479b831635344969c662 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.877612 11603 raft_consensus.cc:504] T 41aa0e288ca4469ba34a07ff159ace91 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
W20240503 08:14:32.878670 30828 leader_election.cc:334] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.878783 30828 leader_election.cc:302] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:32.879042 28389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41aa0e288ca4469ba34a07ff159ace91" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:14:32.879246 28389 raft_consensus.cc:3037] T 41aa0e288ca4469ba34a07ff159ace91 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:32.880712 11603 leader_election.cc:288] T 41aa0e288ca4469ba34a07ff159ace91 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 election: Requested vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:14:32.881232 12048 raft_consensus.cc:482] T f46393257ce9471891698c36a07546ad P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.881395 12048 raft_consensus.cc:504] T f46393257ce9471891698c36a07546ad P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:32.882025 12048 leader_election.cc:288] T f46393257ce9471891698c36a07546ad P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:14:32.882390 12292 raft_consensus.cc:2726] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 08:14:32.882769 30960 leader_election.cc:334] T 41aa0e288ca4469ba34a07ff159ace91 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:32.883752 28067 leader_election.cc:334] T 1cac39d5831d4b6887250f7a9d63dcd5 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:32.887171 28067 leader_election.cc:334] T 1cac39d5831d4b6887250f7a9d63dcd5 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.887724 30759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cac39d5831d4b6887250f7a9d63dcd5" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:14:32.887970 30759 raft_consensus.cc:3037] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:14:32.890151 30828 consensus_peers.cc:462] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.891311 11144 raft_consensus.cc:482] T 46d27fa12990405cae5f94976ac7a014 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.891498 11144 raft_consensus.cc:504] T 46d27fa12990405cae5f94976ac7a014 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:14:32.891973 11539 leader_election.cc:288] T 1cac39d5831d4b6887250f7a9d63dcd5 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:14:32.892167 11144 leader_election.cc:288] T 46d27fa12990405cae5f94976ac7a014 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:14:32.892930 27999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d27fa12990405cae5f94976ac7a014" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:14:32.894323 30759 raft_consensus.cc:2454] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 2.
W20240503 08:14:32.895434 31095 leader_election.cc:334] T 46d27fa12990405cae5f94976ac7a014 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.895573 31095 leader_election.cc:302] T 46d27fa12990405cae5f94976ac7a014 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b00f244c69df401997ea99d492218a98; no voters: 833582ff68c94269be364ebd187a75db, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:32.895447 31141 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:32.895942 11539 raft_consensus.cc:482] T de5dfccb0359462dbb7da776293fa679 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.896108 11539 raft_consensus.cc:504] T de5dfccb0359462dbb7da776293fa679 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:32.896781 11539 leader_election.cc:288] T de5dfccb0359462dbb7da776293fa679 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:14:32.897475 11144 raft_consensus.cc:2726] T 46d27fa12990405cae5f94976ac7a014 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:14:32.897922 28068 leader_election.cc:302] T 1cac39d5831d4b6887250f7a9d63dcd5 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, 6d1d7de19459435fbde1999e3dfd226b; no voters: aae860d7ae2b4002bf7794c278f9cc56
W20240503 08:14:32.899974 28067 leader_election.cc:334] T de5dfccb0359462dbb7da776293fa679 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.900307 31025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de5dfccb0359462dbb7da776293fa679" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:14:32.900534 31025 raft_consensus.cc:2454] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:14:32.900900 28067 leader_election.cc:302] T de5dfccb0359462dbb7da776293fa679 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, ced21ca983b749d0b4d4f6a0e2426bf8; no voters: aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:32.901535 11539 raft_consensus.cc:2781] T 1cac39d5831d4b6887250f7a9d63dcd5 P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:14:32.902033 11965 raft_consensus.cc:2781] T de5dfccb0359462dbb7da776293fa679 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:14:32.902139 11965 raft_consensus.cc:482] T de5dfccb0359462dbb7da776293fa679 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting leader election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.902235 11965 raft_consensus.cc:3037] T de5dfccb0359462dbb7da776293fa679 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:32.907874 28389 raft_consensus.cc:2454] T 41aa0e288ca4469ba34a07ff159ace91 P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 2.
I20240503 08:14:32.910646 30962 leader_election.cc:302] T 41aa0e288ca4469ba34a07ff159ace91 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8, d09a11a03989421b9f28019dc1c14cc2; no voters: aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:32.905570 11965 raft_consensus.cc:504] T de5dfccb0359462dbb7da776293fa679 P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:32.911339 12192 raft_consensus.cc:2781] T 41aa0e288ca4469ba34a07ff159ace91 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:14:32.911480 12192 raft_consensus.cc:686] T 41aa0e288ca4469ba34a07ff159ace91 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 LEADER]: Becoming Leader. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Running, Role: LEADER
I20240503 08:14:32.911518 11965 leader_election.cc:288] T de5dfccb0359462dbb7da776293fa679 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
W20240503 08:14:32.911753 28327 leader_election.cc:334] T f46393257ce9471891698c36a07546ad P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.911760 12192 consensus_queue.cc:237] T 41aa0e288ca4469ba34a07ff159ace91 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:32.913563 31025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46393257ce9471891698c36a07546ad" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:14:32.913856 31025 raft_consensus.cc:2454] T f46393257ce9471891698c36a07546ad P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:14:32.916464 11989 raft_consensus.cc:504] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:14:32.917791 11989 leader_election.cc:288] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:14:32.917894 28129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c714f29abb140379b57e08eb927e2b7" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:14:32.918110 28129 raft_consensus.cc:3037] T 8c714f29abb140379b57e08eb927e2b7 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:14:32.919545 30828 leader_election.cc:334] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.920017 11714 raft_consensus.cc:482] T 46d27fa12990405cae5f94976ac7a014 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.920183 11714 raft_consensus.cc:504] T 46d27fa12990405cae5f94976ac7a014 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:14:32.920859 28129 raft_consensus.cc:2454] T 8c714f29abb140379b57e08eb927e2b7 P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 2.
I20240503 08:14:32.920953 11714 leader_election.cc:288] T 46d27fa12990405cae5f94976ac7a014 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
W20240503 08:14:32.922408 28067 leader_election.cc:334] T de5dfccb0359462dbb7da776293fa679 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.922408 30760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08fb64977309479b831635344969c662" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:14:32.922773 30760 raft_consensus.cc:2454] T 08fb64977309479b831635344969c662 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:14:32.922912 30829 leader_election.cc:302] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, ce1496a47d084365ad786cd09e472a2c; no voters: aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:32.923056 31160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d27fa12990405cae5f94976ac7a014" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:14:32.923274 31160 raft_consensus.cc:2454] T 46d27fa12990405cae5f94976ac7a014 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:14:32.923524 12211 raft_consensus.cc:2781] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:14:32.923643 12211 raft_consensus.cc:686] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c [term 2 LEADER]: Becoming Leader. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Running, Role: LEADER
I20240503 08:14:32.923763 27936 leader_election.cc:302] T 46d27fa12990405cae5f94976ac7a014 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, b00f244c69df401997ea99d492218a98; no voters:
I20240503 08:14:32.923883 12211 consensus_queue.cc:237] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
W20240503 08:14:32.927958 27937 leader_election.cc:334] T 46d27fa12990405cae5f94976ac7a014 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.928936 10051 raft_consensus.cc:482] T 6a96965691644db8b9461d146536a026 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.929111 10051 raft_consensus.cc:504] T 6a96965691644db8b9461d146536a026 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:32.929839 10051 leader_election.cc:288] T 6a96965691644db8b9461d146536a026 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:14:32.930644 27999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a96965691644db8b9461d146536a026" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:14:32.930876 27999 raft_consensus.cc:2454] T 6a96965691644db8b9461d146536a026 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 1.
I20240503 08:14:32.931387 31097 leader_election.cc:302] T 6a96965691644db8b9461d146536a026 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, b00f244c69df401997ea99d492218a98; no voters:
I20240503 08:14:32.932305 10051 raft_consensus.cc:2781] T 6a96965691644db8b9461d146536a026 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:14:32.932404 10051 raft_consensus.cc:482] T 6a96965691644db8b9461d146536a026 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting leader election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.932485 10051 raft_consensus.cc:3037] T 6a96965691644db8b9461d146536a026 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:14:32.933506 31095 leader_election.cc:334] T 6a96965691644db8b9461d146536a026 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.935519 10051 raft_consensus.cc:504] T 6a96965691644db8b9461d146536a026 P b00f244c69df401997ea99d492218a98 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:32.937824 11717 raft_consensus.cc:2781] T 46d27fa12990405cae5f94976ac7a014 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:14:32.937937 11717 raft_consensus.cc:482] T 46d27fa12990405cae5f94976ac7a014 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting leader election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.938040 11717 raft_consensus.cc:3037] T 46d27fa12990405cae5f94976ac7a014 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:32.940716 11717 raft_consensus.cc:504] T 46d27fa12990405cae5f94976ac7a014 P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:14:32.941365 11717 leader_election.cc:288] T 46d27fa12990405cae5f94976ac7a014 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
W20240503 08:14:32.942977 27937 leader_election.cc:334] T 46d27fa12990405cae5f94976ac7a014 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.943444 27999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a96965691644db8b9461d146536a026" candidate_uuid: "b00f244c69df401997ea99d492218a98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:14:32.943641 27999 raft_consensus.cc:3037] T 6a96965691644db8b9461d146536a026 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:14:32.945073 31095 leader_election.cc:334] T 6a96965691644db8b9461d146536a026 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.945389 10051 leader_election.cc:288] T 6a96965691644db8b9461d146536a026 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 election: Requested vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:14:32.946651 27999 raft_consensus.cc:2454] T 6a96965691644db8b9461d146536a026 P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00f244c69df401997ea99d492218a98 in term 2.
I20240503 08:14:32.947055 31097 leader_election.cc:302] T 6a96965691644db8b9461d146536a026 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 833582ff68c94269be364ebd187a75db, b00f244c69df401997ea99d492218a98; no voters: aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:32.947489 10051 raft_consensus.cc:2781] T 6a96965691644db8b9461d146536a026 P b00f244c69df401997ea99d492218a98 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:14:32.948998 10051 raft_consensus.cc:686] T 6a96965691644db8b9461d146536a026 P b00f244c69df401997ea99d492218a98 [term 2 LEADER]: Becoming Leader. State: Replica: b00f244c69df401997ea99d492218a98, State: Running, Role: LEADER
I20240503 08:14:32.949064 31160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d27fa12990405cae5f94976ac7a014" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:14:32.949394 31160 raft_consensus.cc:3037] T 46d27fa12990405cae5f94976ac7a014 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:32.952178 31160 raft_consensus.cc:2454] T 46d27fa12990405cae5f94976ac7a014 P b00f244c69df401997ea99d492218a98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 2.
I20240503 08:14:32.954766 27936 leader_election.cc:302] T 46d27fa12990405cae5f94976ac7a014 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 833582ff68c94269be364ebd187a75db, b00f244c69df401997ea99d492218a98; no voters: aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:32.955231 12319 raft_consensus.cc:2781] T 46d27fa12990405cae5f94976ac7a014 P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:14:32.959432 12319 raft_consensus.cc:686] T 46d27fa12990405cae5f94976ac7a014 P 833582ff68c94269be364ebd187a75db [term 2 LEADER]: Becoming Leader. State: Replica: 833582ff68c94269be364ebd187a75db, State: Running, Role: LEADER
I20240503 08:14:32.960633 12319 consensus_queue.cc:237] T 46d27fa12990405cae5f94976ac7a014 P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:14:32.961490 12314 raft_consensus.cc:482] T 8450e62352a74d2ea1471166d4cb1132 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.961676 12314 raft_consensus.cc:504] T 8450e62352a74d2ea1471166d4cb1132 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
W20240503 08:14:32.964042 28067 leader_election.cc:334] T 8450e62352a74d2ea1471166d4cb1132 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.949291 10051 consensus_queue.cc:237] T 6a96965691644db8b9461d146536a026 P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:32.966015 12286 raft_consensus.cc:482] T 8cc6ae30aa60495d8a4364a11d5bef79 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.966187 12286 raft_consensus.cc:504] T 8cc6ae30aa60495d8a4364a11d5bef79 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:14:32.966826 12286 leader_election.cc:288] T 8cc6ae30aa60495d8a4364a11d5bef79 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:14:32.968052 12314 leader_election.cc:288] T 8450e62352a74d2ea1471166d4cb1132 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
W20240503 08:14:32.968731 30695 leader_election.cc:334] T 8cc6ae30aa60495d8a4364a11d5bef79 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.969507 27999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cc6ae30aa60495d8a4364a11d5bef79" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:14:32.970183 30697 leader_election.cc:302] T 8cc6ae30aa60495d8a4364a11d5bef79 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b; no voters: 833582ff68c94269be364ebd187a75db, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:32.970607 12286 raft_consensus.cc:2726] T 8cc6ae30aa60495d8a4364a11d5bef79 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:14:32.976819 27858 catalog_manager.cc:5484] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c reported cstate change: term changed from 1 to 2, leader changed from aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) to ce1496a47d084365ad786cd09e472a2c (127.20.230.70). New cstate: current_term: 2 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:14:32.978106 28389 raft_consensus.cc:1261] T 607a503b64d8440a81cf66ab158ae273 P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:14:32.978732 12051 raft_consensus.cc:482] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.978879 12051 raft_consensus.cc:504] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:14:32.979050 12286 consensus_queue.cc:1035] T 607a503b64d8440a81cf66ab158ae273 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:32.979519 12051 leader_election.cc:288] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
W20240503 08:14:32.981550 28327 leader_election.cc:334] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.981986 28129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33f0f4bc5722485a8f978a02367ca80e" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48" is_pre_election: true
I20240503 08:14:32.982213 28129 raft_consensus.cc:2454] T 33f0f4bc5722485a8f978a02367ca80e P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:14:32.982584 28328 leader_election.cc:302] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, d09a11a03989421b9f28019dc1c14cc2; no voters: aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:32.982967 12051 raft_consensus.cc:2781] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:14:32.983065 12051 raft_consensus.cc:482] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.983152 12051 raft_consensus.cc:3037] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:32.987134 12051 raft_consensus.cc:504] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
W20240503 08:14:32.989421 30961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.66:37143, user_credentials={real_user=slave}} blocked reactor thread for 76282.4us
W20240503 08:14:32.989441 28327 leader_election.cc:334] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.990294 12314 raft_consensus.cc:482] T f7a8c95ac81942dabef98a07caa84082 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.990546 12314 raft_consensus.cc:504] T f7a8c95ac81942dabef98a07caa84082 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:32.990711 28327 leader_election.cc:302] T f46393257ce9471891698c36a07546ad P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8, d09a11a03989421b9f28019dc1c14cc2; no voters: aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:32.991221 12048 raft_consensus.cc:2781] T f46393257ce9471891698c36a07546ad P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:14:32.991326 12048 raft_consensus.cc:482] T f46393257ce9471891698c36a07546ad P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.991420 12048 raft_consensus.cc:3037] T f46393257ce9471891698c36a07546ad P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:14:32.992834 28067 leader_election.cc:334] T f7a8c95ac81942dabef98a07caa84082 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.993078 12314 leader_election.cc:288] T f7a8c95ac81942dabef98a07caa84082 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:14:32.993247 12314 raft_consensus.cc:482] T 978bf5783b0e47e3bfb30441adc600ae P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.993407 12314 raft_consensus.cc:504] T 978bf5783b0e47e3bfb30441adc600ae P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:32.993953 31026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8450e62352a74d2ea1471166d4cb1132" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:14:32.994199 31026 raft_consensus.cc:2454] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:14:32.994470 11965 raft_consensus.cc:482] T 362590d3d0834d0b8a672ab552f8c209 P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:32.994611 11965 raft_consensus.cc:504] T 362590d3d0834d0b8a672ab552f8c209 P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:32.995203 11965 leader_election.cc:288] T 362590d3d0834d0b8a672ab552f8c209 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
W20240503 08:14:32.995957 28067 leader_election.cc:334] T 978bf5783b0e47e3bfb30441adc600ae P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:32.998576 28067 leader_election.cc:334] T 362590d3d0834d0b8a672ab552f8c209 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:32.999859 12314 leader_election.cc:288] T 978bf5783b0e47e3bfb30441adc600ae P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:14:33.000478 12051 leader_election.cc:288] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:14:33.003643 31025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de5dfccb0359462dbb7da776293fa679" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:14:33.003873 31025 raft_consensus.cc:3037] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:33.008569 12314 raft_consensus.cc:482] T 17c6e3293aad4591a9c1395ebffff2e6 P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:33.008692 12048 raft_consensus.cc:504] T f46393257ce9471891698c36a07546ad P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:33.008770 12314 raft_consensus.cc:504] T 17c6e3293aad4591a9c1395ebffff2e6 P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:33.009459 12314 leader_election.cc:288] T 17c6e3293aad4591a9c1395ebffff2e6 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:14:33.009580 12048 leader_election.cc:288] T f46393257ce9471891698c36a07546ad P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:14:33.011432 31025 raft_consensus.cc:2454] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 2.
W20240503 08:14:33.011518 28067 leader_election.cc:334] T 17c6e3293aad4591a9c1395ebffff2e6 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:33.013008 30892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7a8c95ac81942dabef98a07caa84082" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:14:33.013221 30892 raft_consensus.cc:2454] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:14:33.014892 28066 leader_election.cc:302] T f7a8c95ac81942dabef98a07caa84082 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, ce1496a47d084365ad786cd09e472a2c; no voters: aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:33.018520 11965 raft_consensus.cc:2781] T f7a8c95ac81942dabef98a07caa84082 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:14:33.018631 11965 raft_consensus.cc:482] T f7a8c95ac81942dabef98a07caa84082 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting leader election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:33.018719 11965 raft_consensus.cc:3037] T f7a8c95ac81942dabef98a07caa84082 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:33.019515 27999 raft_consensus.cc:1261] T 607a503b64d8440a81cf66ab158ae273 P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:14:33.021641 11965 raft_consensus.cc:504] T f7a8c95ac81942dabef98a07caa84082 P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:33.023268 11965 leader_election.cc:288] T f7a8c95ac81942dabef98a07caa84082 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 election: Requested vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:14:33.023620 30892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7a8c95ac81942dabef98a07caa84082" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:14:33.023810 30892 raft_consensus.cc:3037] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:33.025511 27858 catalog_manager.cc:5484] T 41aa0e288ca4469ba34a07ff159ace91 P ced21ca983b749d0b4d4f6a0e2426bf8 reported cstate change: term changed from 1 to 2, leader changed from aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) to ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71). New cstate: current_term: 2 leader_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:14:33.026880 30892 raft_consensus.cc:2454] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 2.
I20240503 08:14:33.027346 28129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33f0f4bc5722485a8f978a02367ca80e" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b411c6bb25549368041773d7b710d48"
I20240503 08:14:33.027462 28066 leader_election.cc:302] T f7a8c95ac81942dabef98a07caa84082 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:14:33.027619 28129 raft_consensus.cc:3037] T 33f0f4bc5722485a8f978a02367ca80e P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:33.027745 11965 raft_consensus.cc:2781] T f7a8c95ac81942dabef98a07caa84082 P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:14:33.027861 11965 raft_consensus.cc:686] T f7a8c95ac81942dabef98a07caa84082 P 1b411c6bb25549368041773d7b710d48 [term 2 LEADER]: Becoming Leader. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Running, Role: LEADER
I20240503 08:14:33.028136 11965 consensus_queue.cc:237] T f7a8c95ac81942dabef98a07caa84082 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:33.030922 12286 consensus_queue.cc:1035] T 607a503b64d8440a81cf66ab158ae273 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 08:14:33.031001 28067 leader_election.cc:334] T f7a8c95ac81942dabef98a07caa84082 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:33.031630 11717 raft_consensus.cc:482] T ca38671a760d4534ab903579e0487772 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:33.031797 11717 raft_consensus.cc:504] T ca38671a760d4534ab903579e0487772 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:14:33.032421 11717 leader_election.cc:288] T ca38671a760d4534ab903579e0487772 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:14:33.033064 31161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca38671a760d4534ab903579e0487772" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:14:33.033286 31161 raft_consensus.cc:2454] T ca38671a760d4534ab903579e0487772 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:14:33.033721 27936 leader_election.cc:302] T ca38671a760d4534ab903579e0487772 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, b00f244c69df401997ea99d492218a98; no voters:
I20240503 08:14:33.034026 11717 raft_consensus.cc:2781] T ca38671a760d4534ab903579e0487772 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:14:33.034109 11717 raft_consensus.cc:482] T ca38671a760d4534ab903579e0487772 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting leader election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:33.034191 11717 raft_consensus.cc:3037] T ca38671a760d4534ab903579e0487772 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:33.036700 28129 raft_consensus.cc:2454] T 33f0f4bc5722485a8f978a02367ca80e P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 2.
I20240503 08:14:33.036705 11717 raft_consensus.cc:504] T ca38671a760d4534ab903579e0487772 P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:14:33.037580 28328 leader_election.cc:302] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, d09a11a03989421b9f28019dc1c14cc2; no voters: aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:33.037910 12051 raft_consensus.cc:2781] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:14:33.037909 28067 leader_election.cc:302] T 8450e62352a74d2ea1471166d4cb1132 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, ced21ca983b749d0b4d4f6a0e2426bf8; no voters: aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:33.038050 12051 raft_consensus.cc:686] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2 [term 2 LEADER]: Becoming Leader. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Running, Role: LEADER
I20240503 08:14:33.037345 11717 leader_election.cc:288] T ca38671a760d4534ab903579e0487772 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:14:33.038341 12051 consensus_queue.cc:237] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:14:33.038957 31161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca38671a760d4534ab903579e0487772" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:14:33.039180 31161 raft_consensus.cc:3037] T ca38671a760d4534ab903579e0487772 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:33.042294 31161 raft_consensus.cc:2454] T ca38671a760d4534ab903579e0487772 P b00f244c69df401997ea99d492218a98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 2.
I20240503 08:14:33.042788 27936 leader_election.cc:302] T ca38671a760d4534ab903579e0487772 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, b00f244c69df401997ea99d492218a98; no voters:
I20240503 08:14:33.043110 11717 raft_consensus.cc:2781] T ca38671a760d4534ab903579e0487772 P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:14:33.043241 11717 raft_consensus.cc:686] T ca38671a760d4534ab903579e0487772 P 833582ff68c94269be364ebd187a75db [term 2 LEADER]: Becoming Leader. State: Replica: 833582ff68c94269be364ebd187a75db, State: Running, Role: LEADER
I20240503 08:14:33.043520 11717 consensus_queue.cc:237] T ca38671a760d4534ab903579e0487772 P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
W20240503 08:14:33.044330 12320 log.cc:924] Time spent T 6a96965691644db8b9461d146536a026 P b00f244c69df401997ea99d492218a98: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20240503 08:14:33.045549 31025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "978bf5783b0e47e3bfb30441adc600ae" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:14:33.045835 31025 raft_consensus.cc:2454] T 978bf5783b0e47e3bfb30441adc600ae P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:14:33.046975 12314 raft_consensus.cc:482] T 8fc87f7f3c724cd4a270583ddebd87aa P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:33.047154 12314 raft_consensus.cc:504] T 8fc87f7f3c724cd4a270583ddebd87aa P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:14:33.048123 12314 leader_election.cc:288] T 8fc87f7f3c724cd4a270583ddebd87aa P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:14:33.048373 12314 raft_consensus.cc:2781] T 8450e62352a74d2ea1471166d4cb1132 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:14:33.048620 12314 raft_consensus.cc:482] T 8450e62352a74d2ea1471166d4cb1132 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting leader election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:33.048719 12314 raft_consensus.cc:3037] T 8450e62352a74d2ea1471166d4cb1132 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:33.052330 11714 raft_consensus.cc:482] T 8cc6ae30aa60495d8a4364a11d5bef79 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:33.052510 11714 raft_consensus.cc:504] T 8cc6ae30aa60495d8a4364a11d5bef79 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:14:33.053154 11714 leader_election.cc:288] T 8cc6ae30aa60495d8a4364a11d5bef79 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:14:33.053527 28129 raft_consensus.cc:1261] T 607a503b64d8440a81cf66ab158ae273 P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:14:33.054107 12286 consensus_queue.cc:1035] T 607a503b64d8440a81cf66ab158ae273 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:33.055310 11539 raft_consensus.cc:686] T 1cac39d5831d4b6887250f7a9d63dcd5 P 1b411c6bb25549368041773d7b710d48 [term 2 LEADER]: Becoming Leader. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Running, Role: LEADER
I20240503 08:14:33.055671 11539 consensus_queue.cc:237] T 1cac39d5831d4b6887250f7a9d63dcd5 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:14:33.056439 31025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17c6e3293aad4591a9c1395ebffff2e6" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:14:33.056675 31025 raft_consensus.cc:2454] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 2.
W20240503 08:14:33.058132 27937 leader_election.cc:334] T ca38671a760d4534ab903579e0487772 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:33.058403 27937 leader_election.cc:334] T 8cc6ae30aa60495d8a4364a11d5bef79 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:33.058540 27937 leader_election.cc:334] T ca38671a760d4534ab903579e0487772 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:33.066960 27860 catalog_manager.cc:5484] T 6a96965691644db8b9461d146536a026 P b00f244c69df401997ea99d492218a98 reported cstate change: term changed from 1 to 2, leader changed from aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) to b00f244c69df401997ea99d492218a98 (127.20.230.72). New cstate: current_term: 2 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:33.069324 28132 raft_consensus.cc:1261] T 2222ff7fb5184d6c86010e1475d6c1f7 P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:14:33.070312 12094 consensus_queue.cc:1035] T 2222ff7fb5184d6c86010e1475d6c1f7 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:33.071941 12048 raft_consensus.cc:482] T cdad6ef53b6f413ebfda67f9b42c9c23 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:33.072116 12048 raft_consensus.cc:504] T cdad6ef53b6f413ebfda67f9b42c9c23 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:14:33.072829 12048 leader_election.cc:288] T cdad6ef53b6f413ebfda67f9b42c9c23 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:14:33.073125 31160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc87f7f3c724cd4a270583ddebd87aa" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
I20240503 08:14:33.073343 31160 raft_consensus.cc:2454] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 1.
I20240503 08:14:33.075184 30760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdad6ef53b6f413ebfda67f9b42c9c23" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:14:33.075436 30760 raft_consensus.cc:2454] T cdad6ef53b6f413ebfda67f9b42c9c23 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
I20240503 08:14:33.076122 28390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "362590d3d0834d0b8a672ab552f8c209" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:14:33.076326 28390 raft_consensus.cc:2454] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 2.
I20240503 08:14:33.076498 12094 raft_consensus.cc:482] T cdad6ef53b6f413ebfda67f9b42c9c23 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:33.076673 12094 raft_consensus.cc:504] T cdad6ef53b6f413ebfda67f9b42c9c23 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:14:33.078929 28328 leader_election.cc:302] T cdad6ef53b6f413ebfda67f9b42c9c23 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:14:33.081509 12048 raft_consensus.cc:2781] T cdad6ef53b6f413ebfda67f9b42c9c23 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:14:33.081626 12048 raft_consensus.cc:482] T cdad6ef53b6f413ebfda67f9b42c9c23 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:33.081710 12048 raft_consensus.cc:3037] T cdad6ef53b6f413ebfda67f9b42c9c23 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:14:33.084292 28067 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.230.71:41915, user_credentials={real_user=slave}} blocked reactor thread for 46382.9us
I20240503 08:14:33.084700 12048 raft_consensus.cc:504] T cdad6ef53b6f413ebfda67f9b42c9c23 P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:14:33.085354 31025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46393257ce9471891698c36a07546ad" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:14:33.085582 31025 raft_consensus.cc:3037] T f46393257ce9471891698c36a07546ad P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:33.085669 12048 leader_election.cc:288] T cdad6ef53b6f413ebfda67f9b42c9c23 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:14:33.088534 12314 raft_consensus.cc:504] T 8450e62352a74d2ea1471166d4cb1132 P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:33.088943 31025 raft_consensus.cc:2454] T f46393257ce9471891698c36a07546ad P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 2.
I20240503 08:14:33.089243 12314 leader_election.cc:288] T 8450e62352a74d2ea1471166d4cb1132 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:14:33.094883 12094 leader_election.cc:288] T cdad6ef53b6f413ebfda67f9b42c9c23 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
W20240503 08:14:33.094810 27938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.69:34725, user_credentials={real_user=slave}} blocked reactor thread for 126443us
I20240503 08:14:33.095487 27938 leader_election.cc:302] T 08fb64977309479b831635344969c662 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, 833582ff68c94269be364ebd187a75db; no voters: aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:33.095412 28067 leader_election.cc:302] T de5dfccb0359462dbb7da776293fa679 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, ced21ca983b749d0b4d4f6a0e2426bf8; no voters: aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:33.095744 12243 raft_consensus.cc:2781] T 08fb64977309479b831635344969c662 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:14:33.095839 12243 raft_consensus.cc:482] T 08fb64977309479b831635344969c662 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting leader election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:33.095917 12243 raft_consensus.cc:3037] T 08fb64977309479b831635344969c662 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:33.095992 28067 leader_election.cc:302] T 978bf5783b0e47e3bfb30441adc600ae P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, ced21ca983b749d0b4d4f6a0e2426bf8; no voters: aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:33.096366 28067 leader_election.cc:302] T 17c6e3293aad4591a9c1395ebffff2e6 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, ced21ca983b749d0b4d4f6a0e2426bf8; no voters: aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:33.097285 12330 raft_consensus.cc:2781] T de5dfccb0359462dbb7da776293fa679 P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:14:33.097417 12330 raft_consensus.cc:686] T de5dfccb0359462dbb7da776293fa679 P 1b411c6bb25549368041773d7b710d48 [term 2 LEADER]: Becoming Leader. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Running, Role: LEADER
I20240503 08:14:33.097671 12330 consensus_queue.cc:237] T de5dfccb0359462dbb7da776293fa679 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:33.098731 12243 raft_consensus.cc:504] T 08fb64977309479b831635344969c662 P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:14:33.098937 12314 raft_consensus.cc:2781] T 978bf5783b0e47e3bfb30441adc600ae P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:14:33.099023 12314 raft_consensus.cc:482] T 978bf5783b0e47e3bfb30441adc600ae P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting leader election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:33.099112 12314 raft_consensus.cc:3037] T 978bf5783b0e47e3bfb30441adc600ae P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:33.099419 12243 leader_election.cc:288] T 08fb64977309479b831635344969c662 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:14:33.102586 11965 raft_consensus.cc:2781] T 17c6e3293aad4591a9c1395ebffff2e6 P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 08:14:33.102727 11965 raft_consensus.cc:482] T 17c6e3293aad4591a9c1395ebffff2e6 P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Starting leader election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:33.102720 31025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8450e62352a74d2ea1471166d4cb1132" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:14:33.102826 11965 raft_consensus.cc:3037] T 17c6e3293aad4591a9c1395ebffff2e6 P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:14:33.102906 31025 raft_consensus.cc:3037] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:33.105726 31025 raft_consensus.cc:2454] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 2.
I20240503 08:14:33.105937 11965 raft_consensus.cc:504] T 17c6e3293aad4591a9c1395ebffff2e6 P 1b411c6bb25549368041773d7b710d48 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
W20240503 08:14:33.113009 30959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.72:40893, user_credentials={real_user=slave}} blocked reactor thread for 134922us
I20240503 08:14:33.130954 28373 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 08:14:33.132568 11965 leader_election.cc:288] T 17c6e3293aad4591a9c1395ebffff2e6 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 3 election: Requested vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
W20240503 08:14:33.145747 28066 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.70:33747, user_credentials={real_user=slave}} blocked reactor thread for 90631.1us
I20240503 08:14:33.146315 28066 leader_election.cc:302] T 8fc87f7f3c724cd4a270583ddebd87aa P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, b00f244c69df401997ea99d492218a98; no voters:
I20240503 08:14:33.147385 12313 raft_consensus.cc:2781] T 8fc87f7f3c724cd4a270583ddebd87aa P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:14:33.147998 12313 raft_consensus.cc:482] T 8fc87f7f3c724cd4a270583ddebd87aa P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Starting leader election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:33.148197 12313 raft_consensus.cc:3037] T 8fc87f7f3c724cd4a270583ddebd87aa P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:33.150156 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:14:33.151432 12313 raft_consensus.cc:504] T 8fc87f7f3c724cd4a270583ddebd87aa P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:14:33.152830 12313 leader_election.cc:288] T 8fc87f7f3c724cd4a270583ddebd87aa P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:14:33.154003 30759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cc6ae30aa60495d8a4364a11d5bef79" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:14:33.154250 30759 raft_consensus.cc:2454] T 8cc6ae30aa60495d8a4364a11d5bef79 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:14:33.154799 30759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08fb64977309479b831635344969c662" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:14:33.154974 30759 raft_consensus.cc:3037] T 08fb64977309479b831635344969c662 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:14:33.156586 30697 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.68:46211, user_credentials={real_user=slave}} blocked reactor thread for 77993us
I20240503 08:14:33.157334 28390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdad6ef53b6f413ebfda67f9b42c9c23" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:14:33.157574 28390 raft_consensus.cc:2379] T cdad6ef53b6f413ebfda67f9b42c9c23 P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d1d7de19459435fbde1999e3dfd226b in current term 2: Already voted for candidate d09a11a03989421b9f28019dc1c14cc2 in this term.
I20240503 08:14:33.157791 30760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdad6ef53b6f413ebfda67f9b42c9c23" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:14:33.157716 30759 raft_consensus.cc:2454] T 08fb64977309479b831635344969c662 P 6d1d7de19459435fbde1999e3dfd226b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 2.
I20240503 08:14:33.157979 30760 raft_consensus.cc:3037] T cdad6ef53b6f413ebfda67f9b42c9c23 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:33.160539 30760 raft_consensus.cc:2454] T cdad6ef53b6f413ebfda67f9b42c9c23 P 6d1d7de19459435fbde1999e3dfd226b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 2.
I20240503 08:14:33.165148 31160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc87f7f3c724cd4a270583ddebd87aa" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98"
I20240503 08:14:33.165376 31160 raft_consensus.cc:3037] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:33.168067 27983 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:14:33.168501 31160 raft_consensus.cc:2454] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 2.
I20240503 08:14:33.171964 27858 catalog_manager.cc:5484] T 46d27fa12990405cae5f94976ac7a014 P 833582ff68c94269be364ebd187a75db reported cstate change: term changed from 1 to 2, leader changed from aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) to 833582ff68c94269be364ebd187a75db (127.20.230.65). New cstate: current_term: 2 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:14:33.174137 28328 leader_election.cc:302] T cdad6ef53b6f413ebfda67f9b42c9c23 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:14:33.175479 12051 raft_consensus.cc:2781] T cdad6ef53b6f413ebfda67f9b42c9c23 P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:14:33.176143 12051 raft_consensus.cc:686] T cdad6ef53b6f413ebfda67f9b42c9c23 P d09a11a03989421b9f28019dc1c14cc2 [term 2 LEADER]: Becoming Leader. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Running, Role: LEADER
I20240503 08:14:33.176406 12051 consensus_queue.cc:237] T cdad6ef53b6f413ebfda67f9b42c9c23 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:14:33.180966 28069 leader_election.cc:302] T 362590d3d0834d0b8a672ab552f8c209 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, d09a11a03989421b9f28019dc1c14cc2; no voters: aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:33.183761 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240503 08:14:33.188884 28067 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.230.67:39919, user_credentials={real_user=slave}} blocked reactor thread for 92086.1us
I20240503 08:14:33.192065 12313 raft_consensus.cc:2781] T 362590d3d0834d0b8a672ab552f8c209 P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 08:14:33.192263 12313 raft_consensus.cc:482] T 362590d3d0834d0b8a672ab552f8c209 P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Starting leader election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:33.192446 12313 raft_consensus.cc:3037] T 362590d3d0834d0b8a672ab552f8c209 P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Advancing to term 3
W20240503 08:14:33.193490 27937 leader_election.cc:334] T 08fb64977309479b831635344969c662 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:33.200423 28066 leader_election.cc:302] T 8fc87f7f3c724cd4a270583ddebd87aa P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, b00f244c69df401997ea99d492218a98; no voters:
I20240503 08:14:33.201165 11539 raft_consensus.cc:2781] T 8fc87f7f3c724cd4a270583ddebd87aa P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:14:33.201341 11539 raft_consensus.cc:686] T 8fc87f7f3c724cd4a270583ddebd87aa P 1b411c6bb25549368041773d7b710d48 [term 2 LEADER]: Becoming Leader. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Running, Role: LEADER
I20240503 08:14:33.201637 11539 consensus_queue.cc:237] T 8fc87f7f3c724cd4a270583ddebd87aa P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:14:33.202157 27938 leader_election.cc:302] T 8cc6ae30aa60495d8a4364a11d5bef79 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, 833582ff68c94269be364ebd187a75db; no voters: aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:33.101676 12314 raft_consensus.cc:504] T 978bf5783b0e47e3bfb30441adc600ae P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:33.205197 12314 leader_election.cc:288] T 978bf5783b0e47e3bfb30441adc600ae P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:14:33.206374 27858 catalog_manager.cc:5484] T 1cac39d5831d4b6887250f7a9d63dcd5 P 1b411c6bb25549368041773d7b710d48 reported cstate change: term changed from 1 to 2, leader changed from aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) to 1b411c6bb25549368041773d7b710d48 (127.20.230.66). New cstate: current_term: 2 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:14:33.208937 31025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17c6e3293aad4591a9c1395ebffff2e6" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:14:33.209144 31025 raft_consensus.cc:3037] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:14:33.212010 31025 raft_consensus.cc:2454] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 3.
I20240503 08:14:33.212929 12319 raft_consensus.cc:2781] T 8cc6ae30aa60495d8a4364a11d5bef79 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:14:33.213048 12319 raft_consensus.cc:482] T 8cc6ae30aa60495d8a4364a11d5bef79 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting leader election (detected failure of leader aae860d7ae2b4002bf7794c278f9cc56)
I20240503 08:14:33.213136 12319 raft_consensus.cc:3037] T 8cc6ae30aa60495d8a4364a11d5bef79 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:33.214185 12313 raft_consensus.cc:504] T 362590d3d0834d0b8a672ab552f8c209 P 1b411c6bb25549368041773d7b710d48 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:33.216152 12319 raft_consensus.cc:504] T 8cc6ae30aa60495d8a4364a11d5bef79 P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:14:33.216867 12319 leader_election.cc:288] T 8cc6ae30aa60495d8a4364a11d5bef79 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 election: Requested vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:14:33.219420 12313 leader_election.cc:288] T 362590d3d0834d0b8a672ab552f8c209 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 3 election: Requested vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
W20240503 08:14:33.223004 30695 leader_election.cc:334] T cdad6ef53b6f413ebfda67f9b42c9c23 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:33.223166 30695 leader_election.cc:302] T cdad6ef53b6f413ebfda67f9b42c9c23 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b; no voters: aae860d7ae2b4002bf7794c278f9cc56, d09a11a03989421b9f28019dc1c14cc2
I20240503 08:14:33.224010 27938 leader_election.cc:302] T 08fb64977309479b831635344969c662 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, 833582ff68c94269be364ebd187a75db; no voters: aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:33.224638 11717 raft_consensus.cc:2781] T 08fb64977309479b831635344969c662 P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:14:33.224795 11717 raft_consensus.cc:686] T 08fb64977309479b831635344969c662 P 833582ff68c94269be364ebd187a75db [term 2 LEADER]: Becoming Leader. State: Replica: 833582ff68c94269be364ebd187a75db, State: Running, Role: LEADER
I20240503 08:14:33.225054 11717 consensus_queue.cc:237] T 08fb64977309479b831635344969c662 P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:14:33.239895 28067 leader_election.cc:302] T 8450e62352a74d2ea1471166d4cb1132 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:14:33.240702 12094 raft_consensus.cc:2726] T cdad6ef53b6f413ebfda67f9b42c9c23 P 6d1d7de19459435fbde1999e3dfd226b [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:14:33.242484 12313 raft_consensus.cc:2781] T 8450e62352a74d2ea1471166d4cb1132 P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:14:33.242650 12313 raft_consensus.cc:686] T 8450e62352a74d2ea1471166d4cb1132 P 1b411c6bb25549368041773d7b710d48 [term 2 LEADER]: Becoming Leader. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Running, Role: LEADER
I20240503 08:14:33.242920 12313 consensus_queue.cc:237] T 8450e62352a74d2ea1471166d4cb1132 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:33.248848 28067 leader_election.cc:302] T 17c6e3293aad4591a9c1395ebffff2e6 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:14:33.249516 28327 leader_election.cc:302] T f46393257ce9471891698c36a07546ad P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8, d09a11a03989421b9f28019dc1c14cc2; no voters:
W20240503 08:14:33.262514 28067 leader_election.cc:334] T 8450e62352a74d2ea1471166d4cb1132 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:33.262955 28067 leader_election.cc:334] T 8fc87f7f3c724cd4a270583ddebd87aa P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:33.266449 30760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cc6ae30aa60495d8a4364a11d5bef79" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:14:33.266708 30760 raft_consensus.cc:3037] T 8cc6ae30aa60495d8a4364a11d5bef79 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:14:33.266918 28327 leader_election.cc:334] T f46393257ce9471891698c36a07546ad P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:33.269517 30760 raft_consensus.cc:2454] T 8cc6ae30aa60495d8a4364a11d5bef79 P 6d1d7de19459435fbde1999e3dfd226b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 2.
W20240503 08:14:33.270117 30828 consensus_peers.cc:462] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:33.270502 12327 raft_consensus.cc:2781] T f46393257ce9471891698c36a07546ad P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:14:33.271569 28389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "362590d3d0834d0b8a672ab552f8c209" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:14:33.271785 28389 raft_consensus.cc:3037] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:14:33.276028 27858 catalog_manager.cc:5484] T ca38671a760d4534ab903579e0487772 P 833582ff68c94269be364ebd187a75db reported cstate change: term changed from 1 to 2, leader changed from aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) to 833582ff68c94269be364ebd187a75db (127.20.230.65). New cstate: current_term: 2 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:14:33.277952 12313 raft_consensus.cc:2781] T 17c6e3293aad4591a9c1395ebffff2e6 P 1b411c6bb25549368041773d7b710d48 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:14:33.280761 12313 raft_consensus.cc:686] T 17c6e3293aad4591a9c1395ebffff2e6 P 1b411c6bb25549368041773d7b710d48 [term 3 LEADER]: Becoming Leader. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Running, Role: LEADER
I20240503 08:14:33.281060 12313 consensus_queue.cc:237] T 17c6e3293aad4591a9c1395ebffff2e6 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:33.281697 31026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "978bf5783b0e47e3bfb30441adc600ae" candidate_uuid: "1b411c6bb25549368041773d7b710d48" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:14:33.281908 31026 raft_consensus.cc:3037] T 978bf5783b0e47e3bfb30441adc600ae P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:33.282879 28389 raft_consensus.cc:2454] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 3.
I20240503 08:14:33.284610 31026 raft_consensus.cc:2454] T 978bf5783b0e47e3bfb30441adc600ae P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b411c6bb25549368041773d7b710d48 in term 2.
I20240503 08:14:33.287945 12327 raft_consensus.cc:686] T f46393257ce9471891698c36a07546ad P d09a11a03989421b9f28019dc1c14cc2 [term 2 LEADER]: Becoming Leader. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Running, Role: LEADER
I20240503 08:14:33.288251 12327 consensus_queue.cc:237] T f46393257ce9471891698c36a07546ad P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
W20240503 08:14:33.290940 28067 leader_election.cc:334] T 8fc87f7f3c724cd4a270583ddebd87aa P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:33.291467 27938 leader_election.cc:302] T 8cc6ae30aa60495d8a4364a11d5bef79 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, 833582ff68c94269be364ebd187a75db; no voters:
W20240503 08:14:33.294377 28067 leader_election.cc:334] T 17c6e3293aad4591a9c1395ebffff2e6 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:33.294977 11717 raft_consensus.cc:2781] T 8cc6ae30aa60495d8a4364a11d5bef79 P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:14:33.295267 11717 raft_consensus.cc:686] T 8cc6ae30aa60495d8a4364a11d5bef79 P 833582ff68c94269be364ebd187a75db [term 2 LEADER]: Becoming Leader. State: Replica: 833582ff68c94269be364ebd187a75db, State: Running, Role: LEADER
I20240503 08:14:33.295598 11717 consensus_queue.cc:237] T 8cc6ae30aa60495d8a4364a11d5bef79 P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:14:33.298349 28067 leader_election.cc:302] T 978bf5783b0e47e3bfb30441adc600ae P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:14:33.298609 11539 raft_consensus.cc:2781] T 978bf5783b0e47e3bfb30441adc600ae P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:14:33.298761 11539 raft_consensus.cc:686] T 978bf5783b0e47e3bfb30441adc600ae P 1b411c6bb25549368041773d7b710d48 [term 2 LEADER]: Becoming Leader. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Running, Role: LEADER
I20240503 08:14:33.299036 11539 consensus_queue.cc:237] T 978bf5783b0e47e3bfb30441adc600ae P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
W20240503 08:14:33.305802 28067 leader_election.cc:334] T 362590d3d0834d0b8a672ab552f8c209 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:33.306041 28067 leader_election.cc:334] T 978bf5783b0e47e3bfb30441adc600ae P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:33.309301 27937 leader_election.cc:334] T 8cc6ae30aa60495d8a4364a11d5bef79 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:33.316704 30695 consensus_peers.cc:485] T 2222ff7fb5184d6c86010e1475d6c1f7 P 6d1d7de19459435fbde1999e3dfd226b -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:33.319548 28327 leader_election.cc:334] T cdad6ef53b6f413ebfda67f9b42c9c23 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:33.319994 28327 leader_election.cc:334] T cdad6ef53b6f413ebfda67f9b42c9c23 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:33.333655 31160 raft_consensus.cc:1261] T d9098f3d36b24d578a3f8d7d3ccad052 P b00f244c69df401997ea99d492218a98 [term 2 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:14:33.334517 12319 consensus_queue.cc:1035] T d9098f3d36b24d578a3f8d7d3ccad052 P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:33.337010 28069 leader_election.cc:302] T 362590d3d0834d0b8a672ab552f8c209 P 1b411c6bb25549368041773d7b710d48 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b411c6bb25549368041773d7b710d48, d09a11a03989421b9f28019dc1c14cc2; no voters: aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:33.338485 11539 raft_consensus.cc:2781] T 362590d3d0834d0b8a672ab552f8c209 P 1b411c6bb25549368041773d7b710d48 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:14:33.338781 11539 raft_consensus.cc:686] T 362590d3d0834d0b8a672ab552f8c209 P 1b411c6bb25549368041773d7b710d48 [term 3 LEADER]: Becoming Leader. State: Replica: 1b411c6bb25549368041773d7b710d48, State: Running, Role: LEADER
I20240503 08:14:33.339134 11539 consensus_queue.cc:237] T 362590d3d0834d0b8a672ab552f8c209 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
W20240503 08:14:33.351294 27937 consensus_peers.cc:485] T d9098f3d36b24d578a3f8d7d3ccad052 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:33.362309 28389 raft_consensus.cc:1261] T 41aa0e288ca4469ba34a07ff159ace91 P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Refusing update from remote peer ced21ca983b749d0b4d4f6a0e2426bf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:14:33.364109 12322 consensus_queue.cc:1035] T 41aa0e288ca4469ba34a07ff159ace91 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:33.325187 27858 catalog_manager.cc:5484] T 8cc6ae30aa60495d8a4364a11d5bef79 P 833582ff68c94269be364ebd187a75db reported cstate change: term changed from 1 to 2, leader changed from aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) to 833582ff68c94269be364ebd187a75db (127.20.230.65). New cstate: current_term: 2 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:14:33.370862 27858 catalog_manager.cc:5484] T 08fb64977309479b831635344969c662 P 833582ff68c94269be364ebd187a75db reported cstate change: term changed from 1 to 2, leader changed from aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) to 833582ff68c94269be364ebd187a75db (127.20.230.65). New cstate: current_term: 2 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:14:33.374678 28129 raft_consensus.cc:1261] T 07866de372e541ec9dca8a3d42f8b0ab P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:14:33.375746 12356 consensus_queue.cc:1035] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:33.383718 27860 catalog_manager.cc:5484] T f46393257ce9471891698c36a07546ad P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: term changed from 1 to 2, leader changed from aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) to d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68). New cstate: current_term: 2 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:14:33.384246 27860 catalog_manager.cc:5484] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: term changed from 1 to 2, leader changed from aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) to d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68). New cstate: current_term: 2 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:14:33.384634 27860 catalog_manager.cc:5484] T cdad6ef53b6f413ebfda67f9b42c9c23 P d09a11a03989421b9f28019dc1c14cc2 reported cstate change: term changed from 1 to 2, leader changed from aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) to d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68). New cstate: current_term: 2 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:14:33.390933 27855 catalog_manager.cc:5484] T 8fc87f7f3c724cd4a270583ddebd87aa P 1b411c6bb25549368041773d7b710d48 reported cstate change: term changed from 1 to 2, leader changed from aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) to 1b411c6bb25549368041773d7b710d48 (127.20.230.66). New cstate: current_term: 2 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } }
I20240503 08:14:33.391408 27855 catalog_manager.cc:5484] T f7a8c95ac81942dabef98a07caa84082 P 1b411c6bb25549368041773d7b710d48 reported cstate change: term changed from 1 to 2, leader changed from aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) to 1b411c6bb25549368041773d7b710d48 (127.20.230.66). New cstate: current_term: 2 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:33.391788 27855 catalog_manager.cc:5484] T 17c6e3293aad4591a9c1395ebffff2e6 P 1b411c6bb25549368041773d7b710d48 reported cstate change: term changed from 2 to 3, leader changed from aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) to 1b411c6bb25549368041773d7b710d48 (127.20.230.66). New cstate: current_term: 3 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:33.392261 27855 catalog_manager.cc:5484] T 8450e62352a74d2ea1471166d4cb1132 P 1b411c6bb25549368041773d7b710d48 reported cstate change: term changed from 1 to 2, leader changed from aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) to 1b411c6bb25549368041773d7b710d48 (127.20.230.66). New cstate: current_term: 2 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:14:33.392601 27855 catalog_manager.cc:5484] T de5dfccb0359462dbb7da776293fa679 P 1b411c6bb25549368041773d7b710d48 reported cstate change: term changed from 1 to 2, leader changed from aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) to 1b411c6bb25549368041773d7b710d48 (127.20.230.66). New cstate: current_term: 2 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
W20240503 08:14:33.394440 30960 consensus_peers.cc:485] T 41aa0e288ca4469ba34a07ff159ace91 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:33.394657 28327 consensus_peers.cc:485] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:33.408706 27858 catalog_manager.cc:5484] T 978bf5783b0e47e3bfb30441adc600ae P 1b411c6bb25549368041773d7b710d48 reported cstate change: term changed from 1 to 2, leader changed from aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) to 1b411c6bb25549368041773d7b710d48 (127.20.230.66). New cstate: current_term: 2 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:14:33.409214 27858 catalog_manager.cc:5484] T 362590d3d0834d0b8a672ab552f8c209 P 1b411c6bb25549368041773d7b710d48 reported cstate change: term changed from 2 to 3, leader changed from aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) to 1b411c6bb25549368041773d7b710d48 (127.20.230.66). New cstate: current_term: 3 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:33.419589 28132 raft_consensus.cc:1261] T 8c714f29abb140379b57e08eb927e2b7 P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:14:33.420058 11989 consensus_queue.cc:1035] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:14:33.428334 30828 consensus_peers.cc:485] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:33.444562 31095 consensus_peers.cc:485] T 6a96965691644db8b9461d146536a026 P b00f244c69df401997ea99d492218a98 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:33.458438 31160 raft_consensus.cc:1261] T 46d27fa12990405cae5f94976ac7a014 P b00f244c69df401997ea99d492218a98 [term 2 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:14:33.459635 11717 consensus_queue.cc:1035] T 46d27fa12990405cae5f94976ac7a014 P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:33.468545 28132 raft_consensus.cc:1261] T 33f0f4bc5722485a8f978a02367ca80e P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:14:33.469174 12356 consensus_queue.cc:1035] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:14:33.471011 27937 consensus_peers.cc:485] T 46d27fa12990405cae5f94976ac7a014 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:33.477483 28327 consensus_peers.cc:485] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:33.497869 27937 consensus_peers.cc:485] T ca38671a760d4534ab903579e0487772 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:33.510059 30760 raft_consensus.cc:1261] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b [term 2 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:14:33.510740 12313 consensus_queue.cc:1035] T 1cac39d5831d4b6887250f7a9d63dcd5 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:14:33.515067 28067 consensus_peers.cc:485] T 1cac39d5831d4b6887250f7a9d63dcd5 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:33.524730 28067 consensus_peers.cc:485] T f7a8c95ac81942dabef98a07caa84082 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:33.539686 30892 raft_consensus.cc:1261] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 08:14:33.540242 11539 consensus_queue.cc:1035] T f7a8c95ac81942dabef98a07caa84082 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:33.551059 31160 raft_consensus.cc:1261] T ca38671a760d4534ab903579e0487772 P b00f244c69df401997ea99d492218a98 [term 2 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:14:33.552192 11717 consensus_queue.cc:1035] T ca38671a760d4534ab903579e0487772 P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:33.584143 28000 raft_consensus.cc:1261] T 6a96965691644db8b9461d146536a026 P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 08:14:33.584820 12342 consensus_queue.cc:1035] T 6a96965691644db8b9461d146536a026 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 08:14:33.592098 28327 consensus_peers.cc:485] T cdad6ef53b6f413ebfda67f9b42c9c23 P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:33.603868 31160 raft_consensus.cc:1261] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98 [term 2 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
W20240503 08:14:33.604025 28067 consensus_peers.cc:485] T de5dfccb0359462dbb7da776293fa679 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:33.604470 11539 consensus_queue.cc:1035] T 8fc87f7f3c724cd4a270583ddebd87aa P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 08:14:33.609962 28067 consensus_peers.cc:485] T 8fc87f7f3c724cd4a270583ddebd87aa P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:33.650002 31025 raft_consensus.cc:1261] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:14:33.650699 11539 consensus_queue.cc:1035] T de5dfccb0359462dbb7da776293fa679 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:33.662719 31025 raft_consensus.cc:1261] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:14:33.664121 11539 consensus_queue.cc:1035] T 8450e62352a74d2ea1471166d4cb1132 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:33.682013 30759 raft_consensus.cc:1261] T cdad6ef53b6f413ebfda67f9b42c9c23 P 6d1d7de19459435fbde1999e3dfd226b [term 2 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:14:33.682672 12327 consensus_queue.cc:1035] T cdad6ef53b6f413ebfda67f9b42c9c23 P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 08:14:33.690766 28067 consensus_peers.cc:485] T 8450e62352a74d2ea1471166d4cb1132 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:33.694108 30760 raft_consensus.cc:1261] T 08fb64977309479b831635344969c662 P 6d1d7de19459435fbde1999e3dfd226b [term 2 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:14:33.694715 12243 consensus_queue.cc:1035] T 08fb64977309479b831635344969c662 P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 08:14:33.728174 27937 consensus_peers.cc:485] T 08fb64977309479b831635344969c662 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:33.730800 28389 raft_consensus.cc:1261] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [term 3 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 08:14:33.731359 12313 consensus_queue.cc:1035] T 362590d3d0834d0b8a672ab552f8c209 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 08:14:33.765569 28067 consensus_peers.cc:485] T 362590d3d0834d0b8a672ab552f8c209 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:33.777992 28327 consensus_peers.cc:485] T f46393257ce9471891698c36a07546ad P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:33.794438 31026 raft_consensus.cc:1261] T f46393257ce9471891698c36a07546ad P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:14:33.797605 12355 consensus_queue.cc:1035] T f46393257ce9471891698c36a07546ad P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:33.800951 31025 raft_consensus.cc:1261] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 3 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 08:14:33.801695 12313 consensus_queue.cc:1035] T 17c6e3293aad4591a9c1395ebffff2e6 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 08:14:33.821995 28067 consensus_peers.cc:485] T 17c6e3293aad4591a9c1395ebffff2e6 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:33.832924 27937 consensus_peers.cc:485] T 8cc6ae30aa60495d8a4364a11d5bef79 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:33.841255 30761 raft_consensus.cc:1261] T 8cc6ae30aa60495d8a4364a11d5bef79 P 6d1d7de19459435fbde1999e3dfd226b [term 2 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:14:33.842000 12243 consensus_queue.cc:1035] T 8cc6ae30aa60495d8a4364a11d5bef79 P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:33.851649 31026 raft_consensus.cc:1261] T 978bf5783b0e47e3bfb30441adc600ae P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
W20240503 08:14:33.852279 30828 consensus_peers.cc:462] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:33.853238 12314 consensus_queue.cc:1035] T 978bf5783b0e47e3bfb30441adc600ae P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 08:14:33.881052 28067 consensus_peers.cc:485] T 978bf5783b0e47e3bfb30441adc600ae P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:34.007400 28327 consensus_peers.cc:485] T eeee6a265fc94e91bf9daa8301662e5f P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:34.015590 30695 consensus_peers.cc:485] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:14:34.019522 27983 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:34.035037 28111 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240503 08:14:34.054453 31095 consensus_peers.cc:485] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:14:34.063102 30742 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:34.070540 31141 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 08:14:34.087150 28067 consensus_peers.cc:485] T df90a0d9d338416b81e253fa23542cca P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:34.087651 30828 consensus_peers.cc:485] T 045b3ce4b1d44f94a802b41a12e2704d P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:14:34.102010 30874 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:14:34.117321 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240503 08:14:34.130385 31095 consensus_peers.cc:485] T 2c595805729d48fd974d761ed8a60158 P b00f244c69df401997ea99d492218a98 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:34.175554 27937 consensus_peers.cc:485] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:34.187214 28327 consensus_peers.cc:485] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:14:34.189369 28439 maintenance_manager.cc:419] P d09a11a03989421b9f28019dc1c14cc2: Scheduling FlushMRSOp(cdad6ef53b6f413ebfda67f9b42c9c23): perf score=0.033342
I20240503 08:14:34.198100 28373 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:14:34.210343 31095 consensus_peers.cc:485] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:34.214406 30828 consensus_peers.cc:485] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:34.237382 30828 consensus_peers.cc:462] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:34.252775 28327 consensus_peers.cc:485] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:14:34.281024 28330 maintenance_manager.cc:643] P d09a11a03989421b9f28019dc1c14cc2: FlushMRSOp(cdad6ef53b6f413ebfda67f9b42c9c23) complete. Timing: real 0.091s user 0.030s sys 0.011s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":8030,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":2146,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":973,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":7949,"threads_started":1}
I20240503 08:14:34.283851 28439 maintenance_manager.cc:419] P d09a11a03989421b9f28019dc1c14cc2: Scheduling UndoDeltaBlockGCOp(cdad6ef53b6f413ebfda67f9b42c9c23): 4923 bytes on disk
W20240503 08:14:34.284229 30828 consensus_peers.cc:485] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:14:34.284412 28330 maintenance_manager.cc:643] P d09a11a03989421b9f28019dc1c14cc2: UndoDeltaBlockGCOp(cdad6ef53b6f413ebfda67f9b42c9c23) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240503 08:14:34.287458 28439 maintenance_manager.cc:419] P d09a11a03989421b9f28019dc1c14cc2: Scheduling FlushMRSOp(41aa0e288ca4469ba34a07ff159ace91): perf score=0.033368
I20240503 08:14:34.305310 28389 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e3a6b72060804beb81a177988d287f63"
dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47688
I20240503 08:14:34.305513 28389 raft_consensus.cc:595] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Received request to transfer leadership
W20240503 08:14:34.318084 28327 consensus_peers.cc:485] T bdfe68a127f043f08f04749843a38375 P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:14:34.318780 28049 maintenance_manager.cc:419] P 833582ff68c94269be364ebd187a75db: Scheduling FlushMRSOp(0ceedfb43869465bb63931c39c28583a): perf score=0.033382
W20240503 08:14:34.324028 28067 consensus_peers.cc:485] T 2dce5b2aaea045ccb7424cbbd44576c2 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:34.325320 30828 consensus_peers.cc:485] T 0f25688be4e74d31aaab85b7a4329a34 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:14:34.327597 28330 maintenance_manager.cc:643] P d09a11a03989421b9f28019dc1c14cc2: FlushMRSOp(41aa0e288ca4469ba34a07ff159ace91) complete. Timing: real 0.040s user 0.033s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":38,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":1393,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":733,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635}
I20240503 08:14:34.331321 28439 maintenance_manager.cc:419] P d09a11a03989421b9f28019dc1c14cc2: Scheduling UndoDeltaBlockGCOp(41aa0e288ca4469ba34a07ff159ace91): 4583 bytes on disk
I20240503 08:14:34.331773 28330 maintenance_manager.cc:643] P d09a11a03989421b9f28019dc1c14cc2: UndoDeltaBlockGCOp(41aa0e288ca4469ba34a07ff159ace91) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
W20240503 08:14:34.341754 28067 consensus_peers.cc:485] T 099e37089b2e42b4839e681f8f37365d P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:34.348656 28067 consensus_peers.cc:485] T 658865ee3f8d424ca5cd35e12e6d7c77 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:14:34.364264 27940 maintenance_manager.cc:643] P 833582ff68c94269be364ebd187a75db: FlushMRSOp(0ceedfb43869465bb63931c39c28583a) complete. Timing: real 0.045s user 0.027s sys 0.009s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":315,"dirs.run_cpu_time_us":167,"dirs.run_wall_time_us":1632,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3568,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":553,"threads_started":2,"wal-append.queue_time_us":272}
I20240503 08:14:34.366954 28049 maintenance_manager.cc:419] P 833582ff68c94269be364ebd187a75db: Scheduling UndoDeltaBlockGCOp(0ceedfb43869465bb63931c39c28583a): 4838 bytes on disk
I20240503 08:14:34.367689 27940 maintenance_manager.cc:643] P 833582ff68c94269be364ebd187a75db: UndoDeltaBlockGCOp(0ceedfb43869465bb63931c39c28583a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
W20240503 08:14:34.374971 30960 consensus_peers.cc:485] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:34.376547 27937 consensus_peers.cc:485] T c8ca21ffe16e41ab80f5693dc4dbb682 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:34.386206 28327 consensus_peers.cc:485] T 8bcbc022caa94bb08a2db4de9b060557 P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:14:34.396384 28179 maintenance_manager.cc:419] P 1b411c6bb25549368041773d7b710d48: Scheduling FlushMRSOp(0ceedfb43869465bb63931c39c28583a): perf score=0.033395
I20240503 08:14:34.421859 12356 raft_consensus.cc:979] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2: : Instructing follower ce1496a47d084365ad786cd09e472a2c to start an election
I20240503 08:14:34.421975 12356 raft_consensus.cc:1067] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Signalling peer ce1496a47d084365ad786cd09e472a2c to start an election
I20240503 08:14:34.422848 30894 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e3a6b72060804beb81a177988d287f63"
dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
from {username='slave'} at 127.20.230.68:45821
I20240503 08:14:34.423029 30894 raft_consensus.cc:482] T e3a6b72060804beb81a177988d287f63 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:14:34.423110 30894 raft_consensus.cc:3037] T e3a6b72060804beb81a177988d287f63 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:14:34.430435 30960 consensus_peers.cc:485] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:34.430905 27937 consensus_peers.cc:485] T e0b904461c1c44ed93e1a227d93498eb P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:14:34.435425 30894 raft_consensus.cc:504] T e3a6b72060804beb81a177988d287f63 P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:14:34.436398 30894 leader_election.cc:288] T e3a6b72060804beb81a177988d287f63 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 election: Requested vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:14:34.436683 30759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3a6b72060804beb81a177988d287f63" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:14:34.436883 30759 raft_consensus.cc:3037] T e3a6b72060804beb81a177988d287f63 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:34.436864 28389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3a6b72060804beb81a177988d287f63" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:14:34.437024 28389 raft_consensus.cc:3032] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:14:34.437083 28389 raft_consensus.cc:729] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Running, Role: LEADER
I20240503 08:14:34.437304 28389 consensus_queue.cc:260] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:14:34.437687 28389 raft_consensus.cc:3037] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:34.439357 30759 raft_consensus.cc:2454] T e3a6b72060804beb81a177988d287f63 P 6d1d7de19459435fbde1999e3dfd226b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 2.
I20240503 08:14:34.439697 28000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3a6b72060804beb81a177988d287f63" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:14:34.439844 28000 raft_consensus.cc:3037] T e3a6b72060804beb81a177988d287f63 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:34.440135 28389 raft_consensus.cc:2454] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 2.
I20240503 08:14:34.442857 30829 leader_election.cc:302] T e3a6b72060804beb81a177988d287f63 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, ce1496a47d084365ad786cd09e472a2c, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:14:34.443145 12211 raft_consensus.cc:2781] T e3a6b72060804beb81a177988d287f63 P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:14:34.443270 12211 raft_consensus.cc:686] T e3a6b72060804beb81a177988d287f63 P ce1496a47d084365ad786cd09e472a2c [term 2 LEADER]: Becoming Leader. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Running, Role: LEADER
I20240503 08:14:34.443504 12211 consensus_queue.cc:237] T e3a6b72060804beb81a177988d287f63 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
W20240503 08:14:34.445235 28327 consensus_peers.cc:485] T 45e7a03f838d4beaae2080d523319d58 P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:14:34.446383 28000 raft_consensus.cc:2454] T e3a6b72060804beb81a177988d287f63 P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 2.
W20240503 08:14:34.458175 28067 consensus_peers.cc:485] T fa7e18041c294a239eb7b8f735b66c53 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:14:34.462752 27853 catalog_manager.cc:5484] T e3a6b72060804beb81a177988d287f63 P ce1496a47d084365ad786cd09e472a2c reported cstate change: term changed from 1 to 2, leader changed from d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) to ce1496a47d084365ad786cd09e472a2c (127.20.230.70). New cstate: current_term: 2 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
W20240503 08:14:34.483254 30828 consensus_peers.cc:485] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:34.513427 27937 consensus_peers.cc:485] T 7c8c7659181040cabf20ae116fae78c8 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:34.534577 31095 consensus_peers.cc:485] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:14:34.551946 28070 maintenance_manager.cc:643] P 1b411c6bb25549368041773d7b710d48: FlushMRSOp(0ceedfb43869465bb63931c39c28583a) complete. Timing: real 0.155s user 0.070s sys 0.024s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":452,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":3067,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":891,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":3671936,"thread_start_us":56207,"threads_started":2,"wal-append.queue_time_us":55177}
I20240503 08:14:34.555013 28179 maintenance_manager.cc:419] P 1b411c6bb25549368041773d7b710d48: Scheduling UndoDeltaBlockGCOp(0ceedfb43869465bb63931c39c28583a): 4838 bytes on disk
I20240503 08:14:34.555601 28070 maintenance_manager.cc:643] P 1b411c6bb25549368041773d7b710d48: UndoDeltaBlockGCOp(0ceedfb43869465bb63931c39c28583a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
W20240503 08:14:34.562512 28067 consensus_peers.cc:485] T d94f5824a71f454f8a1bfa13dd032684 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:34.568856 28327 consensus_peers.cc:485] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:34.577430 27937 consensus_peers.cc:485] T 3d0498256b5f4b8fb1681759429f10ed P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:34.613409 30960 consensus_peers.cc:485] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:34.618135 28327 consensus_peers.cc:485] T 8425cfe5c6134a60852e12f9af55227b P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:34.625697 27937 consensus_peers.cc:485] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:34.634835 30828 consensus_peers.cc:485] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:34.685904 27937 consensus_peers.cc:485] T 0ceedfb43869465bb63931c39c28583a P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:34.712378 28067 consensus_peers.cc:485] T bab5c25c31a64b578489b1006f631cc0 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:34.734252 28327 consensus_peers.cc:485] T ba1fc93781714efb8eb7ec67c2fceb8a P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:14:34.764942 28111 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240503 08:14:34.766752 30828 consensus_peers.cc:462] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:34.805016 27983 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:34.815491 30742 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:34.882071 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:14:34.905537 31141 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 08:14:34.927191 28373 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 08:14:34.931317 30874 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:34.979781 30759 raft_consensus.cc:1261] T e3a6b72060804beb81a177988d287f63 P 6d1d7de19459435fbde1999e3dfd226b [term 2 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:14:34.980405 12292 consensus_queue.cc:1035] T e3a6b72060804beb81a177988d287f63 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:35.041098 28389 raft_consensus.cc:1261] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:14:35.041821 12211 consensus_queue.cc:1035] T e3a6b72060804beb81a177988d287f63 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:35.052320 27999 raft_consensus.cc:1261] T e3a6b72060804beb81a177988d287f63 P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:14:35.055346 11989 consensus_queue.cc:1035] T e3a6b72060804beb81a177988d287f63 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:14:35.222551 28133 consensus_queue.cc:237] T d94f5824a71f454f8a1bfa13dd032684 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:35.224912 28389 raft_consensus.cc:1261] T d94f5824a71f454f8a1bfa13dd032684 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:14:35.225359 28067 consensus_peers.cc:485] T d94f5824a71f454f8a1bfa13dd032684 P 1b411c6bb25549368041773d7b710d48 -> Peer ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): Couldn't send request to peer ced21ca983b749d0b4d4f6a0e2426bf8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d94f5824a71f454f8a1bfa13dd032684. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:35.225986 12314 consensus_queue.cc:1035] T d94f5824a71f454f8a1bfa13dd032684 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:14:35.227265 28067 consensus_peers.cc:485] T d94f5824a71f454f8a1bfa13dd032684 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:35.371573 30828 consensus_peers.cc:462] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:35.608021 12630 ts_tablet_manager.cc:902] T d94f5824a71f454f8a1bfa13dd032684 P ced21ca983b749d0b4d4f6a0e2426bf8: Initiating tablet copy from peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:14:35.608992 12630 tablet_copy_client.cc:308] T d94f5824a71f454f8a1bfa13dd032684 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.66:37143
I20240503 08:14:35.609508 28153 tablet_copy_service.cc:140] P 1b411c6bb25549368041773d7b710d48: Received BeginTabletCopySession request for tablet d94f5824a71f454f8a1bfa13dd032684 from peer ced21ca983b749d0b4d4f6a0e2426bf8 ({username='slave'} at 127.20.230.71:43205)
I20240503 08:14:35.609622 28153 tablet_copy_service.cc:161] P 1b411c6bb25549368041773d7b710d48: Beginning new tablet copy session on tablet d94f5824a71f454f8a1bfa13dd032684 from peer ced21ca983b749d0b4d4f6a0e2426bf8 at {username='slave'} at 127.20.230.71:43205: session id = ced21ca983b749d0b4d4f6a0e2426bf8-d94f5824a71f454f8a1bfa13dd032684
I20240503 08:14:35.612171 28153 tablet_copy_source_session.cc:205] T d94f5824a71f454f8a1bfa13dd032684 P 1b411c6bb25549368041773d7b710d48: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:14:35.613894 12630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d94f5824a71f454f8a1bfa13dd032684. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:35.619115 12630 tablet_copy_client.cc:783] T d94f5824a71f454f8a1bfa13dd032684 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet copy: Starting download of 5 data blocks...
I20240503 08:14:35.631139 12630 tablet_copy_client.cc:647] T d94f5824a71f454f8a1bfa13dd032684 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:35.634091 12630 tablet_copy_client.cc:523] T d94f5824a71f454f8a1bfa13dd032684 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:35.637203 12630 tablet_bootstrap.cc:492] T d94f5824a71f454f8a1bfa13dd032684 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
I20240503 08:14:35.656462 12630 tablet_bootstrap.cc:492] T d94f5824a71f454f8a1bfa13dd032684 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:14:35.656878 12630 tablet_bootstrap.cc:492] T d94f5824a71f454f8a1bfa13dd032684 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap complete.
I20240503 08:14:35.657109 12630 ts_tablet_manager.cc:1367] T d94f5824a71f454f8a1bfa13dd032684 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.020s user 0.021s sys 0.000s
I20240503 08:14:35.657795 12630 raft_consensus.cc:348] T d94f5824a71f454f8a1bfa13dd032684 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:14:35.658277 12630 raft_consensus.cc:729] T d94f5824a71f454f8a1bfa13dd032684 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:14:35.658443 12630 consensus_queue.cc:260] T d94f5824a71f454f8a1bfa13dd032684 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:35.659022 12630 ts_tablet_manager.cc:1398] T d94f5824a71f454f8a1bfa13dd032684 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:14:35.660892 28153 tablet_copy_service.cc:342] P 1b411c6bb25549368041773d7b710d48: Request end of tablet copy session ced21ca983b749d0b4d4f6a0e2426bf8-d94f5824a71f454f8a1bfa13dd032684 received from {username='slave'} at 127.20.230.71:43205
I20240503 08:14:35.661015 28153 tablet_copy_service.cc:433] P 1b411c6bb25549368041773d7b710d48: ending tablet copy session ced21ca983b749d0b4d4f6a0e2426bf8-d94f5824a71f454f8a1bfa13dd032684 on tablet d94f5824a71f454f8a1bfa13dd032684 with peer ced21ca983b749d0b4d4f6a0e2426bf8
I20240503 08:14:35.668255 31074 maintenance_manager.cc:419] P ced21ca983b749d0b4d4f6a0e2426bf8: Scheduling UndoDeltaBlockGCOp(d94f5824a71f454f8a1bfa13dd032684): 4923 bytes on disk
I20240503 08:14:35.669500 30963 maintenance_manager.cc:643] P ced21ca983b749d0b4d4f6a0e2426bf8: UndoDeltaBlockGCOp(d94f5824a71f454f8a1bfa13dd032684) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
W20240503 08:14:35.742233 27937 consensus_peers.cc:485] T ca38671a760d4534ab903579e0487772 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:35.753015 30960 consensus_peers.cc:485] T 41aa0e288ca4469ba34a07ff159ace91 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:35.783785 27937 consensus_peers.cc:485] T d9098f3d36b24d578a3f8d7d3ccad052 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:35.789477 30828 consensus_peers.cc:462] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:35.821061 28327 consensus_peers.cc:485] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:35.875648 30828 consensus_peers.cc:485] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:35.940109 28067 consensus_peers.cc:485] T 8450e62352a74d2ea1471166d4cb1132 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:35.984958 30695 consensus_peers.cc:485] T 2222ff7fb5184d6c86010e1475d6c1f7 P 6d1d7de19459435fbde1999e3dfd226b -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:35.987092 28327 consensus_peers.cc:485] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:35.992811 28067 consensus_peers.cc:485] T 1cac39d5831d4b6887250f7a9d63dcd5 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:36.016767 28327 consensus_peers.cc:485] T cdad6ef53b6f413ebfda67f9b42c9c23 P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:36.050097 28067 consensus_peers.cc:485] T 8fc87f7f3c724cd4a270583ddebd87aa P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:36.069329 27937 consensus_peers.cc:485] T 46d27fa12990405cae5f94976ac7a014 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:36.069641 28327 consensus_peers.cc:485] T f46393257ce9471891698c36a07546ad P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:36.139608 28067 consensus_peers.cc:485] T 362590d3d0834d0b8a672ab552f8c209 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:36.175979 28067 consensus_peers.cc:485] T de5dfccb0359462dbb7da776293fa679 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:36.187541 28067 consensus_peers.cc:485] T 17c6e3293aad4591a9c1395ebffff2e6 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:14:36.213514 31027 raft_consensus.cc:1203] T d94f5824a71f454f8a1bfa13dd032684 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:14:36.214461 11539 raft_consensus.cc:2932] T d94f5824a71f454f8a1bfa13dd032684 P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:14:36.223765 28391 raft_consensus.cc:2932] T d94f5824a71f454f8a1bfa13dd032684 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:14:36.224926 31027 raft_consensus.cc:2932] T d94f5824a71f454f8a1bfa13dd032684 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:14:36.239773 27858 catalog_manager.cc:5484] T d94f5824a71f454f8a1bfa13dd032684 P ced21ca983b749d0b4d4f6a0e2426bf8 reported cstate change: config changed from index -1 to 4, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) added. New cstate: current_term: 1 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
W20240503 08:14:36.247678 28067 consensus_peers.cc:485] T f7a8c95ac81942dabef98a07caa84082 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:36.268461 31095 consensus_peers.cc:485] T 6a96965691644db8b9461d146536a026 P b00f244c69df401997ea99d492218a98 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:14:36.310532 28130 consensus_queue.cc:237] T 099e37089b2e42b4839e681f8f37365d P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:14:36.312893 27999 raft_consensus.cc:1261] T 099e37089b2e42b4839e681f8f37365d P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:14:36.313174 28068 consensus_peers.cc:485] T 099e37089b2e42b4839e681f8f37365d P 1b411c6bb25549368041773d7b710d48 -> Peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): Couldn't send request to peer 6d1d7de19459435fbde1999e3dfd226b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 099e37089b2e42b4839e681f8f37365d. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:36.313757 11539 consensus_queue.cc:1035] T 099e37089b2e42b4839e681f8f37365d P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:14:36.314806 28067 consensus_peers.cc:485] T 099e37089b2e42b4839e681f8f37365d P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:36.324440 27937 consensus_peers.cc:485] T 8cc6ae30aa60495d8a4364a11d5bef79 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:36.381685 28067 consensus_peers.cc:485] T 978bf5783b0e47e3bfb30441adc600ae P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:36.401757 30828 consensus_peers.cc:462] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:36.465586 31095 consensus_peers.cc:485] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:36.540174 30828 consensus_peers.cc:485] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:36.575088 27937 consensus_peers.cc:485] T 08fb64977309479b831635344969c662 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:36.592089 30828 consensus_peers.cc:485] T 045b3ce4b1d44f94a802b41a12e2704d P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:36.595017 30960 consensus_peers.cc:485] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:36.639520 30828 consensus_peers.cc:485] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:36.641616 28327 consensus_peers.cc:485] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:36.653055 31095 consensus_peers.cc:485] T 2c595805729d48fd974d761ed8a60158 P b00f244c69df401997ea99d492218a98 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:36.678050 27937 consensus_peers.cc:485] T c8ca21ffe16e41ab80f5693dc4dbb682 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:36.697906 28327 consensus_peers.cc:485] T eeee6a265fc94e91bf9daa8301662e5f P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:36.722509 30695 consensus_peers.cc:485] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:36.725134 27937 consensus_peers.cc:485] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:36.748183 28327 consensus_peers.cc:485] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:36.757700 28327 consensus_peers.cc:485] T bdfe68a127f043f08f04749843a38375 P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:36.773337 27937 consensus_peers.cc:485] T e0b904461c1c44ed93e1a227d93498eb P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:36.784725 28067 consensus_peers.cc:485] T df90a0d9d338416b81e253fa23542cca P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:36.814934 28067 consensus_peers.cc:485] T 2dce5b2aaea045ccb7424cbbd44576c2 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:14:36.846164 12667 ts_tablet_manager.cc:902] T 099e37089b2e42b4839e681f8f37365d P 6d1d7de19459435fbde1999e3dfd226b: Initiating tablet copy from peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:14:36.847312 12667 tablet_copy_client.cc:308] T 099e37089b2e42b4839e681f8f37365d P 6d1d7de19459435fbde1999e3dfd226b: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.66:37143
I20240503 08:14:36.847899 28153 tablet_copy_service.cc:140] P 1b411c6bb25549368041773d7b710d48: Received BeginTabletCopySession request for tablet 099e37089b2e42b4839e681f8f37365d from peer 6d1d7de19459435fbde1999e3dfd226b ({username='slave'} at 127.20.230.69:49059)
I20240503 08:14:36.848047 28153 tablet_copy_service.cc:161] P 1b411c6bb25549368041773d7b710d48: Beginning new tablet copy session on tablet 099e37089b2e42b4839e681f8f37365d from peer 6d1d7de19459435fbde1999e3dfd226b at {username='slave'} at 127.20.230.69:49059: session id = 6d1d7de19459435fbde1999e3dfd226b-099e37089b2e42b4839e681f8f37365d
I20240503 08:14:36.850253 28153 tablet_copy_source_session.cc:205] T 099e37089b2e42b4839e681f8f37365d P 1b411c6bb25549368041773d7b710d48: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:14:36.851898 12667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 099e37089b2e42b4839e681f8f37365d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:36.857116 12667 tablet_copy_client.cc:783] T 099e37089b2e42b4839e681f8f37365d P 6d1d7de19459435fbde1999e3dfd226b: tablet copy: Starting download of 5 data blocks...
I20240503 08:14:36.864774 12667 tablet_copy_client.cc:647] T 099e37089b2e42b4839e681f8f37365d P 6d1d7de19459435fbde1999e3dfd226b: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:36.867082 12667 tablet_copy_client.cc:523] T 099e37089b2e42b4839e681f8f37365d P 6d1d7de19459435fbde1999e3dfd226b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:36.870247 12667 tablet_bootstrap.cc:492] T 099e37089b2e42b4839e681f8f37365d P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap starting.
I20240503 08:14:36.883661 12667 tablet_bootstrap.cc:492] T 099e37089b2e42b4839e681f8f37365d P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:14:36.884016 12667 tablet_bootstrap.cc:492] T 099e37089b2e42b4839e681f8f37365d P 6d1d7de19459435fbde1999e3dfd226b: Bootstrap complete.
I20240503 08:14:36.884233 12667 ts_tablet_manager.cc:1367] T 099e37089b2e42b4839e681f8f37365d P 6d1d7de19459435fbde1999e3dfd226b: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20240503 08:14:36.884706 12667 raft_consensus.cc:348] T 099e37089b2e42b4839e681f8f37365d P 6d1d7de19459435fbde1999e3dfd226b [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:36.885035 12667 raft_consensus.cc:729] T 099e37089b2e42b4839e681f8f37365d P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Initialized, Role: FOLLOWER
I20240503 08:14:36.885165 12667 consensus_queue.cc:260] T 099e37089b2e42b4839e681f8f37365d P 6d1d7de19459435fbde1999e3dfd226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:14:36.885563 12667 ts_tablet_manager.cc:1398] T 099e37089b2e42b4839e681f8f37365d P 6d1d7de19459435fbde1999e3dfd226b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240503 08:14:36.888829 31095 consensus_peers.cc:485] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:14:36.886519 28153 tablet_copy_service.cc:342] P 1b411c6bb25549368041773d7b710d48: Request end of tablet copy session 6d1d7de19459435fbde1999e3dfd226b-099e37089b2e42b4839e681f8f37365d received from {username='slave'} at 127.20.230.69:49059
I20240503 08:14:36.889914 28153 tablet_copy_service.cc:433] P 1b411c6bb25549368041773d7b710d48: ending tablet copy session 6d1d7de19459435fbde1999e3dfd226b-099e37089b2e42b4839e681f8f37365d on tablet 099e37089b2e42b4839e681f8f37365d with peer 6d1d7de19459435fbde1999e3dfd226b
W20240503 08:14:36.892680 31095 consensus_peers.cc:485] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:36.894595 30828 consensus_peers.cc:462] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:36.913482 27937 consensus_peers.cc:485] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:36.923760 27937 consensus_peers.cc:485] T 0ceedfb43869465bb63931c39c28583a P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:36.954317 27937 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240503 08:14:36.963333 28327 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240503 08:14:36.966125 30695 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111) [suppressed 23 similar messages]
W20240503 08:14:36.970069 30828 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111) [suppressed 84 similar messages]
W20240503 08:14:36.970774 30828 consensus_peers.cc:485] T 0f25688be4e74d31aaab85b7a4329a34 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:36.977319 28067 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111) [suppressed 150 similar messages]
W20240503 08:14:36.978427 28067 consensus_peers.cc:485] T fa7e18041c294a239eb7b8f735b66c53 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:36.981429 31095 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111) [suppressed 56 similar messages]
W20240503 08:14:36.981560 30960 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111) [suppressed 44 similar messages]
W20240503 08:14:36.982362 30960 consensus_peers.cc:485] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:14:36.988291 30809 maintenance_manager.cc:419] P 6d1d7de19459435fbde1999e3dfd226b: Scheduling UndoDeltaBlockGCOp(099e37089b2e42b4839e681f8f37365d): 4838 bytes on disk
I20240503 08:14:36.989428 30699 maintenance_manager.cc:643] P 6d1d7de19459435fbde1999e3dfd226b: UndoDeltaBlockGCOp(099e37089b2e42b4839e681f8f37365d) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
W20240503 08:14:37.031069 27937 consensus_peers.cc:485] T 7c8c7659181040cabf20ae116fae78c8 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:37.043138 28067 consensus_peers.cc:485] T 658865ee3f8d424ca5cd35e12e6d7c77 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:37.043825 30828 consensus_peers.cc:485] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:37.053258 28327 consensus_peers.cc:485] T 8425cfe5c6134a60852e12f9af55227b P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:37.077301 28327 consensus_peers.cc:485] T 45e7a03f838d4beaae2080d523319d58 P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:37.107978 28327 consensus_peers.cc:485] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:37.113158 28327 consensus_peers.cc:485] T 8bcbc022caa94bb08a2db4de9b060557 P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:37.126596 27937 consensus_peers.cc:485] T 3d0498256b5f4b8fb1681759429f10ed P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:37.213162 30828 consensus_peers.cc:485] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:37.234659 30960 consensus_peers.cc:485] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:14:37.267019 30761 raft_consensus.cc:1203] T 099e37089b2e42b4839e681f8f37365d P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:14:37.268103 12673 raft_consensus.cc:2932] T 099e37089b2e42b4839e681f8f37365d P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:14:37.268662 30761 raft_consensus.cc:2932] T 099e37089b2e42b4839e681f8f37365d P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:14:37.276744 27857 catalog_manager.cc:5484] T 099e37089b2e42b4839e681f8f37365d P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: config changed from index -1 to 4, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) added. New cstate: current_term: 1 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:14:37.283330 28000 raft_consensus.cc:2932] T 099e37089b2e42b4839e681f8f37365d P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
I20240503 08:14:37.331897 28179 maintenance_manager.cc:419] P 1b411c6bb25549368041773d7b710d48: Scheduling FlushMRSOp(978bf5783b0e47e3bfb30441adc600ae): perf score=0.033364
I20240503 08:14:37.337926 28391 consensus_queue.cc:237] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:14:37.340564 28130 raft_consensus.cc:1261] T 33f0f4bc5722485a8f978a02367ca80e P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 08:14:37.341099 12356 consensus_queue.cc:1035] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 08:14:37.342159 28326 consensus_peers.cc:485] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2 -> Peer ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): Couldn't send request to peer ce1496a47d084365ad786cd09e472a2c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 33f0f4bc5722485a8f978a02367ca80e. This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:37.342916 28327 consensus_peers.cc:485] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:37.361018 28439 maintenance_manager.cc:419] P d09a11a03989421b9f28019dc1c14cc2: Scheduling FlushMRSOp(bdfe68a127f043f08f04749843a38375): perf score=0.033381
I20240503 08:14:37.384151 28070 maintenance_manager.cc:643] P 1b411c6bb25549368041773d7b710d48: FlushMRSOp(978bf5783b0e47e3bfb30441adc600ae) complete. Timing: real 0.052s user 0.033s sys 0.007s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":342,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":3153,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":815,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":466,"threads_started":2,"wal-append.queue_time_us":348}
I20240503 08:14:37.386946 28179 maintenance_manager.cc:419] P 1b411c6bb25549368041773d7b710d48: Scheduling UndoDeltaBlockGCOp(978bf5783b0e47e3bfb30441adc600ae): 4838 bytes on disk
I20240503 08:14:37.387591 28070 maintenance_manager.cc:643] P 1b411c6bb25549368041773d7b710d48: UndoDeltaBlockGCOp(978bf5783b0e47e3bfb30441adc600ae) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
I20240503 08:14:37.388800 28330 maintenance_manager.cc:643] P d09a11a03989421b9f28019dc1c14cc2: FlushMRSOp(bdfe68a127f043f08f04749843a38375) complete. Timing: real 0.028s user 0.024s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":369,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":1376,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":624,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":452,"threads_started":2,"wal-append.queue_time_us":310}
I20240503 08:14:37.390995 28439 maintenance_manager.cc:419] P d09a11a03989421b9f28019dc1c14cc2: Scheduling UndoDeltaBlockGCOp(bdfe68a127f043f08f04749843a38375): 4583 bytes on disk
I20240503 08:14:37.392120 28330 maintenance_manager.cc:643] P d09a11a03989421b9f28019dc1c14cc2: UndoDeltaBlockGCOp(bdfe68a127f043f08f04749843a38375) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20240503 08:14:37.397387 28049 maintenance_manager.cc:419] P 833582ff68c94269be364ebd187a75db: Scheduling FlushMRSOp(08fb64977309479b831635344969c662): perf score=0.033393
W20240503 08:14:37.410785 30828 consensus_peers.cc:462] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:37.419369 28067 consensus_peers.cc:485] T bab5c25c31a64b578489b1006f631cc0 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:14:37.423496 27940 maintenance_manager.cc:643] P 833582ff68c94269be364ebd187a75db: FlushMRSOp(08fb64977309479b831635344969c662) complete. Timing: real 0.026s user 0.020s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":305,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":1269,"drs_written":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4,"lbm_write_time_us":533,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":468,"threads_started":2,"wal-append.queue_time_us":338}
I20240503 08:14:37.425926 28049 maintenance_manager.cc:419] P 833582ff68c94269be364ebd187a75db: Scheduling FlushMRSOp(bdfe68a127f043f08f04749843a38375): perf score=0.033392
I20240503 08:14:37.459252 27940 maintenance_manager.cc:643] P 833582ff68c94269be364ebd187a75db: FlushMRSOp(bdfe68a127f043f08f04749843a38375) complete. Timing: real 0.033s user 0.026s sys 0.005s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":57,"dirs.run_cpu_time_us":153,"dirs.run_wall_time_us":1512,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":692,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":267,"threads_started":1,"wal-append.queue_time_us":354}
I20240503 08:14:37.462314 28049 maintenance_manager.cc:419] P 833582ff68c94269be364ebd187a75db: Scheduling UndoDeltaBlockGCOp(08fb64977309479b831635344969c662): 4923 bytes on disk
I20240503 08:14:37.462852 27940 maintenance_manager.cc:643] P 833582ff68c94269be364ebd187a75db: UndoDeltaBlockGCOp(08fb64977309479b831635344969c662) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
I20240503 08:14:37.464546 28049 maintenance_manager.cc:419] P 833582ff68c94269be364ebd187a75db: Scheduling UndoDeltaBlockGCOp(bdfe68a127f043f08f04749843a38375): 4583 bytes on disk
I20240503 08:14:37.464939 27940 maintenance_manager.cc:643] P 833582ff68c94269be364ebd187a75db: UndoDeltaBlockGCOp(bdfe68a127f043f08f04749843a38375) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
W20240503 08:14:37.475946 28327 consensus_peers.cc:485] T ba1fc93781714efb8eb7ec67c2fceb8a P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:37.795331 30828 consensus_peers.cc:462] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:37.800873 28067 consensus_peers.cc:485] T d94f5824a71f454f8a1bfa13dd032684 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:14:37.861726 12709 ts_tablet_manager.cc:902] T 33f0f4bc5722485a8f978a02367ca80e P ce1496a47d084365ad786cd09e472a2c: Initiating tablet copy from peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:14:37.862810 12709 tablet_copy_client.cc:308] T 33f0f4bc5722485a8f978a02367ca80e P ce1496a47d084365ad786cd09e472a2c: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.68:46211
I20240503 08:14:37.863351 28413 tablet_copy_service.cc:140] P d09a11a03989421b9f28019dc1c14cc2: Received BeginTabletCopySession request for tablet 33f0f4bc5722485a8f978a02367ca80e from peer ce1496a47d084365ad786cd09e472a2c ({username='slave'} at 127.20.230.70:51715)
I20240503 08:14:37.863487 28413 tablet_copy_service.cc:161] P d09a11a03989421b9f28019dc1c14cc2: Beginning new tablet copy session on tablet 33f0f4bc5722485a8f978a02367ca80e from peer ce1496a47d084365ad786cd09e472a2c at {username='slave'} at 127.20.230.70:51715: session id = ce1496a47d084365ad786cd09e472a2c-33f0f4bc5722485a8f978a02367ca80e
I20240503 08:14:37.866017 28413 tablet_copy_source_session.cc:205] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:14:37.867373 12709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33f0f4bc5722485a8f978a02367ca80e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:37.871651 12709 tablet_copy_client.cc:783] T 33f0f4bc5722485a8f978a02367ca80e P ce1496a47d084365ad786cd09e472a2c: tablet copy: Starting download of 5 data blocks...
I20240503 08:14:37.877447 12709 tablet_copy_client.cc:647] T 33f0f4bc5722485a8f978a02367ca80e P ce1496a47d084365ad786cd09e472a2c: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:37.879192 12709 tablet_copy_client.cc:523] T 33f0f4bc5722485a8f978a02367ca80e P ce1496a47d084365ad786cd09e472a2c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:37.881728 12709 tablet_bootstrap.cc:492] T 33f0f4bc5722485a8f978a02367ca80e P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:14:37.893369 12709 tablet_bootstrap.cc:492] T 33f0f4bc5722485a8f978a02367ca80e P ce1496a47d084365ad786cd09e472a2c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=4 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:14:37.893913 12709 tablet_bootstrap.cc:492] T 33f0f4bc5722485a8f978a02367ca80e P ce1496a47d084365ad786cd09e472a2c: Bootstrap complete.
I20240503 08:14:37.894186 12709 ts_tablet_manager.cc:1367] T 33f0f4bc5722485a8f978a02367ca80e P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.003s
I20240503 08:14:37.894868 12709 raft_consensus.cc:348] T 33f0f4bc5722485a8f978a02367ca80e P ce1496a47d084365ad786cd09e472a2c [term 2 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:14:37.895320 12709 raft_consensus.cc:729] T 33f0f4bc5722485a8f978a02367ca80e P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:14:37.895524 12709 consensus_queue.cc:260] T 33f0f4bc5722485a8f978a02367ca80e P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:14:37.896034 12709 ts_tablet_manager.cc:1398] T 33f0f4bc5722485a8f978a02367ca80e P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:14:37.897806 28413 tablet_copy_service.cc:342] P d09a11a03989421b9f28019dc1c14cc2: Request end of tablet copy session ce1496a47d084365ad786cd09e472a2c-33f0f4bc5722485a8f978a02367ca80e received from {username='slave'} at 127.20.230.70:51715
I20240503 08:14:37.897944 28413 tablet_copy_service.cc:433] P d09a11a03989421b9f28019dc1c14cc2: ending tablet copy session ce1496a47d084365ad786cd09e472a2c-33f0f4bc5722485a8f978a02367ca80e on tablet 33f0f4bc5722485a8f978a02367ca80e with peer ce1496a47d084365ad786cd09e472a2c
W20240503 08:14:37.956672 27937 consensus_peers.cc:485] T d9098f3d36b24d578a3f8d7d3ccad052 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:38.062361 30960 consensus_peers.cc:485] T 41aa0e288ca4469ba34a07ff159ace91 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:14:38.098474 30941 maintenance_manager.cc:419] P ce1496a47d084365ad786cd09e472a2c: Scheduling UndoDeltaBlockGCOp(33f0f4bc5722485a8f978a02367ca80e): 4838 bytes on disk
I20240503 08:14:38.099020 30831 maintenance_manager.cc:643] P ce1496a47d084365ad786cd09e472a2c: UndoDeltaBlockGCOp(33f0f4bc5722485a8f978a02367ca80e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
W20240503 08:14:38.213903 30828 consensus_peers.cc:485] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:38.298471 28327 consensus_peers.cc:485] T cdad6ef53b6f413ebfda67f9b42c9c23 P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:38.371989 30828 consensus_peers.cc:462] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:38.380834 27937 consensus_peers.cc:485] T ca38671a760d4534ab903579e0487772 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:14:38.381863 30894 raft_consensus.cc:1203] T 33f0f4bc5722485a8f978a02367ca80e P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.4->[2.5-2.5] Dedup: 2.5->[]
I20240503 08:14:38.382714 12356 raft_consensus.cc:2932] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index -1 to 5, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:14:38.383298 30894 raft_consensus.cc:2932] T 33f0f4bc5722485a8f978a02367ca80e P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index -1 to 5, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:14:38.388623 12356 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:54776 (request call id 0) took 1051 ms. Trace:
I20240503 08:14:38.390581 27861 catalog_manager.cc:5484] T 33f0f4bc5722485a8f978a02367ca80e P ce1496a47d084365ad786cd09e472a2c reported cstate change: config changed from index -1 to 5, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New cstate: current_term: 2 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:14:38.388839 12356 rpcz_store.cc:276] 0503 08:14:37.337184 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 08:14:37.337230 (+ 46us) service_pool.cc:225] Handling call
0503 08:14:37.339168 (+ 1938us) log.cc:842] Serialized 498 byte log entry
0503 08:14:38.388287 (+1049119us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":122,"mutex_wait_us":831,"raft.queue_time_us":343,"raft.run_cpu_time_us":596,"raft.run_wall_time_us":595,"spinlock_wait_cycles":174208,"thread_start_us":11074,"threads_started":2,"wal-append.queue_time_us":305,"wal-append.run_cpu_time_us":447,"wal-append.run_wall_time_us":1000510}
I20240503 08:14:38.393355 28133 raft_consensus.cc:2932] T 33f0f4bc5722485a8f978a02367ca80e P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index -1 to 5, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
W20240503 08:14:38.421357 30695 consensus_peers.cc:485] T 2222ff7fb5184d6c86010e1475d6c1f7 P 6d1d7de19459435fbde1999e3dfd226b -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:38.422195 27937 consensus_peers.cc:485] T 8cc6ae30aa60495d8a4364a11d5bef79 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:38.432581 28327 consensus_peers.cc:485] T f46393257ce9471891698c36a07546ad P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:14:38.471242 28000 consensus_queue.cc:237] T f1b2f3b8111649a7b5725647b8251698 P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:38.473914 31160 raft_consensus.cc:1261] T f1b2f3b8111649a7b5725647b8251698 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:38.473914 31025 raft_consensus.cc:1261] T f1b2f3b8111649a7b5725647b8251698 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:38.474602 12665 consensus_queue.cc:1035] T f1b2f3b8111649a7b5725647b8251698 P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:38.475158 12665 consensus_queue.cc:1035] T f1b2f3b8111649a7b5725647b8251698 P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 08:14:38.476447 27937 consensus_peers.cc:485] T f1b2f3b8111649a7b5725647b8251698 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:38.489404 12665 raft_consensus.cc:2932] T f1b2f3b8111649a7b5725647b8251698 P 833582ff68c94269be364ebd187a75db [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:38.490097 31160 raft_consensus.cc:2932] T f1b2f3b8111649a7b5725647b8251698 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
I20240503 08:14:38.494369 31025 raft_consensus.cc:2932] T f1b2f3b8111649a7b5725647b8251698 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
W20240503 08:14:38.495926 28327 consensus_peers.cc:485] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:38.496165 28067 consensus_peers.cc:485] T f7a8c95ac81942dabef98a07caa84082 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:14:38.498102 27858 catalog_manager.cc:5484] T f1b2f3b8111649a7b5725647b8251698 P b00f244c69df401997ea99d492218a98 reported cstate change: config changed from index -1 to 2, VOTER aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67) added. New cstate: current_term: 1 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } }
W20240503 08:14:38.506112 28067 consensus_peers.cc:485] T 1cac39d5831d4b6887250f7a9d63dcd5 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:38.515195 28067 consensus_peers.cc:485] T 8450e62352a74d2ea1471166d4cb1132 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:38.525192 28067 consensus_peers.cc:485] T 362590d3d0834d0b8a672ab552f8c209 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:38.541692 28067 consensus_peers.cc:485] T 8fc87f7f3c724cd4a270583ddebd87aa P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:14:38.568076 28133 consensus_queue.cc:237] T bab5c25c31a64b578489b1006f631cc0 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
W20240503 08:14:38.572007 28067 consensus_peers.cc:485] T bab5c25c31a64b578489b1006f631cc0 P 1b411c6bb25549368041773d7b710d48 -> Peer ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): Couldn't send request to peer ced21ca983b749d0b4d4f6a0e2426bf8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bab5c25c31a64b578489b1006f631cc0. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:38.572690 28391 raft_consensus.cc:1261] T bab5c25c31a64b578489b1006f631cc0 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer 1b411c6bb25549368041773d7b710d48: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:38.573576 12681 consensus_queue.cc:1035] T bab5c25c31a64b578489b1006f631cc0 P 1b411c6bb25549368041773d7b710d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:14:38.574394 28067 consensus_peers.cc:485] T bab5c25c31a64b578489b1006f631cc0 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:38.578783 27937 consensus_peers.cc:485] T 46d27fa12990405cae5f94976ac7a014 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:38.692847 28067 consensus_peers.cc:485] T 17c6e3293aad4591a9c1395ebffff2e6 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:38.725337 31095 consensus_peers.cc:485] T 6a96965691644db8b9461d146536a026 P b00f244c69df401997ea99d492218a98 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:38.801658 27937 consensus_peers.cc:485] T 08fb64977309479b831635344969c662 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:38.842875 28067 consensus_peers.cc:485] T de5dfccb0359462dbb7da776293fa679 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:38.893366 28327 consensus_peers.cc:485] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:14:38.895768 30828 consensus_peers.cc:485] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:14:38.942513 28067 consensus_peers.cc:485] T 099e37089b2e42b4839e681f8f37365d P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:38.944696 28067 consensus_peers.cc:485] T 978bf5783b0e47e3bfb30441adc600ae P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:38.950129 30828 consensus_peers.cc:462] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:38.959546 30960 consensus_peers.cc:485] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 08:14:38.983246 12771 ts_tablet_manager.cc:902] T bab5c25c31a64b578489b1006f631cc0 P ced21ca983b749d0b4d4f6a0e2426bf8: Initiating tablet copy from peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:14:38.984431 12771 tablet_copy_client.cc:308] T bab5c25c31a64b578489b1006f631cc0 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.66:37143
I20240503 08:14:38.985085 28153 tablet_copy_service.cc:140] P 1b411c6bb25549368041773d7b710d48: Received BeginTabletCopySession request for tablet bab5c25c31a64b578489b1006f631cc0 from peer ced21ca983b749d0b4d4f6a0e2426bf8 ({username='slave'} at 127.20.230.71:43205)
I20240503 08:14:38.985249 28153 tablet_copy_service.cc:161] P 1b411c6bb25549368041773d7b710d48: Beginning new tablet copy session on tablet bab5c25c31a64b578489b1006f631cc0 from peer ced21ca983b749d0b4d4f6a0e2426bf8 at {username='slave'} at 127.20.230.71:43205: session id = ced21ca983b749d0b4d4f6a0e2426bf8-bab5c25c31a64b578489b1006f631cc0
I20240503 08:14:38.988406 28153 tablet_copy_source_session.cc:205] T bab5c25c31a64b578489b1006f631cc0 P 1b411c6bb25549368041773d7b710d48: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:14:38.989850 12771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab5c25c31a64b578489b1006f631cc0. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 08:14:38.992333 30828 consensus_peers.cc:485] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 08:14:38.996212 12771 tablet_copy_client.cc:783] T bab5c25c31a64b578489b1006f631cc0 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet copy: Starting download of 5 data blocks...
I20240503 08:14:39.002727 12771 tablet_copy_client.cc:647] T bab5c25c31a64b578489b1006f631cc0 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:39.005714 12771 tablet_copy_client.cc:523] T bab5c25c31a64b578489b1006f631cc0 P ced21ca983b749d0b4d4f6a0e2426bf8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:39.009330 12771 tablet_bootstrap.cc:492] T bab5c25c31a64b578489b1006f631cc0 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap starting.
W20240503 08:14:39.020925 31095 consensus_peers.cc:485] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 08:14:39.039363 12771 tablet_bootstrap.cc:492] T bab5c25c31a64b578489b1006f631cc0 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:14:39.039724 12771 tablet_bootstrap.cc:492] T bab5c25c31a64b578489b1006f631cc0 P ced21ca983b749d0b4d4f6a0e2426bf8: Bootstrap complete.
I20240503 08:14:39.039949 12771 ts_tablet_manager.cc:1367] T bab5c25c31a64b578489b1006f631cc0 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent bootstrapping tablet: real 0.031s user 0.012s sys 0.012s
I20240503 08:14:39.040573 12771 raft_consensus.cc:348] T bab5c25c31a64b578489b1006f631cc0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:14:39.041013 12771 raft_consensus.cc:729] T bab5c25c31a64b578489b1006f631cc0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Initialized, Role: FOLLOWER
I20240503 08:14:39.041215 12771 consensus_queue.cc:260] T bab5c25c31a64b578489b1006f631cc0 P ced21ca983b749d0b4d4f6a0e2426bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:39.041751 12771 ts_tablet_manager.cc:1398] T bab5c25c31a64b578489b1006f631cc0 P ced21ca983b749d0b4d4f6a0e2426bf8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:14:39.042565 28153 tablet_copy_service.cc:342] P 1b411c6bb25549368041773d7b710d48: Request end of tablet copy session ced21ca983b749d0b4d4f6a0e2426bf8-bab5c25c31a64b578489b1006f631cc0 received from {username='slave'} at 127.20.230.71:43205
I20240503 08:14:39.042680 28153 tablet_copy_service.cc:433] P 1b411c6bb25549368041773d7b710d48: ending tablet copy session ced21ca983b749d0b4d4f6a0e2426bf8-bab5c25c31a64b578489b1006f631cc0 on tablet bab5c25c31a64b578489b1006f631cc0 with peer ced21ca983b749d0b4d4f6a0e2426bf8
W20240503 08:14:39.117012 30828 consensus_peers.cc:485] T 045b3ce4b1d44f94a802b41a12e2704d P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:14:39.139168 31095 consensus_peers.cc:485] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:14:39.144011 28067 consensus_peers.cc:485] T df90a0d9d338416b81e253fa23542cca P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:14:39.145879 30828 consensus_peers.cc:485] T 0f25688be4e74d31aaab85b7a4329a34 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:14:39.157646 27937 consensus_peers.cc:485] T c8ca21ffe16e41ab80f5693dc4dbb682 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 08:14:39.199983 31074 maintenance_manager.cc:419] P ced21ca983b749d0b4d4f6a0e2426bf8: Scheduling UndoDeltaBlockGCOp(bab5c25c31a64b578489b1006f631cc0): 4923 bytes on disk
I20240503 08:14:39.200531 30963 maintenance_manager.cc:643] P ced21ca983b749d0b4d4f6a0e2426bf8: UndoDeltaBlockGCOp(bab5c25c31a64b578489b1006f631cc0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
W20240503 08:14:39.244844 27937 consensus_peers.cc:485] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:14:39.255043 28327 consensus_peers.cc:485] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:14:39.263752 28327 consensus_peers.cc:485] T bdfe68a127f043f08f04749843a38375 P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:14:39.303093 31095 consensus_peers.cc:485] T 2c595805729d48fd974d761ed8a60158 P b00f244c69df401997ea99d492218a98 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:14:39.314702 30695 consensus_peers.cc:485] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:14:39.321609 27937 consensus_peers.cc:485] T 3d0498256b5f4b8fb1681759429f10ed P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:14:39.364775 30828 consensus_peers.cc:485] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:14:39.369329 28067 consensus_peers.cc:485] T fa7e18041c294a239eb7b8f735b66c53 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:14:39.395145 28327 consensus_peers.cc:485] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:14:39.411295 27937 consensus_peers.cc:485] T e0b904461c1c44ed93e1a227d93498eb P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:14:39.426373 30828 consensus_peers.cc:462] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:39.462579 27937 consensus_peers.cc:485] T 7c8c7659181040cabf20ae116fae78c8 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:14:39.489267 30960 consensus_peers.cc:485] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:14:39.502790 31095 consensus_peers.cc:485] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:14:39.507071 27937 consensus_peers.cc:485] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 08:14:39.542502 31027 raft_consensus.cc:1203] T bab5c25c31a64b578489b1006f631cc0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:14:39.543332 12681 raft_consensus.cc:2932] T bab5c25c31a64b578489b1006f631cc0 P 1b411c6bb25549368041773d7b710d48 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:14:39.543992 31027 raft_consensus.cc:2932] T bab5c25c31a64b578489b1006f631cc0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
W20240503 08:14:39.551659 30960 consensus_peers.cc:485] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 08:14:39.554333 28389 raft_consensus.cc:2932] T bab5c25c31a64b578489b1006f631cc0 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
I20240503 08:14:39.555732 27857 catalog_manager.cc:5484] T bab5c25c31a64b578489b1006f631cc0 P ced21ca983b749d0b4d4f6a0e2426bf8 reported cstate change: config changed from index -1 to 4, VOTER ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71) added. New cstate: current_term: 1 leader_uuid: "1b411c6bb25549368041773d7b710d48" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
W20240503 08:14:39.585103 28067 consensus_peers.cc:485] T 658865ee3f8d424ca5cd35e12e6d7c77 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:14:39.590194 28327 consensus_peers.cc:485] T 8425cfe5c6134a60852e12f9af55227b P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:14:39.590989 28067 consensus_peers.cc:485] T 2dce5b2aaea045ccb7424cbbd44576c2 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:14:39.601256 28327 consensus_peers.cc:485] T eeee6a265fc94e91bf9daa8301662e5f P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 08:14:39.618650 31161 consensus_queue.cc:237] T 23fc18f9417148c2a6d451bcca395338 P b00f244c69df401997ea99d492218a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
W20240503 08:14:39.621637 31097 consensus_peers.cc:485] T 23fc18f9417148c2a6d451bcca395338 P b00f244c69df401997ea99d492218a98 -> Peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): Couldn't send request to peer d09a11a03989421b9f28019dc1c14cc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 23fc18f9417148c2a6d451bcca395338. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:39.622157 31025 raft_consensus.cc:1261] T 23fc18f9417148c2a6d451bcca395338 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:39.622686 12801 consensus_queue.cc:1035] T 23fc18f9417148c2a6d451bcca395338 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:39.623414 28130 raft_consensus.cc:1261] T 23fc18f9417148c2a6d451bcca395338 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Refusing update from remote peer b00f244c69df401997ea99d492218a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:39.624663 12801 consensus_queue.cc:1035] T 23fc18f9417148c2a6d451bcca395338 P b00f244c69df401997ea99d492218a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:39.626999 11144 raft_consensus.cc:2932] T 23fc18f9417148c2a6d451bcca395338 P b00f244c69df401997ea99d492218a98 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:14:39.631177 28130 raft_consensus.cc:2932] T 23fc18f9417148c2a6d451bcca395338 P 1b411c6bb25549368041773d7b710d48 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:14:39.631561 31025 raft_consensus.cc:2932] T 23fc18f9417148c2a6d451bcca395338 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
I20240503 08:14:39.635424 27861 catalog_manager.cc:5484] T 23fc18f9417148c2a6d451bcca395338 P b00f244c69df401997ea99d492218a98 reported cstate change: config changed from index -1 to 2, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) added. New cstate: current_term: 1 leader_uuid: "b00f244c69df401997ea99d492218a98" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } }
W20240503 08:14:39.642812 27937 consensus_peers.cc:485] T 0ceedfb43869465bb63931c39c28583a P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 08:14:39.714495 28389 consensus_queue.cc:237] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:14:39.717006 28130 raft_consensus.cc:1261] T 07866de372e541ec9dca8a3d42f8b0ab P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Refusing update from remote peer d09a11a03989421b9f28019dc1c14cc2: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20240503 08:14:39.717125 28326 consensus_peers.cc:485] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2 -> Peer ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): Couldn't send request to peer ce1496a47d084365ad786cd09e472a2c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 07866de372e541ec9dca8a3d42f8b0ab. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:39.717552 12356 consensus_queue.cc:1035] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 08:14:39.718389 28327 consensus_peers.cc:485] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:39.791697 28327 consensus_peers.cc:485] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:39.791810 28327 consensus_peers.cc:485] T 45e7a03f838d4beaae2080d523319d58 P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:14:39.895035 28327 consensus_peers.cc:485] T 8bcbc022caa94bb08a2db4de9b060557 P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:14:39.913533 30828 consensus_peers.cc:485] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:14:40.002815 30828 consensus_peers.cc:462] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:40.154024 27937 consensus_peers.cc:485] T d9098f3d36b24d578a3f8d7d3ccad052 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 08:14:40.157935 12824 ts_tablet_manager.cc:902] T 23fc18f9417148c2a6d451bcca395338 P d09a11a03989421b9f28019dc1c14cc2: Initiating tablet copy from peer b00f244c69df401997ea99d492218a98 (127.20.230.72:40893)
I20240503 08:14:40.158986 12824 tablet_copy_client.cc:308] T 23fc18f9417148c2a6d451bcca395338 P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.72:40893
I20240503 08:14:40.159713 31181 tablet_copy_service.cc:140] P b00f244c69df401997ea99d492218a98: Received BeginTabletCopySession request for tablet 23fc18f9417148c2a6d451bcca395338 from peer d09a11a03989421b9f28019dc1c14cc2 ({username='slave'} at 127.20.230.68:37433)
I20240503 08:14:40.159891 31181 tablet_copy_service.cc:161] P b00f244c69df401997ea99d492218a98: Beginning new tablet copy session on tablet 23fc18f9417148c2a6d451bcca395338 from peer d09a11a03989421b9f28019dc1c14cc2 at {username='slave'} at 127.20.230.68:37433: session id = d09a11a03989421b9f28019dc1c14cc2-23fc18f9417148c2a6d451bcca395338
I20240503 08:14:40.162550 31181 tablet_copy_source_session.cc:205] T 23fc18f9417148c2a6d451bcca395338 P b00f244c69df401997ea99d492218a98: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:40.163898 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23fc18f9417148c2a6d451bcca395338. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:40.169860 12824 tablet_copy_client.cc:783] T 23fc18f9417148c2a6d451bcca395338 P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:40.170092 12824 tablet_copy_client.cc:647] T 23fc18f9417148c2a6d451bcca395338 P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:40.171617 12824 tablet_copy_client.cc:523] T 23fc18f9417148c2a6d451bcca395338 P d09a11a03989421b9f28019dc1c14cc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:40.174175 12824 tablet_bootstrap.cc:492] T 23fc18f9417148c2a6d451bcca395338 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap starting.
I20240503 08:14:40.178846 12824 tablet_bootstrap.cc:492] T 23fc18f9417148c2a6d451bcca395338 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:40.179116 12824 tablet_bootstrap.cc:492] T 23fc18f9417148c2a6d451bcca395338 P d09a11a03989421b9f28019dc1c14cc2: Bootstrap complete.
I20240503 08:14:40.179299 12824 ts_tablet_manager.cc:1367] T 23fc18f9417148c2a6d451bcca395338 P d09a11a03989421b9f28019dc1c14cc2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240503 08:14:40.179760 12824 raft_consensus.cc:348] T 23fc18f9417148c2a6d451bcca395338 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:14:40.179916 12824 raft_consensus.cc:729] T 23fc18f9417148c2a6d451bcca395338 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a11a03989421b9f28019dc1c14cc2, State: Initialized, Role: FOLLOWER
I20240503 08:14:40.180107 12824 consensus_queue.cc:260] T 23fc18f9417148c2a6d451bcca395338 P d09a11a03989421b9f28019dc1c14cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:14:40.180528 12824 ts_tablet_manager.cc:1398] T 23fc18f9417148c2a6d451bcca395338 P d09a11a03989421b9f28019dc1c14cc2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:14:40.181182 31181 tablet_copy_service.cc:342] P b00f244c69df401997ea99d492218a98: Request end of tablet copy session d09a11a03989421b9f28019dc1c14cc2-23fc18f9417148c2a6d451bcca395338 received from {username='slave'} at 127.20.230.68:37433
I20240503 08:14:40.181315 31181 tablet_copy_service.cc:433] P b00f244c69df401997ea99d492218a98: ending tablet copy session d09a11a03989421b9f28019dc1c14cc2-23fc18f9417148c2a6d451bcca395338 on tablet 23fc18f9417148c2a6d451bcca395338 with peer d09a11a03989421b9f28019dc1c14cc2
W20240503 08:14:40.214428 28067 consensus_peers.cc:485] T d94f5824a71f454f8a1bfa13dd032684 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:40.218276 28327 consensus_peers.cc:485] T ba1fc93781714efb8eb7ec67c2fceb8a P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 08:14:40.256474 12831 ts_tablet_manager.cc:902] T 07866de372e541ec9dca8a3d42f8b0ab P ce1496a47d084365ad786cd09e472a2c: Initiating tablet copy from peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:14:40.265544 12831 tablet_copy_client.cc:308] T 07866de372e541ec9dca8a3d42f8b0ab P ce1496a47d084365ad786cd09e472a2c: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.68:46211
I20240503 08:14:40.266458 28413 tablet_copy_service.cc:140] P d09a11a03989421b9f28019dc1c14cc2: Received BeginTabletCopySession request for tablet 07866de372e541ec9dca8a3d42f8b0ab from peer ce1496a47d084365ad786cd09e472a2c ({username='slave'} at 127.20.230.70:51715)
I20240503 08:14:40.266621 28413 tablet_copy_service.cc:161] P d09a11a03989421b9f28019dc1c14cc2: Beginning new tablet copy session on tablet 07866de372e541ec9dca8a3d42f8b0ab from peer ce1496a47d084365ad786cd09e472a2c at {username='slave'} at 127.20.230.70:51715: session id = ce1496a47d084365ad786cd09e472a2c-07866de372e541ec9dca8a3d42f8b0ab
I20240503 08:14:40.269327 28413 tablet_copy_source_session.cc:205] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:14:40.270928 12831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07866de372e541ec9dca8a3d42f8b0ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:40.277040 12831 tablet_copy_client.cc:783] T 07866de372e541ec9dca8a3d42f8b0ab P ce1496a47d084365ad786cd09e472a2c: tablet copy: Starting download of 5 data blocks...
I20240503 08:14:40.286315 12831 tablet_copy_client.cc:647] T 07866de372e541ec9dca8a3d42f8b0ab P ce1496a47d084365ad786cd09e472a2c: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:40.287851 12831 tablet_copy_client.cc:523] T 07866de372e541ec9dca8a3d42f8b0ab P ce1496a47d084365ad786cd09e472a2c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:40.290966 12831 tablet_bootstrap.cc:492] T 07866de372e541ec9dca8a3d42f8b0ab P ce1496a47d084365ad786cd09e472a2c: Bootstrap starting.
I20240503 08:14:40.303382 12831 tablet_bootstrap.cc:492] T 07866de372e541ec9dca8a3d42f8b0ab P ce1496a47d084365ad786cd09e472a2c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=4 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:14:40.303789 12831 tablet_bootstrap.cc:492] T 07866de372e541ec9dca8a3d42f8b0ab P ce1496a47d084365ad786cd09e472a2c: Bootstrap complete.
I20240503 08:14:40.304044 12831 ts_tablet_manager.cc:1367] T 07866de372e541ec9dca8a3d42f8b0ab P ce1496a47d084365ad786cd09e472a2c: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.001s
I20240503 08:14:40.304724 12831 raft_consensus.cc:348] T 07866de372e541ec9dca8a3d42f8b0ab P ce1496a47d084365ad786cd09e472a2c [term 2 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:14:40.305151 12831 raft_consensus.cc:729] T 07866de372e541ec9dca8a3d42f8b0ab P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Initialized, Role: FOLLOWER
I20240503 08:14:40.305383 12831 consensus_queue.cc:260] T 07866de372e541ec9dca8a3d42f8b0ab P ce1496a47d084365ad786cd09e472a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:14:40.305976 12831 ts_tablet_manager.cc:1398] T 07866de372e541ec9dca8a3d42f8b0ab P ce1496a47d084365ad786cd09e472a2c: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 08:14:40.306833 28413 tablet_copy_service.cc:342] P d09a11a03989421b9f28019dc1c14cc2: Request end of tablet copy session ce1496a47d084365ad786cd09e472a2c-07866de372e541ec9dca8a3d42f8b0ab received from {username='slave'} at 127.20.230.70:51715
I20240503 08:14:40.306931 28413 tablet_copy_service.cc:433] P d09a11a03989421b9f28019dc1c14cc2: ending tablet copy session ce1496a47d084365ad786cd09e472a2c-07866de372e541ec9dca8a3d42f8b0ab on tablet 07866de372e541ec9dca8a3d42f8b0ab with peer ce1496a47d084365ad786cd09e472a2c
I20240503 08:14:40.366326 30941 maintenance_manager.cc:419] P ce1496a47d084365ad786cd09e472a2c: Scheduling UndoDeltaBlockGCOp(07866de372e541ec9dca8a3d42f8b0ab): 4583 bytes on disk
I20240503 08:14:40.366853 30831 maintenance_manager.cc:643] P ce1496a47d084365ad786cd09e472a2c: UndoDeltaBlockGCOp(07866de372e541ec9dca8a3d42f8b0ab) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
W20240503 08:14:40.482808 30828 consensus_peers.cc:462] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:40.599267 30828 consensus_peers.cc:485] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:14:40.628021 30960 consensus_peers.cc:485] T 41aa0e288ca4469ba34a07ff159ace91 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 08:14:40.706802 30892 raft_consensus.cc:1203] T 07866de372e541ec9dca8a3d42f8b0ab P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.4->[2.5-2.5] Dedup: 2.5->[]
I20240503 08:14:40.707612 12821 raft_consensus.cc:2932] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index -1 to 5, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:14:40.708570 30892 raft_consensus.cc:2932] T 07866de372e541ec9dca8a3d42f8b0ab P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index -1 to 5, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:14:40.716127 28389 raft_consensus.cc:1203] T 23fc18f9417148c2a6d451bcca395338 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:14:40.716065 27859 catalog_manager.cc:5484] T 07866de372e541ec9dca8a3d42f8b0ab P ce1496a47d084365ad786cd09e472a2c reported cstate change: config changed from index -1 to 5, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New cstate: current_term: 2 leader_uuid: "d09a11a03989421b9f28019dc1c14cc2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:14:40.734050 28130 raft_consensus.cc:2932] T 07866de372e541ec9dca8a3d42f8b0ab P 1b411c6bb25549368041773d7b710d48 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index -1 to 5, VOTER ce1496a47d084365ad786cd09e472a2c (127.20.230.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:14:40.813578 30760 consensus_queue.cc:237] T 91c573181144456aa7e27a3026a639c4 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:14:40.816437 31027 raft_consensus.cc:1261] T 91c573181144456aa7e27a3026a639c4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:14:40.816869 30696 consensus_peers.cc:485] T 91c573181144456aa7e27a3026a639c4 P 6d1d7de19459435fbde1999e3dfd226b -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 91c573181144456aa7e27a3026a639c4. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:40.817704 28389 raft_consensus.cc:1261] T 91c573181144456aa7e27a3026a639c4 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:40.817806 12584 consensus_queue.cc:1035] T 91c573181144456aa7e27a3026a639c4 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:40.819837 12865 consensus_queue.cc:1035] T 91c573181144456aa7e27a3026a639c4 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:40.822854 12584 raft_consensus.cc:2932] T 91c573181144456aa7e27a3026a639c4 P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:14:40.823415 31027 raft_consensus.cc:2932] T 91c573181144456aa7e27a3026a639c4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:14:40.828881 27861 catalog_manager.cc:5484] T 91c573181144456aa7e27a3026a639c4 P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:14:40.829115 28389 raft_consensus.cc:2932] T 91c573181144456aa7e27a3026a639c4 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
W20240503 08:14:40.836247 28067 consensus_peers.cc:485] T bab5c25c31a64b578489b1006f631cc0 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:14:40.866958 30760 consensus_queue.cc:237] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:14:40.869369 28389 raft_consensus.cc:1261] T 2c77993083a049668875b1ecd5a893ec P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:40.869364 31027 raft_consensus.cc:1261] T 2c77993083a049668875b1ecd5a893ec P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:40.869856 12584 consensus_queue.cc:1035] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 08:14:40.870157 30696 consensus_peers.cc:485] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2c77993083a049668875b1ecd5a893ec. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:40.870162 12865 consensus_queue.cc:1035] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:40.874543 12584 raft_consensus.cc:2932] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:14:40.875108 31027 raft_consensus.cc:2932] T 2c77993083a049668875b1ecd5a893ec P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:14:40.878810 28389 raft_consensus.cc:2932] T 2c77993083a049668875b1ecd5a893ec P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:14:40.882148 27859 catalog_manager.cc:5484] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: config changed from index -1 to 2, VOTER 1b411c6bb25549368041773d7b710d48 (127.20.230.66) added. New cstate: current_term: 1 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
W20240503 08:14:40.928750 28327 consensus_peers.cc:485] T cdad6ef53b6f413ebfda67f9b42c9c23 P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:14:40.957366 28067 consensus_peers.cc:485] T f7a8c95ac81942dabef98a07caa84082 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:14:40.963009 27937 consensus_peers.cc:485] T f1b2f3b8111649a7b5725647b8251698 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:40.975452 28327 consensus_peers.cc:485] T f46393257ce9471891698c36a07546ad P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:14:40.997792 30828 consensus_peers.cc:462] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:41.043183 28067 consensus_peers.cc:485] T 8450e62352a74d2ea1471166d4cb1132 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:14:41.044080 27937 consensus_peers.cc:485] T 08fb64977309479b831635344969c662 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:14:41.056111 28067 consensus_peers.cc:485] T 1cac39d5831d4b6887250f7a9d63dcd5 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:14:41.109378 27937 consensus_peers.cc:485] T ca38671a760d4534ab903579e0487772 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:14:41.111395 28067 consensus_peers.cc:485] T 8fc87f7f3c724cd4a270583ddebd87aa P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:14:41.114217 28067 consensus_peers.cc:485] T 17c6e3293aad4591a9c1395ebffff2e6 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:14:41.125660 27937 consensus_peers.cc:485] T 8cc6ae30aa60495d8a4364a11d5bef79 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 08:14:41.148082 28111 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240503 08:14:41.176250 30695 consensus_peers.cc:485] T 2222ff7fb5184d6c86010e1475d6c1f7 P 6d1d7de19459435fbde1999e3dfd226b -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 08:14:41.231945 31141 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 08:14:41.237368 30742 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:14:41.257452 28067 consensus_peers.cc:485] T 978bf5783b0e47e3bfb30441adc600ae P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:14:41.257627 28067 consensus_peers.cc:485] T 362590d3d0834d0b8a672ab552f8c209 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:14:41.290517 27937 consensus_peers.cc:485] T 46d27fa12990405cae5f94976ac7a014 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 08:14:41.295787 27983 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 08:14:41.319214 28067 consensus_peers.cc:485] T 099e37089b2e42b4839e681f8f37365d P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:41.326325 31095 consensus_peers.cc:485] T 6a96965691644db8b9461d146536a026 P b00f244c69df401997ea99d492218a98 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 08:14:41.343119 30874 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:41.367655 12959 ts_tablet_manager.cc:902] T 91c573181144456aa7e27a3026a639c4 P 1b411c6bb25549368041773d7b710d48: Initiating tablet copy from peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:14:41.368597 28373 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 08:14:41.385159 28067 consensus_peers.cc:485] T de5dfccb0359462dbb7da776293fa679 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:14:41.388500 30828 consensus_peers.cc:485] T 0f25688be4e74d31aaab85b7a4329a34 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:14:41.403640 30828 consensus_peers.cc:485] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:14:41.418834 31095 consensus_peers.cc:485] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 08:14:41.426338 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240503 08:14:41.437549 30960 consensus_peers.cc:485] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 08:14:41.446792 12959 tablet_copy_client.cc:308] T 91c573181144456aa7e27a3026a639c4 P 1b411c6bb25549368041773d7b710d48: tablet copy: Beginning tablet copy session from remote peer at address 127.20.230.69:34725
I20240503 08:14:41.454800 30783 tablet_copy_service.cc:140] P 6d1d7de19459435fbde1999e3dfd226b: Received BeginTabletCopySession request for tablet 91c573181144456aa7e27a3026a639c4 from peer 1b411c6bb25549368041773d7b710d48 ({username='slave'} at 127.20.230.66:42445)
I20240503 08:14:41.454968 30783 tablet_copy_service.cc:161] P 6d1d7de19459435fbde1999e3dfd226b: Beginning new tablet copy session on tablet 91c573181144456aa7e27a3026a639c4 from peer 1b411c6bb25549368041773d7b710d48 at {username='slave'} at 127.20.230.66:42445: session id = 1b411c6bb25549368041773d7b710d48-91c573181144456aa7e27a3026a639c4
I20240503 08:14:41.457715 30783 tablet_copy_source_session.cc:205] T 91c573181144456aa7e27a3026a639c4 P 6d1d7de19459435fbde1999e3dfd226b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:41.462045 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91c573181144456aa7e27a3026a639c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:41.464972 12967 ts_tablet_manager.cc:902] T 2c77993083a049668875b1ecd5a893ec P 1b411c6bb25549368041773d7b710d48: Initiating tablet copy from peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
F20240503 08:14:41.465519 12967 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-1cac39d5831d4b6887250f7a9d63dcd5.queue_time_us, tablet-download-pool-1cac39d5831d4b6887250f7a9d63dcd5.queue_time_us) (tablet-download-pool-1cac39d5831d4b6887250f7a9d63dcd5.run_cpu_time_us, tablet-download-pool-1cac39d5831d4b6887250f7a9d63dcd5.run_cpu_time_us) (tablet-download-pool-1cac39d5831d4b6887250f7a9d63dcd5.run_wall_time_us, tablet-download-pool-1cac39d5831d4b6887250f7a9d63dcd5.run_wall_time_us) (tablet-download-pool-1fca3d7f4b734eceb210e1186478eb72.queue_time_us, tablet-download-pool-1fca3d7f4b734eceb210e1186478eb72.queue_time_us) (tablet-download-pool-1fca3d7f4b734eceb210e1186478eb72.run_cpu_time_us, tablet-download-pool-1fca3d7f4b734eceb210e1186478eb72.run_cpu_time_us) (tablet-download-pool-1fca3d7f4b734eceb210e1186478eb72.run_wall_time_us, tablet-download-pool-1fca3d7f4b734eceb210e1186478eb72.run_wall_time_us) (tablet-download-pool-2222ff7fb5184d6c86010e1475d6c1f7.queue_time_us, tablet-download-pool-2222ff7fb5184d6c86010e1475d6c1f7.queue_time_us) (tablet-download-pool-2222ff7fb5184d6c86010e1475d6c1f7.run_cpu_time_us, tablet-download-pool-2222ff7fb5184d6c86010e1475d6c1f7.run_cpu_time_us) (tablet-download-pool-2222ff7fb5184d6c86010e1475d6c1f7.run_wall_time_us, tablet-download-pool-2222ff7fb5184d6c86010e1475d6c1f7.run_wall_time_us) (tablet-download-pool-2c77993083a049668875b1ecd5a893ec.queue_time_us, tablet-download-pool-2c77993083a049668875b1ecd5a893ec.queue_time_us) (tablet-download-pool-417cedc3fa5c4a85a7a130005291d64b.queue_time_us, tablet-download-pool-417cedc3fa5c4a85a7a130005291d64b.queue_time_us) (tablet-download-pool-417cedc3fa5c4a85a7a130005291d64b.run_cpu_time_us, tablet-download-pool-417cedc3fa5c4a85a7a130005291d64b.run_cpu_time_us) (tablet-download-pool-417cedc3fa5c4a85a7a130005291d64b.run_wall_time_us, tablet-download-pool-417cedc3fa5c4a85a7a130005291d64b.run_wall_time_us) (tablet-download-pool-41e26e13355d4e0cb56bb526415a3998.queue_time_us, tablet-download-pool-41e26e13355d4e0cb56bb526415a3998.queue_time_us) (tablet-download-pool-41e26e13355d4e0cb56bb526415a3998.run_cpu_time_us, tablet-download-pool-41e26e13355d4e0cb56bb526415a3998.run_cpu_time_us) (tablet-download-pool-41e26e13355d4e0cb56bb526415a3998.run_wall_time_us, tablet-download-pool-41e26e13355d4e0cb56bb526415a3998.run_wall_time_us) (tablet-download-pool-50e4d2c569344dde915eec12f6f3d747.queue_time_us, tablet-download-pool-50e4d2c569344dde915eec12f6f3d747.queue_time_us) (tablet-download-pool-50e4d2c569344dde915eec12f6f3d747.run_cpu_time_us, tablet-download-pool-50e4d2c569344dde915eec12f6f3d747.run_cpu_time_us) (tablet-download-pool-50e4d2c569344dde915eec12f6f3d747.run_wall_time_us, tablet-download-pool-50e4d2c569344dde915eec12f6f3d747.run_wall_time_us) (tablet-download-pool-62ca502a089f4a37999d2252776b741b.queue_time_us, tablet-download-pool-62ca502a089f4a37999d2252776b741b.queue_time_us) (tablet-download-pool-62ca502a089f4a37999d2252776b741b.run_cpu_time_us, tablet-download-pool-62ca502a089f4a37999d2252776b741b.run_cpu_time_us) (tablet-download-pool-62ca502a089f4a37999d2252776b741b.run_wall_time_us, tablet-download-pool-62ca502a089f4a37999d2252776b741b.run_wall_time_us) (tablet-download-pool-6c2824aa878845e098f600f1fcfea087.queue_time_us, tablet-download-pool-6c2824aa878845e098f600f1fcfea087.queue_time_us) (tablet-download-pool-6c2824aa878845e098f600f1fcfea087.run_cpu_time_us, tablet-download-pool-6c2824aa878845e098f600f1fcfea087.run_cpu_time_us) (tablet-download-pool-6c2824aa878845e098f600f1fcfea087.run_wall_time_us, tablet-download-pool-6c2824aa878845e098f600f1fcfea087.run_wall_time_us) (tablet-download-pool-720f36100d794158be13ff02595e6ddf.queue_time_us, tablet-download-pool-720f36100d794158be13ff02595e6ddf.queue_time_us) (tablet-download-pool-720f36100d794158be13ff02595e6ddf.run_cpu_time_us, tablet-download-pool-720f36100d794158be13ff02595e6ddf.run_cpu_time_us) (tablet-download-pool-720f36100d794158be13ff02595e6ddf.run_wall_time_us, tablet-download-pool-720f36100d794158be13ff02595e6ddf.run_wall_time_us) (tablet-download-pool-76d79d8be9e94e99a7ba51d903ca8a91.queue_time_us, tablet-download-pool-76d79d8be9e94e99a7ba51d903ca8a91.queue_time_us) (tablet-download-pool-76d79d8be9e94e99a7ba51d903ca8a91.run_cpu_time_us, tablet-download-pool-76d79d8be9e94e99a7ba51d903ca8a91.run_cpu_time_us) (tablet-download-pool-76d79d8be9e94e99a7ba51d903ca8a91.run_wall_time_us, tablet-download-pool-76d79d8be9e94e99a7ba51d903ca8a91.run_wall_time_us) (tablet-download-pool-7df30eb244914dcabc6965324eced236.queue_time_us, tablet-download-pool-7df30eb244914dcabc6965324eced236.queue_time_us) (tablet-download-pool-7df30eb244914dcabc6965324eced236.run_cpu_time_us, tablet-download-pool-7df30eb244914dcabc6965324eced236.run_cpu_time_us) (tablet-download-pool-7df30eb244914dcabc6965324eced236.run_wall_time_us, tablet-download-pool-7df30eb244914dcabc6965324eced236.run_wall_time_us) (tablet-download-pool-8c714f29abb140379b57e08eb927e2b7.queue_time_us, tablet-download-pool-8c714f29abb140379b57e08eb927e2b7.queue_time_us) (tablet-download-pool-8c714f29abb140379b57e08eb927e2b7.run_cpu_time_us, tablet-download-pool-8c714f29abb140379b57e08eb927e2b7.run_cpu_time_us) (tablet-download-pool-8c714f29abb140379b57e08eb927e2b7.run_wall_time_us, tablet-download-pool-8c714f29abb140379b57e08eb927e2b7.run_wall_time_us) (tablet-download-pool-91c573181144456aa7e27a3026a639c4.queue_time_us, tablet-download-pool-91c573181144456aa7e27a3026a639c4.queue_time_us) (tablet-download-pool-91c573181144456aa7e27a3026a639c4.run_cpu_time_us, tablet-download-pool-91c573181144456aa7e27a3026a639c4.run_cpu_time_us) (tablet-download-pool-91c573181144456aa7e27a3026a639c4.run_wall_time_us, tablet-download-pool-91c573181144456aa7e27a3026a639c4.run_wall_time_us) (tablet-download-pool-ab7b01c2e05a430a9873398268b1e41f.queue_time_us, tablet-download-pool-ab7b01c2e05a430a9873398268b1e41f.queue_time_us) (tablet-download-pool-ab7b01c2e05a430a9873398268b1e41f.run_cpu_time_us, tablet-download-pool-ab7b01c2e05a430a9873398268b1e41f.run_cpu_time_us) (tablet-download-pool-ab7b01c2e05a430a9873398268b1e41f.run_wall_time_us, tablet-download-pool-ab7b01c2e05a430a9873398268b1e41f.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
W20240503 08:14:41.470468 31095 consensus_peers.cc:485] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 08:14:41.474089 12959 tablet_copy_client.cc:783] T 91c573181144456aa7e27a3026a639c4 P 1b411c6bb25549368041773d7b710d48: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:41.474304 12959 tablet_copy_client.cc:647] T 91c573181144456aa7e27a3026a639c4 P 1b411c6bb25549368041773d7b710d48: tablet copy: Starting download of 1 WAL segments...
@ 0x7f7be704cdcd google::LogMessage::Fail() at ??:0
@ 0x7f7be7050b93 google::LogMessage::SendToLog() at ??:0
I20240503 08:14:41.478410 12959 tablet_copy_client.cc:523] T 91c573181144456aa7e27a3026a639c4 P 1b411c6bb25549368041773d7b710d48: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:41.481956 12959 tablet_bootstrap.cc:492] T 91c573181144456aa7e27a3026a639c4 P 1b411c6bb25549368041773d7b710d48: Bootstrap starting.
@ 0x7f7be704c7cc google::LogMessage::Flush() at ??:0
@ 0x7f7be704df59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7be9d8d742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f7be9d59f8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f7be9d53cd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f7bf91a78de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f7bf91bfdaf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
W20240503 08:14:41.530642 30828 consensus_peers.cc:485] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
@ 0x7f7bf92ea480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
W20240503 08:14:41.538763 28327 consensus_peers.cc:485] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
@ 0x7f7bf9322a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f7bf9305981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
W20240503 08:14:41.562410 28067 consensus_peers.cc:485] T fa7e18041c294a239eb7b8f735b66c53 P 1b411c6bb25549368041773d7b710d48 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
@ 0x7f7bf930570e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f7be9d5db75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7be9d6119c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f7be9d61019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
W20240503 08:14:41.582334 30828 consensus_peers.cc:462] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
@ 0x7f7be9d3c01e kudu::Thread::SuperviseThread() at ??:0
W20240503 08:14:41.582962 27937 consensus_peers.cc:485] T c8ca21ffe16e41ab80f5693dc4dbb682 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
@ 0x7f7bed0c16db start_thread at ??:0
@ 0x7f7be5eff71f clone at ??:0
W20240503 08:14:41.611474 28327 consensus_peers.cc:485] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:14:41.641142 30695 consensus_peers.cc:485] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:14:41.661820 27937 consensus_peers.cc:485] T 3d0498256b5f4b8fb1681759429f10ed P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:14:41.673683 28327 consensus_peers.cc:485] T bdfe68a127f043f08f04749843a38375 P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:14:41.770398 30828 consensus_peers.cc:485] T 045b3ce4b1d44f94a802b41a12e2704d P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:14:41.781549 30828 consensus_peers.cc:485] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:14:41.803634 27937 consensus_peers.cc:485] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:14:41.856956 30960 consensus_peers.cc:485] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:14:41.900676 28327 consensus_peers.cc:485] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:14:41.906862 27937 consensus_peers.cc:485] T 7c8c7659181040cabf20ae116fae78c8 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:14:41.917928 30960 consensus_peers.cc:485] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:14:41.921733 31095 consensus_peers.cc:485] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:14:41.956099 27937 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111) [suppressed 127 similar messages]
W20240503 08:14:41.957329 27937 consensus_peers.cc:485] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:14:41.961478 31095 consensus_peers.cc:485] T 2c595805729d48fd974d761ed8a60158 P b00f244c69df401997ea99d492218a98 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:14:41.970789 27937 consensus_peers.cc:485] T e0b904461c1c44ed93e1a227d93498eb P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:14:41.983289 28327 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240503 08:14:41.994737 28327 consensus_peers.cc:485] T 8425cfe5c6134a60852e12f9af55227b P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:14:41.995711 30960 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 08:14:42.018183 30828 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240503 08:14:42.019657 31095 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:14:42.019755 30829 connection.cc:535] client connection to 127.20.230.66:37143 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:14:42.021411 30696 connection.cc:535] client connection to 127.20.230.66:37143 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:14:42.021453 27938 connection.cc:535] client connection to 127.20.230.66:37143 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:14:42.021540 30696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 19 similar messages]
W20240503 08:14:42.021529 30961 connection.cc:535] client connection to 127.20.230.66:37143 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:14:42.022508 31096 connection.cc:535] client connection to 127.20.230.66:37143 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:14:42.022722 28328 connection.cc:535] client connection to 127.20.230.66:37143 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:14:42.022871 12210 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:42.020289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.230.66:37143 (local address 127.20.230.70:39707)
0503 08:14:42.020416 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:42.020432 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:42.020549 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:42.022350 (+ 1801us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":33}
W20240503 08:14:42.023387 30829 consensus_peers.cc:485] T d0657f27fd474a41b0c33db55989daa3 P ce1496a47d084365ad786cd09e472a2c -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:42.023500 30829 consensus_peers.cc:485] T 62ca502a089f4a37999d2252776b741b P ce1496a47d084365ad786cd09e472a2c -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:42.023605 30829 consensus_peers.cc:485] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:42.023712 12206 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:42.022120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.230.66:37143 (local address 127.20.230.65:46585)
0503 08:14:42.023173 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:42.023249 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":938}
W20240503 08:14:42.023896 30961 consensus_peers.cc:485] T 7df30eb244914dcabc6965324eced236 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:42.024015 30961 consensus_peers.cc:485] T 50e4d2c569344dde915eec12f6f3d747 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:42.024114 30961 consensus_peers.cc:485] T 74f43feb3bc342a18723bc5a143bdd99 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:42.024204 30961 consensus_peers.cc:485] T 4ed8026daee840c6b31ca7d989ed25f0 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:42.024322 27938 consensus_peers.cc:485] T c68032ba399547e3bd272f424927fbce P 833582ff68c94269be364ebd187a75db -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:42.024433 27938 consensus_peers.cc:485] T 45f47478f49149b895060791655503ae P 833582ff68c94269be364ebd187a75db -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:42.024516 27938 consensus_peers.cc:485] T 0ceedfb43869465bb63931c39c28583a P 833582ff68c94269be364ebd187a75db -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:42.024575 27938 consensus_peers.cc:485] T 3d0498256b5f4b8fb1681759429f10ed P 833582ff68c94269be364ebd187a75db -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:42.024629 27938 consensus_peers.cc:485] T 417cedc3fa5c4a85a7a130005291d64b P 833582ff68c94269be364ebd187a75db -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:42.024782 31096 consensus_peers.cc:485] T 23fc18f9417148c2a6d451bcca395338 P b00f244c69df401997ea99d492218a98 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:42.026121 31096 consensus_peers.cc:485] T 2c595805729d48fd974d761ed8a60158 P b00f244c69df401997ea99d492218a98 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:42.027346 30696 consensus_peers.cc:462] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
W20240503 08:14:42.027474 30696 consensus_peers.cc:485] T 41e26e13355d4e0cb56bb526415a3998 P 6d1d7de19459435fbde1999e3dfd226b -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:42.027582 30696 consensus_peers.cc:485] T 6c2824aa878845e098f600f1fcfea087 P 6d1d7de19459435fbde1999e3dfd226b -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:42.027700 30696 consensus_peers.cc:485] T 2222ff7fb5184d6c86010e1475d6c1f7 P 6d1d7de19459435fbde1999e3dfd226b -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:42.027783 30696 consensus_peers.cc:485] T 607a503b64d8440a81cf66ab158ae273 P 6d1d7de19459435fbde1999e3dfd226b -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:42.028066 28328 consensus_peers.cc:485] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:42.028190 28328 consensus_peers.cc:485] T 720f36100d794158be13ff02595e6ddf P d09a11a03989421b9f28019dc1c14cc2 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:42.028291 28328 consensus_peers.cc:485] T ab7b01c2e05a430a9873398268b1e41f P d09a11a03989421b9f28019dc1c14cc2 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:42.028389 28328 consensus_peers.cc:485] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:42.028475 28328 consensus_peers.cc:485] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:42.028563 28328 consensus_peers.cc:485] T 1fca3d7f4b734eceb210e1186478eb72 P d09a11a03989421b9f28019dc1c14cc2 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:42.028645 28328 consensus_peers.cc:485] T 8bcbc022caa94bb08a2db4de9b060557 P d09a11a03989421b9f28019dc1c14cc2 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:42.048647 27937 consensus_peers.cc:485] T 0ceedfb43869465bb63931c39c28583a P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:14:42.061300 30829 consensus_peers.cc:485] T 7815cb2db9514c02b0ee3e3cd12b2212 P ce1496a47d084365ad786cd09e472a2c -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:42.073422 27983 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 08:14:42.095777 30828 consensus_peers.cc:485] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:14:42.099104 28327 consensus_peers.cc:485] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:14:42.100054 31141 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 08:14:42.108604 28373 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 08:14:42.112056 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240503 08:14:42.116662 28328 consensus_peers.cc:485] T 76d79d8be9e94e99a7ba51d903ca8a91 P d09a11a03989421b9f28019dc1c14cc2 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:42.125175 30742 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:42.143375 30874 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 08:14:42.158959 27938 consensus_peers.cc:485] T fa714cf4cc254703a2d356d2a536db8d P 833582ff68c94269be364ebd187a75db -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:42.176630 30696 consensus_peers.cc:485] T 1fcc779a649b461db4d14b5cd8b17744 P 6d1d7de19459435fbde1999e3dfd226b -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:42.180845 30828 consensus_peers.cc:462] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:42.197976 28327 consensus_peers.cc:485] T eeee6a265fc94e91bf9daa8301662e5f P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:14:42.259932 30696 consensus_peers.cc:462] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
W20240503 08:14:42.352953 28327 consensus_peers.cc:485] T 45e7a03f838d4beaae2080d523319d58 P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:14:42.391049 28327 consensus_peers.cc:485] T 8bcbc022caa94bb08a2db4de9b060557 P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:14:42.402266 28327 consensus_peers.cc:485] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:14:42.475644 30742 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:14:42.477459 27983 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:14:42.502966 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:14:42.507581 30874 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 08:14:42.508687 28327 consensus_peers.cc:485] T ba1fc93781714efb8eb7ec67c2fceb8a P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 08:14:42.513475 31141 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 08:14:42.515347 28373 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 08:14:42.549803 13061 raft_consensus.cc:482] T 7a101bee3af34166848ee7fa7c733892 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:42.549965 13061 raft_consensus.cc:504] T 7a101bee3af34166848ee7fa7c733892 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:14:42.550560 13061 leader_election.cc:288] T 7a101bee3af34166848ee7fa7c733892 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
W20240503 08:14:42.551985 28328 leader_election.cc:334] T 7a101bee3af34166848ee7fa7c733892 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:42.555296 30892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a101bee3af34166848ee7fa7c733892" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:14:42.558143 28326 leader_election.cc:302] T 7a101bee3af34166848ee7fa7c733892 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d09a11a03989421b9f28019dc1c14cc2; no voters: 1b411c6bb25549368041773d7b710d48, ce1496a47d084365ad786cd09e472a2c
I20240503 08:14:42.558599 12840 raft_consensus.cc:2726] T 7a101bee3af34166848ee7fa7c733892 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:14:42.574501 12664 raft_consensus.cc:482] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:42.574656 12664 raft_consensus.cc:504] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:42.575285 12664 leader_election.cc:288] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
W20240503 08:14:42.578128 30961 leader_election.cc:334] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:42.580651 11989 raft_consensus.cc:482] T d0a06e419c6c4f3db6705cb2377e9336 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:42.580801 11989 raft_consensus.cc:504] T d0a06e419c6c4f3db6705cb2377e9336 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:14:42.581372 11989 leader_election.cc:288] T d0a06e419c6c4f3db6705cb2377e9336 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:14:42.581904 27999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0a06e419c6c4f3db6705cb2377e9336" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
W20240503 08:14:42.583106 30829 leader_election.cc:334] T d0a06e419c6c4f3db6705cb2377e9336 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:42.583221 30829 leader_election.cc:302] T d0a06e419c6c4f3db6705cb2377e9336 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c; no voters: 1b411c6bb25549368041773d7b710d48, 833582ff68c94269be364ebd187a75db
I20240503 08:14:42.583603 11989 raft_consensus.cc:2726] T d0a06e419c6c4f3db6705cb2377e9336 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 08:14:42.601166 30960 leader_election.cc:334] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:42.601347 30960 leader_election.cc:302] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:42.601616 12664 raft_consensus.cc:2726] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:14:42.642419 27937 consensus_peers.cc:485] T d9098f3d36b24d578a3f8d7d3ccad052 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:14:42.652047 30828 consensus_peers.cc:462] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:42.654402 12664 raft_consensus.cc:482] T b52e4b59153d40378e2a37809b04a2c8 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:42.654558 12664 raft_consensus.cc:504] T b52e4b59153d40378e2a37809b04a2c8 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:14:42.655256 12664 leader_election.cc:288] T b52e4b59153d40378e2a37809b04a2c8 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:14:42.656180 30892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b52e4b59153d40378e2a37809b04a2c8" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:14:42.656904 12664 raft_consensus.cc:482] T 658865ee3f8d424ca5cd35e12e6d7c77 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:42.657048 12664 raft_consensus.cc:504] T 658865ee3f8d424ca5cd35e12e6d7c77 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:42.657605 12664 leader_election.cc:288] T 658865ee3f8d424ca5cd35e12e6d7c77 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
W20240503 08:14:42.657862 30961 leader_election.cc:334] T b52e4b59153d40378e2a37809b04a2c8 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:42.657980 30961 leader_election.cc:302] T b52e4b59153d40378e2a37809b04a2c8 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8; no voters: 1b411c6bb25549368041773d7b710d48, ce1496a47d084365ad786cd09e472a2c
I20240503 08:14:42.658197 12664 raft_consensus.cc:2726] T b52e4b59153d40378e2a37809b04a2c8 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 08:14:42.658865 30961 leader_election.cc:334] T 658865ee3f8d424ca5cd35e12e6d7c77 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
W20240503 08:14:42.659714 30960 leader_election.cc:334] T 658865ee3f8d424ca5cd35e12e6d7c77 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:42.659843 30960 leader_election.cc:302] T 658865ee3f8d424ca5cd35e12e6d7c77 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:42.660089 12664 raft_consensus.cc:2726] T 658865ee3f8d424ca5cd35e12e6d7c77 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 08:14:42.666401 30696 consensus_peers.cc:462] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:42.689965 11989 raft_consensus.cc:482] T b52e4b59153d40378e2a37809b04a2c8 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:42.690135 11989 raft_consensus.cc:504] T b52e4b59153d40378e2a37809b04a2c8 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:14:42.690752 11989 leader_election.cc:288] T b52e4b59153d40378e2a37809b04a2c8 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:14:42.691273 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b52e4b59153d40378e2a37809b04a2c8" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:14:42.691466 31027 raft_consensus.cc:2454] T b52e4b59153d40378e2a37809b04a2c8 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:14:42.692764 30828 leader_election.cc:302] T b52e4b59153d40378e2a37809b04a2c8 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
W20240503 08:14:42.694715 30829 leader_election.cc:334] T b52e4b59153d40378e2a37809b04a2c8 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:42.695178 11989 raft_consensus.cc:2781] T b52e4b59153d40378e2a37809b04a2c8 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:14:42.695259 11989 raft_consensus.cc:482] T b52e4b59153d40378e2a37809b04a2c8 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:42.695346 11989 raft_consensus.cc:3037] T b52e4b59153d40378e2a37809b04a2c8 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:42.697382 11989 raft_consensus.cc:504] T b52e4b59153d40378e2a37809b04a2c8 P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:14:42.718878 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b52e4b59153d40378e2a37809b04a2c8" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:14:42.718986 11989 leader_election.cc:288] T b52e4b59153d40378e2a37809b04a2c8 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 election: Requested vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:14:42.719105 31027 raft_consensus.cc:3037] T b52e4b59153d40378e2a37809b04a2c8 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:42.721450 31027 raft_consensus.cc:2454] T b52e4b59153d40378e2a37809b04a2c8 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 2.
I20240503 08:14:42.722719 30828 leader_election.cc:302] T b52e4b59153d40378e2a37809b04a2c8 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
W20240503 08:14:42.723356 30829 leader_election.cc:334] T b52e4b59153d40378e2a37809b04a2c8 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:42.723958 12941 raft_consensus.cc:2781] T b52e4b59153d40378e2a37809b04a2c8 P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:14:42.733193 12664 raft_consensus.cc:482] T ab35ebabd6fe45fab890739a800794b1 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:42.733342 12664 raft_consensus.cc:504] T ab35ebabd6fe45fab890739a800794b1 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:14:42.734053 12664 leader_election.cc:288] T ab35ebabd6fe45fab890739a800794b1 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:14:42.734549 28000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab35ebabd6fe45fab890739a800794b1" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
W20240503 08:14:42.736291 30961 leader_election.cc:334] T ab35ebabd6fe45fab890739a800794b1 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:42.736419 30961 leader_election.cc:302] T ab35ebabd6fe45fab890739a800794b1 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8; no voters: 1b411c6bb25549368041773d7b710d48, 833582ff68c94269be364ebd187a75db
I20240503 08:14:42.736899 12664 raft_consensus.cc:2726] T ab35ebabd6fe45fab890739a800794b1 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:14:42.737519 12941 raft_consensus.cc:686] T b52e4b59153d40378e2a37809b04a2c8 P ce1496a47d084365ad786cd09e472a2c [term 2 LEADER]: Becoming Leader. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Running, Role: LEADER
I20240503 08:14:42.737854 12941 consensus_queue.cc:237] T b52e4b59153d40378e2a37809b04a2c8 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:14:42.760519 27860 catalog_manager.cc:5484] T b52e4b59153d40378e2a37809b04a2c8 P ce1496a47d084365ad786cd09e472a2c reported cstate change: term changed from 1 to 2, leader changed from 1b411c6bb25549368041773d7b710d48 (127.20.230.66) to ce1496a47d084365ad786cd09e472a2c (127.20.230.70). New cstate: current_term: 2 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:14:42.769701 12941 raft_consensus.cc:482] T fa7e18041c294a239eb7b8f735b66c53 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:42.769845 12941 raft_consensus.cc:504] T fa7e18041c294a239eb7b8f735b66c53 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
W20240503 08:14:42.771453 30829 leader_election.cc:334] T fa7e18041c294a239eb7b8f735b66c53 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:42.771764 12941 leader_election.cc:288] T fa7e18041c294a239eb7b8f735b66c53 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
W20240503 08:14:42.772431 30828 leader_election.cc:334] T fa7e18041c294a239eb7b8f735b66c53 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:42.772532 30828 leader_election.cc:302] T fa7e18041c294a239eb7b8f735b66c53 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:42.773238 12941 raft_consensus.cc:2726] T fa7e18041c294a239eb7b8f735b66c53 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:14:42.782487 12963 raft_consensus.cc:482] T d0a06e419c6c4f3db6705cb2377e9336 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:42.782619 12963 raft_consensus.cc:504] T d0a06e419c6c4f3db6705cb2377e9336 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:14:42.783299 12963 leader_election.cc:288] T d0a06e419c6c4f3db6705cb2377e9336 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:14:42.784163 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0a06e419c6c4f3db6705cb2377e9336" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c" is_pre_election: true
I20240503 08:14:42.784354 30895 raft_consensus.cc:2454] T d0a06e419c6c4f3db6705cb2377e9336 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
I20240503 08:14:42.785271 27936 leader_election.cc:302] T d0a06e419c6c4f3db6705cb2377e9336 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, ce1496a47d084365ad786cd09e472a2c; no voters:
I20240503 08:14:42.785537 12963 raft_consensus.cc:2781] T d0a06e419c6c4f3db6705cb2377e9336 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:14:42.785621 12963 raft_consensus.cc:482] T d0a06e419c6c4f3db6705cb2377e9336 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:42.785682 12963 raft_consensus.cc:3037] T d0a06e419c6c4f3db6705cb2377e9336 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:14:42.786042 27938 leader_election.cc:334] T d0a06e419c6c4f3db6705cb2377e9336 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:42.788142 12963 raft_consensus.cc:504] T d0a06e419c6c4f3db6705cb2377e9336 P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:14:42.788735 12963 leader_election.cc:288] T d0a06e419c6c4f3db6705cb2377e9336 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 election: Requested vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
I20240503 08:14:42.789261 30892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0a06e419c6c4f3db6705cb2377e9336" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce1496a47d084365ad786cd09e472a2c"
I20240503 08:14:42.789428 30892 raft_consensus.cc:3037] T d0a06e419c6c4f3db6705cb2377e9336 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:14:42.790616 27938 leader_election.cc:334] T d0a06e419c6c4f3db6705cb2377e9336 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:42.791918 30892 raft_consensus.cc:2454] T d0a06e419c6c4f3db6705cb2377e9336 P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 2.
I20240503 08:14:42.792261 27936 leader_election.cc:302] T d0a06e419c6c4f3db6705cb2377e9336 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 833582ff68c94269be364ebd187a75db, ce1496a47d084365ad786cd09e472a2c; no voters: 1b411c6bb25549368041773d7b710d48
I20240503 08:14:42.792479 12963 raft_consensus.cc:2781] T d0a06e419c6c4f3db6705cb2377e9336 P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:14:42.792585 12963 raft_consensus.cc:686] T d0a06e419c6c4f3db6705cb2377e9336 P 833582ff68c94269be364ebd187a75db [term 2 LEADER]: Becoming Leader. State: Replica: 833582ff68c94269be364ebd187a75db, State: Running, Role: LEADER
I20240503 08:14:42.792804 12963 consensus_queue.cc:237] T d0a06e419c6c4f3db6705cb2377e9336 P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:14:42.796627 12840 raft_consensus.cc:482] T bab5c25c31a64b578489b1006f631cc0 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:42.796810 12840 raft_consensus.cc:504] T bab5c25c31a64b578489b1006f631cc0 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:42.796897 27860 catalog_manager.cc:5484] T d0a06e419c6c4f3db6705cb2377e9336 P 833582ff68c94269be364ebd187a75db reported cstate change: term changed from 1 to 2, leader changed from 1b411c6bb25549368041773d7b710d48 (127.20.230.66) to 833582ff68c94269be364ebd187a75db (127.20.230.65). New cstate: current_term: 2 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:14:42.797551 12840 leader_election.cc:288] T bab5c25c31a64b578489b1006f631cc0 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:14:42.798166 12830 raft_consensus.cc:482] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:42.798296 12830 raft_consensus.cc:504] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:14:42.798843 12830 leader_election.cc:288] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:14:42.802078 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab5c25c31a64b578489b1006f631cc0" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
W20240503 08:14:42.803385 28328 leader_election.cc:334] T bab5c25c31a64b578489b1006f631cc0 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:42.803504 28328 leader_election.cc:302] T bab5c25c31a64b578489b1006f631cc0 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d09a11a03989421b9f28019dc1c14cc2; no voters: 1b411c6bb25549368041773d7b710d48, ced21ca983b749d0b4d4f6a0e2426bf8
W20240503 08:14:42.803496 31096 leader_election.cc:334] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:42.803766 13061 raft_consensus.cc:2726] T bab5c25c31a64b578489b1006f631cc0 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 08:14:42.804076 31095 leader_election.cc:334] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:42.804172 31095 leader_election.cc:302] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b00f244c69df401997ea99d492218a98; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:42.804733 12654 raft_consensus.cc:2726] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:14:42.804869 28327 leader_election.cc:334] T bab5c25c31a64b578489b1006f631cc0 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:42.851119 13119 raft_consensus.cc:482] T 7a101bee3af34166848ee7fa7c733892 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:42.851261 13119 raft_consensus.cc:504] T 7a101bee3af34166848ee7fa7c733892 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:14:42.852002 13119 leader_election.cc:288] T 7a101bee3af34166848ee7fa7c733892 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:14:42.852761 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a101bee3af34166848ee7fa7c733892" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:14:42.852979 28391 raft_consensus.cc:2454] T 7a101bee3af34166848ee7fa7c733892 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 1.
I20240503 08:14:42.853521 30830 leader_election.cc:302] T 7a101bee3af34166848ee7fa7c733892 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c, d09a11a03989421b9f28019dc1c14cc2; no voters:
I20240503 08:14:42.854104 13119 raft_consensus.cc:2781] T 7a101bee3af34166848ee7fa7c733892 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:14:42.854207 13119 raft_consensus.cc:482] T 7a101bee3af34166848ee7fa7c733892 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:42.854292 13119 raft_consensus.cc:3037] T 7a101bee3af34166848ee7fa7c733892 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:42.856927 13119 raft_consensus.cc:504] T 7a101bee3af34166848ee7fa7c733892 P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:14:42.857481 13119 leader_election.cc:288] T 7a101bee3af34166848ee7fa7c733892 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 election: Requested vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
W20240503 08:14:42.858026 30829 leader_election.cc:334] T 7a101bee3af34166848ee7fa7c733892 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:42.859282 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a101bee3af34166848ee7fa7c733892" candidate_uuid: "ce1496a47d084365ad786cd09e472a2c" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:14:42.859493 28391 raft_consensus.cc:3037] T 7a101bee3af34166848ee7fa7c733892 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:14:42.861526 30829 leader_election.cc:334] T 7a101bee3af34166848ee7fa7c733892 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:42.862764 28391 raft_consensus.cc:2454] T 7a101bee3af34166848ee7fa7c733892 P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1496a47d084365ad786cd09e472a2c in term 2.
I20240503 08:14:42.863090 30830 leader_election.cc:302] T 7a101bee3af34166848ee7fa7c733892 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c, d09a11a03989421b9f28019dc1c14cc2; no voters: 1b411c6bb25549368041773d7b710d48
I20240503 08:14:42.863330 12941 raft_consensus.cc:2781] T 7a101bee3af34166848ee7fa7c733892 P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:14:42.863471 12941 raft_consensus.cc:686] T 7a101bee3af34166848ee7fa7c733892 P ce1496a47d084365ad786cd09e472a2c [term 2 LEADER]: Becoming Leader. State: Replica: ce1496a47d084365ad786cd09e472a2c, State: Running, Role: LEADER
I20240503 08:14:42.863718 12941 consensus_queue.cc:237] T 7a101bee3af34166848ee7fa7c733892 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:14:42.867060 12716 raft_consensus.cc:482] T 35c62760e1de4c2f88d06b97d0286c4c P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:42.867197 12716 raft_consensus.cc:504] T 35c62760e1de4c2f88d06b97d0286c4c P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:14:42.867789 12716 leader_election.cc:288] T 35c62760e1de4c2f88d06b97d0286c4c P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:14:42.868139 27856 catalog_manager.cc:5484] T 7a101bee3af34166848ee7fa7c733892 P ce1496a47d084365ad786cd09e472a2c reported cstate change: term changed from 1 to 2, leader changed from 1b411c6bb25549368041773d7b710d48 (127.20.230.66) to ce1496a47d084365ad786cd09e472a2c (127.20.230.70). New cstate: current_term: 2 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:14:42.868952 28000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35c62760e1de4c2f88d06b97d0286c4c" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
W20240503 08:14:42.870429 30696 leader_election.cc:334] T 35c62760e1de4c2f88d06b97d0286c4c P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:42.870530 30696 leader_election.cc:302] T 35c62760e1de4c2f88d06b97d0286c4c P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b; no voters: 1b411c6bb25549368041773d7b710d48, 833582ff68c94269be364ebd187a75db
I20240503 08:14:42.872898 12716 raft_consensus.cc:2726] T 35c62760e1de4c2f88d06b97d0286c4c P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:14:42.885913 12962 raft_consensus.cc:482] T ab35ebabd6fe45fab890739a800794b1 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:42.886056 12962 raft_consensus.cc:504] T ab35ebabd6fe45fab890739a800794b1 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:14:42.886718 12962 leader_election.cc:288] T ab35ebabd6fe45fab890739a800794b1 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:14:42.888586 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab35ebabd6fe45fab890739a800794b1" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:14:42.888824 31027 raft_consensus.cc:2454] T ab35ebabd6fe45fab890739a800794b1 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
W20240503 08:14:42.890023 27938 leader_election.cc:334] T ab35ebabd6fe45fab890739a800794b1 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:42.891849 27937 leader_election.cc:302] T ab35ebabd6fe45fab890739a800794b1 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 833582ff68c94269be364ebd187a75db, ced21ca983b749d0b4d4f6a0e2426bf8; no voters: 1b411c6bb25549368041773d7b710d48
I20240503 08:14:42.892223 12963 raft_consensus.cc:2781] T ab35ebabd6fe45fab890739a800794b1 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:14:42.892323 12963 raft_consensus.cc:482] T ab35ebabd6fe45fab890739a800794b1 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:42.892423 12963 raft_consensus.cc:3037] T ab35ebabd6fe45fab890739a800794b1 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:42.895160 12963 raft_consensus.cc:504] T ab35ebabd6fe45fab890739a800794b1 P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:14:42.895742 12963 leader_election.cc:288] T ab35ebabd6fe45fab890739a800794b1 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 election: Requested vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:14:42.897478 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab35ebabd6fe45fab890739a800794b1" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8"
I20240503 08:14:42.897703 31027 raft_consensus.cc:3037] T ab35ebabd6fe45fab890739a800794b1 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:42.900349 31027 raft_consensus.cc:2454] T ab35ebabd6fe45fab890739a800794b1 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 2.
I20240503 08:14:42.903577 12962 raft_consensus.cc:482] T 099e37089b2e42b4839e681f8f37365d P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:42.903717 12962 raft_consensus.cc:504] T 099e37089b2e42b4839e681f8f37365d P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:14:42.904503 12962 leader_election.cc:288] T 099e37089b2e42b4839e681f8f37365d P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:14:42.905447 27937 leader_election.cc:302] T ab35ebabd6fe45fab890739a800794b1 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
W20240503 08:14:42.906147 27938 leader_election.cc:334] T ab35ebabd6fe45fab890739a800794b1 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:42.907112 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099e37089b2e42b4839e681f8f37365d" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:14:42.909694 12962 raft_consensus.cc:2781] T ab35ebabd6fe45fab890739a800794b1 P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:14:42.909844 12962 raft_consensus.cc:686] T ab35ebabd6fe45fab890739a800794b1 P 833582ff68c94269be364ebd187a75db [term 2 LEADER]: Becoming Leader. State: Replica: 833582ff68c94269be364ebd187a75db, State: Running, Role: LEADER
I20240503 08:14:42.910123 12962 consensus_queue.cc:237] T ab35ebabd6fe45fab890739a800794b1 P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
W20240503 08:14:42.911316 27937 leader_election.cc:334] T 099e37089b2e42b4839e681f8f37365d P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:42.911439 27937 leader_election.cc:302] T 099e37089b2e42b4839e681f8f37365d P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 833582ff68c94269be364ebd187a75db; no voters: 6d1d7de19459435fbde1999e3dfd226b, aae860d7ae2b4002bf7794c278f9cc56
W20240503 08:14:42.911690 27938 leader_election.cc:334] T 099e37089b2e42b4839e681f8f37365d P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:42.911830 12963 raft_consensus.cc:2726] T 099e37089b2e42b4839e681f8f37365d P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 08:14:42.917119 30960 consensus_peers.cc:485] T 41aa0e288ca4469ba34a07ff159ace91 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 08:14:42.921063 27857 catalog_manager.cc:5484] T ab35ebabd6fe45fab890739a800794b1 P 833582ff68c94269be364ebd187a75db reported cstate change: term changed from 1 to 2, leader changed from 1b411c6bb25549368041773d7b710d48 (127.20.230.66) to 833582ff68c94269be364ebd187a75db (127.20.230.65). New cstate: current_term: 2 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } }
I20240503 08:14:42.923743 12664 raft_consensus.cc:482] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:42.923897 12664 raft_consensus.cc:504] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:42.924538 12664 leader_election.cc:288] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
W20240503 08:14:42.926656 30960 leader_election.cc:334] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:42.929744 12958 raft_consensus.cc:482] T 099e37089b2e42b4839e681f8f37365d P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:42.929910 12958 raft_consensus.cc:504] T 099e37089b2e42b4839e681f8f37365d P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:14:42.930685 12958 leader_election.cc:288] T 099e37089b2e42b4839e681f8f37365d P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:14:42.930826 12958 raft_consensus.cc:482] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
W20240503 08:14:42.930814 30961 leader_election.cc:334] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:42.930922 30961 leader_election.cc:302] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:42.930922 12958 raft_consensus.cc:504] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:14:42.931372 12664 raft_consensus.cc:2726] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240503 08:14:42.931550 12958 leader_election.cc:288] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
W20240503 08:14:42.932164 30696 leader_election.cc:334] T 099e37089b2e42b4839e681f8f37365d P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:42.932214 12840 raft_consensus.cc:482] T 7083414bc16c4d499c97b93ca4e3f631 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:42.932376 12840 raft_consensus.cc:504] T 7083414bc16c4d499c97b93ca4e3f631 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:14:42.932484 28000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099e37089b2e42b4839e681f8f37365d" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:14:42.932698 28000 raft_consensus.cc:2454] T 099e37089b2e42b4839e681f8f37365d P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:14:42.933136 12840 leader_election.cc:288] T 7083414bc16c4d499c97b93ca4e3f631 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
W20240503 08:14:42.934775 30696 leader_election.cc:334] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:42.935801 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7083414bc16c4d499c97b93ca4e3f631" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
W20240503 08:14:42.937776 28328 leader_election.cc:334] T 7083414bc16c4d499c97b93ca4e3f631 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:42.937911 28328 leader_election.cc:302] T 7083414bc16c4d499c97b93ca4e3f631 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d09a11a03989421b9f28019dc1c14cc2; no voters: 1b411c6bb25549368041773d7b710d48, 6d1d7de19459435fbde1999e3dfd226b
I20240503 08:14:42.938736 12840 raft_consensus.cc:2726] T 7083414bc16c4d499c97b93ca4e3f631 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 08:14:42.939145 30695 leader_election.cc:334] T 099e37089b2e42b4839e681f8f37365d P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:42.939245 30695 leader_election.cc:302] T 099e37089b2e42b4839e681f8f37365d P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, 833582ff68c94269be364ebd187a75db; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
W20240503 08:14:42.939559 30695 leader_election.cc:334] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:42.939631 30695 leader_election.cc:302] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:42.939996 12958 raft_consensus.cc:2726] T 099e37089b2e42b4839e681f8f37365d P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:14:42.940146 12958 raft_consensus.cc:2726] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 08:14:42.953012 12664 raft_consensus.cc:482] T 978bf5783b0e47e3bfb30441adc600ae P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:42.953169 12664 raft_consensus.cc:504] T 978bf5783b0e47e3bfb30441adc600ae P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:42.953631 12840 raft_consensus.cc:482] T d94f5824a71f454f8a1bfa13dd032684 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:42.953785 12664 leader_election.cc:288] T 978bf5783b0e47e3bfb30441adc600ae P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:14:42.954777 12840 raft_consensus.cc:504] T d94f5824a71f454f8a1bfa13dd032684 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:42.955582 12840 leader_election.cc:288] T d94f5824a71f454f8a1bfa13dd032684 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
W20240503 08:14:42.956920 28327 leader_election.cc:334] T d94f5824a71f454f8a1bfa13dd032684 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:42.958597 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d94f5824a71f454f8a1bfa13dd032684" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
W20240503 08:14:42.959259 30961 leader_election.cc:334] T 978bf5783b0e47e3bfb30441adc600ae P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
W20240503 08:14:42.959481 30960 leader_election.cc:334] T 978bf5783b0e47e3bfb30441adc600ae P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:42.959595 30960 leader_election.cc:302] T 978bf5783b0e47e3bfb30441adc600ae P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:42.959820 12664 raft_consensus.cc:2726] T 978bf5783b0e47e3bfb30441adc600ae P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:14:42.959949 28328 leader_election.cc:334] T d94f5824a71f454f8a1bfa13dd032684 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:42.960058 28328 leader_election.cc:302] T d94f5824a71f454f8a1bfa13dd032684 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d09a11a03989421b9f28019dc1c14cc2; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:42.961383 12840 raft_consensus.cc:2726] T d94f5824a71f454f8a1bfa13dd032684 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:14:42.972018 12962 raft_consensus.cc:482] T 35c62760e1de4c2f88d06b97d0286c4c P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:42.972185 12962 raft_consensus.cc:504] T 35c62760e1de4c2f88d06b97d0286c4c P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:14:42.972831 12962 leader_election.cc:288] T 35c62760e1de4c2f88d06b97d0286c4c P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:14:42.973337 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35c62760e1de4c2f88d06b97d0286c4c" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:14:42.973534 30761 raft_consensus.cc:2454] T 35c62760e1de4c2f88d06b97d0286c4c P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
W20240503 08:14:42.974402 27938 leader_election.cc:334] T 35c62760e1de4c2f88d06b97d0286c4c P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:42.974642 27938 leader_election.cc:302] T 35c62760e1de4c2f88d06b97d0286c4c P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, 833582ff68c94269be364ebd187a75db; no voters: 1b411c6bb25549368041773d7b710d48
I20240503 08:14:42.974901 12962 raft_consensus.cc:2781] T 35c62760e1de4c2f88d06b97d0286c4c P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:14:42.975005 12962 raft_consensus.cc:482] T 35c62760e1de4c2f88d06b97d0286c4c P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:42.975100 12962 raft_consensus.cc:3037] T 35c62760e1de4c2f88d06b97d0286c4c P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:42.977738 12962 raft_consensus.cc:504] T 35c62760e1de4c2f88d06b97d0286c4c P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:14:42.978350 12962 leader_election.cc:288] T 35c62760e1de4c2f88d06b97d0286c4c P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 election: Requested vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:14:42.979036 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35c62760e1de4c2f88d06b97d0286c4c" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b"
I20240503 08:14:42.979225 30761 raft_consensus.cc:3037] T 35c62760e1de4c2f88d06b97d0286c4c P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:14:42.979619 27938 leader_election.cc:334] T 35c62760e1de4c2f88d06b97d0286c4c P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:42.981740 30761 raft_consensus.cc:2454] T 35c62760e1de4c2f88d06b97d0286c4c P 6d1d7de19459435fbde1999e3dfd226b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 2.
I20240503 08:14:42.982522 27938 leader_election.cc:302] T 35c62760e1de4c2f88d06b97d0286c4c P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, 833582ff68c94269be364ebd187a75db; no voters: 1b411c6bb25549368041773d7b710d48
I20240503 08:14:42.983620 12962 raft_consensus.cc:2781] T 35c62760e1de4c2f88d06b97d0286c4c P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:14:42.983768 12962 raft_consensus.cc:686] T 35c62760e1de4c2f88d06b97d0286c4c P 833582ff68c94269be364ebd187a75db [term 2 LEADER]: Becoming Leader. State: Replica: 833582ff68c94269be364ebd187a75db, State: Running, Role: LEADER
I20240503 08:14:42.984014 12962 consensus_queue.cc:237] T 35c62760e1de4c2f88d06b97d0286c4c P 833582ff68c94269be364ebd187a75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:14:42.989015 27859 catalog_manager.cc:5484] T 35c62760e1de4c2f88d06b97d0286c4c P 833582ff68c94269be364ebd187a75db reported cstate change: term changed from 1 to 2, leader changed from 1b411c6bb25549368041773d7b710d48 (127.20.230.66) to 833582ff68c94269be364ebd187a75db (127.20.230.65). New cstate: current_term: 2 leader_uuid: "833582ff68c94269be364ebd187a75db" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } }
I20240503 08:14:42.992341 12716 raft_consensus.cc:482] T 7083414bc16c4d499c97b93ca4e3f631 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:42.992480 12716 raft_consensus.cc:504] T 7083414bc16c4d499c97b93ca4e3f631 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:14:42.993003 12716 leader_election.cc:288] T 7083414bc16c4d499c97b93ca4e3f631 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:14:42.993489 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7083414bc16c4d499c97b93ca4e3f631" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:14:42.993702 28391 raft_consensus.cc:2454] T 7083414bc16c4d499c97b93ca4e3f631 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:14:42.994164 30697 leader_election.cc:302] T 7083414bc16c4d499c97b93ca4e3f631 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, d09a11a03989421b9f28019dc1c14cc2; no voters:
W20240503 08:14:42.994414 30696 leader_election.cc:334] T 7083414bc16c4d499c97b93ca4e3f631 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:42.994470 12716 raft_consensus.cc:2781] T 7083414bc16c4d499c97b93ca4e3f631 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:14:42.994585 12716 raft_consensus.cc:482] T 7083414bc16c4d499c97b93ca4e3f631 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:42.994601 13119 raft_consensus.cc:482] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:42.994678 12716 raft_consensus.cc:3037] T 7083414bc16c4d499c97b93ca4e3f631 P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:42.994743 13119 raft_consensus.cc:504] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:42.995352 13119 leader_election.cc:288] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:14:42.998004 12716 raft_consensus.cc:504] T 7083414bc16c4d499c97b93ca4e3f631 P 6d1d7de19459435fbde1999e3dfd226b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
W20240503 08:14:42.999644 30829 leader_election.cc:334] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:43.000018 12716 leader_election.cc:288] T 7083414bc16c4d499c97b93ca4e3f631 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 2 election: Requested vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:14:43.000476 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7083414bc16c4d499c97b93ca4e3f631" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2"
I20240503 08:14:43.000650 28391 raft_consensus.cc:3037] T 7083414bc16c4d499c97b93ca4e3f631 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:14:43.000772 30828 leader_election.cc:334] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:43.000877 30828 leader_election.cc:302] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:43.001252 13119 raft_consensus.cc:2726] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:14:43.001722 30696 leader_election.cc:334] T 7083414bc16c4d499c97b93ca4e3f631 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:43.003286 28391 raft_consensus.cc:2454] T 7083414bc16c4d499c97b93ca4e3f631 P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 2.
I20240503 08:14:43.004200 30697 leader_election.cc:302] T 7083414bc16c4d499c97b93ca4e3f631 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, d09a11a03989421b9f28019dc1c14cc2; no voters: 1b411c6bb25549368041773d7b710d48
I20240503 08:14:43.004572 12716 raft_consensus.cc:2781] T 7083414bc16c4d499c97b93ca4e3f631 P 6d1d7de19459435fbde1999e3dfd226b [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:14:43.004701 12716 raft_consensus.cc:686] T 7083414bc16c4d499c97b93ca4e3f631 P 6d1d7de19459435fbde1999e3dfd226b [term 2 LEADER]: Becoming Leader. State: Replica: 6d1d7de19459435fbde1999e3dfd226b, State: Running, Role: LEADER
I20240503 08:14:43.004961 12716 consensus_queue.cc:237] T 7083414bc16c4d499c97b93ca4e3f631 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:14:43.013593 12962 raft_consensus.cc:482] T fe4d26ee022b424195e836834bd2f051 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:43.013736 12962 raft_consensus.cc:504] T fe4d26ee022b424195e836834bd2f051 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:43.014341 12962 leader_election.cc:288] T fe4d26ee022b424195e836834bd2f051 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:14:43.017442 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4d26ee022b424195e836834bd2f051" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:14:43.018776 27859 catalog_manager.cc:5484] T 7083414bc16c4d499c97b93ca4e3f631 P 6d1d7de19459435fbde1999e3dfd226b reported cstate change: term changed from 1 to 2, leader changed from 1b411c6bb25549368041773d7b710d48 (127.20.230.66) to 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69). New cstate: current_term: 2 leader_uuid: "6d1d7de19459435fbde1999e3dfd226b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } }
W20240503 08:14:43.020192 27938 leader_election.cc:334] T fe4d26ee022b424195e836834bd2f051 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:43.021157 27937 leader_election.cc:302] T fe4d26ee022b424195e836834bd2f051 P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 833582ff68c94269be364ebd187a75db; no voters: 1b411c6bb25549368041773d7b710d48, ced21ca983b749d0b4d4f6a0e2426bf8
I20240503 08:14:43.021584 12963 raft_consensus.cc:2726] T fe4d26ee022b424195e836834bd2f051 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:14:43.027019 12664 raft_consensus.cc:482] T d94f5824a71f454f8a1bfa13dd032684 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:43.027184 12664 raft_consensus.cc:504] T d94f5824a71f454f8a1bfa13dd032684 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:43.027966 12664 leader_election.cc:288] T d94f5824a71f454f8a1bfa13dd032684 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:14:43.029022 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d94f5824a71f454f8a1bfa13dd032684" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
W20240503 08:14:43.030611 30961 leader_election.cc:334] T d94f5824a71f454f8a1bfa13dd032684 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
W20240503 08:14:43.031774 30960 leader_election.cc:334] T d94f5824a71f454f8a1bfa13dd032684 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:43.031904 30960 leader_election.cc:302] T d94f5824a71f454f8a1bfa13dd032684 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:43.032199 12963 raft_consensus.cc:482] T df90a0d9d338416b81e253fa23542cca P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:43.032344 12963 raft_consensus.cc:504] T df90a0d9d338416b81e253fa23542cca P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:14:43.032954 12963 leader_election.cc:288] T df90a0d9d338416b81e253fa23542cca P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:14:43.033414 12654 raft_consensus.cc:482] T 2dce5b2aaea045ccb7424cbbd44576c2 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:43.033537 12654 raft_consensus.cc:504] T 2dce5b2aaea045ccb7424cbbd44576c2 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:43.034730 12654 leader_election.cc:288] T 2dce5b2aaea045ccb7424cbbd44576c2 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:14:43.036226 12664 raft_consensus.cc:2726] T d94f5824a71f454f8a1bfa13dd032684 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 08:14:43.036492 27938 leader_election.cc:334] T df90a0d9d338416b81e253fa23542cca P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:43.036769 28391 raft_consensus.cc:2454] T d94f5824a71f454f8a1bfa13dd032684 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 1.
W20240503 08:14:43.040936 31095 leader_election.cc:334] T 2dce5b2aaea045ccb7424cbbd44576c2 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:43.042235 31096 leader_election.cc:334] T 2dce5b2aaea045ccb7424cbbd44576c2 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:43.042347 31096 leader_election.cc:302] T 2dce5b2aaea045ccb7424cbbd44576c2 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b00f244c69df401997ea99d492218a98; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:43.042541 12654 raft_consensus.cc:2726] T 2dce5b2aaea045ccb7424cbbd44576c2 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 08:14:43.043460 27937 leader_election.cc:334] T df90a0d9d338416b81e253fa23542cca P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:43.043588 27937 leader_election.cc:302] T df90a0d9d338416b81e253fa23542cca P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 833582ff68c94269be364ebd187a75db; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:43.043844 12963 raft_consensus.cc:2726] T df90a0d9d338416b81e253fa23542cca P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:14:43.053613 30742 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 08:14:43.063979 12664 raft_consensus.cc:482] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:43.064145 12664 raft_consensus.cc:504] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:43.064795 12664 leader_election.cc:288] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
W20240503 08:14:43.065987 30961 leader_election.cc:334] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
W20240503 08:14:43.066247 30960 leader_election.cc:334] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:43.066354 30960 leader_election.cc:302] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:43.066612 12664 raft_consensus.cc:2726] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 08:14:43.067312 13061 raft_consensus.cc:482] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:43.067477 13061 raft_consensus.cc:504] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:43.068171 13061 leader_election.cc:288] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
W20240503 08:14:43.070917 28327 leader_election.cc:334] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:43.075649 12664 raft_consensus.cc:482] T fe4d26ee022b424195e836834bd2f051 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:43.075800 12664 raft_consensus.cc:504] T fe4d26ee022b424195e836834bd2f051 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:43.076345 12664 leader_election.cc:288] T fe4d26ee022b424195e836834bd2f051 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:14:43.077032 28000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4d26ee022b424195e836834bd2f051" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:14:43.077250 28000 raft_consensus.cc:2454] T fe4d26ee022b424195e836834bd2f051 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 1.
W20240503 08:14:43.077673 28328 leader_election.cc:334] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:43.077682 30962 leader_election.cc:302] T fe4d26ee022b424195e836834bd2f051 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833582ff68c94269be364ebd187a75db, ced21ca983b749d0b4d4f6a0e2426bf8; no voters:
I20240503 08:14:43.077816 28328 leader_election.cc:302] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d09a11a03989421b9f28019dc1c14cc2; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:43.078084 13061 raft_consensus.cc:2726] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240503 08:14:43.078593 12664 raft_consensus.cc:2781] T fe4d26ee022b424195e836834bd2f051 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:14:43.078691 12664 raft_consensus.cc:482] T fe4d26ee022b424195e836834bd2f051 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:43.078780 12664 raft_consensus.cc:3037] T fe4d26ee022b424195e836834bd2f051 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:43.081348 12664 raft_consensus.cc:504] T fe4d26ee022b424195e836834bd2f051 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
W20240503 08:14:43.083724 30961 leader_election.cc:334] T fe4d26ee022b424195e836834bd2f051 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:43.088493 12664 leader_election.cc:288] T fe4d26ee022b424195e836834bd2f051 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 election: Requested vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:14:43.088644 12664 raft_consensus.cc:482] T bab5c25c31a64b578489b1006f631cc0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:43.088774 12664 raft_consensus.cc:504] T bab5c25c31a64b578489b1006f631cc0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
W20240503 08:14:43.089306 30961 leader_election.cc:334] T fe4d26ee022b424195e836834bd2f051 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:43.090016 27983 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:14:43.091871 30874 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 08:14:43.103106 30828 consensus_peers.cc:485] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 08:14:43.106315 31141 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 08:14:43.108049 30960 leader_election.cc:334] T bab5c25c31a64b578489b1006f631cc0 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:43.108493 12664 leader_election.cc:288] T bab5c25c31a64b578489b1006f631cc0 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
W20240503 08:14:43.112035 30961 leader_election.cc:334] T bab5c25c31a64b578489b1006f631cc0 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:43.112171 30961 leader_election.cc:302] T bab5c25c31a64b578489b1006f631cc0 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:43.113437 28373 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 08:14:43.117167 13188 raft_consensus.cc:2726] T bab5c25c31a64b578489b1006f631cc0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 08:14:43.124131 30696 consensus_peers.cc:462] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:43.131650 28000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4d26ee022b424195e836834bd2f051" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db"
I20240503 08:14:43.131891 28000 raft_consensus.cc:3037] T fe4d26ee022b424195e836834bd2f051 P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:43.134564 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab5c25c31a64b578489b1006f631cc0" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:14:43.134773 28391 raft_consensus.cc:2454] T bab5c25c31a64b578489b1006f631cc0 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 1.
W20240503 08:14:43.135298 30828 consensus_peers.cc:462] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:43.140661 28000 raft_consensus.cc:2454] T fe4d26ee022b424195e836834bd2f051 P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 2.
I20240503 08:14:43.144189 30962 leader_election.cc:302] T fe4d26ee022b424195e836834bd2f051 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 833582ff68c94269be364ebd187a75db, ced21ca983b749d0b4d4f6a0e2426bf8; no voters: 1b411c6bb25549368041773d7b710d48
I20240503 08:14:43.144609 13187 raft_consensus.cc:2781] T fe4d26ee022b424195e836834bd2f051 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:14:43.144850 13187 raft_consensus.cc:686] T fe4d26ee022b424195e836834bd2f051 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 LEADER]: Becoming Leader. State: Replica: ced21ca983b749d0b4d4f6a0e2426bf8, State: Running, Role: LEADER
I20240503 08:14:43.146333 31006 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:14:43.145195 13187 consensus_queue.cc:237] T fe4d26ee022b424195e836834bd2f051 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:43.182915 27859 catalog_manager.cc:5484] T fe4d26ee022b424195e836834bd2f051 P ced21ca983b749d0b4d4f6a0e2426bf8 reported cstate change: term changed from 1 to 2, leader changed from 1b411c6bb25549368041773d7b710d48 (127.20.230.66) to ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71). New cstate: current_term: 2 leader_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } }
W20240503 08:14:43.237352 27938 consensus_peers.cc:485] T d0a06e419c6c4f3db6705cb2377e9336 P 833582ff68c94269be364ebd187a75db -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:43.254740 30829 consensus_peers.cc:485] T b52e4b59153d40378e2a37809b04a2c8 P ce1496a47d084365ad786cd09e472a2c -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:43.266122 27937 consensus_peers.cc:485] T f1b2f3b8111649a7b5725647b8251698 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:14:43.288331 31025 raft_consensus.cc:1261] T b52e4b59153d40378e2a37809b04a2c8 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 08:14:43.288975 13119 consensus_queue.cc:1035] T b52e4b59153d40378e2a37809b04a2c8 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:43.294739 30892 raft_consensus.cc:1261] T d0a06e419c6c4f3db6705cb2377e9336 P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 08:14:43.295627 13189 consensus_queue.cc:1035] T d0a06e419c6c4f3db6705cb2377e9336 P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:14:43.323032 28391 raft_consensus.cc:1261] T 7a101bee3af34166848ee7fa7c733892 P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:14:43.323534 11989 consensus_queue.cc:1035] T 7a101bee3af34166848ee7fa7c733892 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 08:14:43.329995 27937 consensus_peers.cc:485] T 8cc6ae30aa60495d8a4364a11d5bef79 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:14:43.338742 30829 consensus_peers.cc:485] T 7a101bee3af34166848ee7fa7c733892 P ce1496a47d084365ad786cd09e472a2c -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:43.357384 31027 raft_consensus.cc:1261] T ab35ebabd6fe45fab890739a800794b1 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 08:14:43.357864 12962 consensus_queue.cc:1035] T ab35ebabd6fe45fab890739a800794b1 P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 08:14:43.366894 27938 consensus_peers.cc:485] T ab35ebabd6fe45fab890739a800794b1 P 833582ff68c94269be364ebd187a75db -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:43.462131 30695 consensus_peers.cc:485] T 2222ff7fb5184d6c86010e1475d6c1f7 P 6d1d7de19459435fbde1999e3dfd226b -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 08:14:43.484624 30761 raft_consensus.cc:1261] T 35c62760e1de4c2f88d06b97d0286c4c P 6d1d7de19459435fbde1999e3dfd226b [term 2 FOLLOWER]: Refusing update from remote peer 833582ff68c94269be364ebd187a75db: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:14:43.485175 12962 consensus_queue.cc:1035] T 35c62760e1de4c2f88d06b97d0286c4c P 833582ff68c94269be364ebd187a75db [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:14:43.500746 27938 consensus_peers.cc:485] T 35c62760e1de4c2f88d06b97d0286c4c P 833582ff68c94269be364ebd187a75db -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:43.509886 30742 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 08:14:43.540792 27983 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 08:14:43.541641 30696 consensus_peers.cc:485] T 7083414bc16c4d499c97b93ca4e3f631 P 6d1d7de19459435fbde1999e3dfd226b -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:43.542762 31141 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 08:14:43.552315 28373 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 08:14:43.555912 31006 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 08:14:43.557391 30696 consensus_peers.cc:485] T 91c573181144456aa7e27a3026a639c4 P 6d1d7de19459435fbde1999e3dfd226b -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:43.558539 30828 consensus_peers.cc:485] T 0f25688be4e74d31aaab85b7a4329a34 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:14:43.565492 30874 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 08:14:43.602108 28391 raft_consensus.cc:1261] T 7083414bc16c4d499c97b93ca4e3f631 P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Refusing update from remote peer 6d1d7de19459435fbde1999e3dfd226b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:14:43.602809 12958 consensus_queue.cc:1035] T 7083414bc16c4d499c97b93ca4e3f631 P 6d1d7de19459435fbde1999e3dfd226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 08:14:43.605710 28327 consensus_peers.cc:485] T cdad6ef53b6f413ebfda67f9b42c9c23 P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:14:43.616308 30828 consensus_peers.cc:462] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:43.656105 30696 consensus_peers.cc:462] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
W20240503 08:14:43.678012 27937 consensus_peers.cc:485] T 08fb64977309479b831635344969c662 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:14:43.712460 30961 consensus_peers.cc:485] T fe4d26ee022b424195e836834bd2f051 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:43.718209 31095 consensus_peers.cc:485] T 6a96965691644db8b9461d146536a026 P b00f244c69df401997ea99d492218a98 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 08:14:43.721031 28000 raft_consensus.cc:1261] T fe4d26ee022b424195e836834bd2f051 P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Refusing update from remote peer ced21ca983b749d0b4d4f6a0e2426bf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:14:43.722369 12664 consensus_queue.cc:1035] T fe4d26ee022b424195e836834bd2f051 P ced21ca983b749d0b4d4f6a0e2426bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 08:14:43.747520 28327 consensus_peers.cc:485] T f46393257ce9471891698c36a07546ad P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:14:43.826639 31095 consensus_peers.cc:485] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:14:43.832706 31095 consensus_peers.cc:485] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:14:43.852213 27937 consensus_peers.cc:485] T ca38671a760d4534ab903579e0487772 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:14:43.890764 30828 consensus_peers.cc:485] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:14:43.895299 27937 consensus_peers.cc:485] T 46d27fa12990405cae5f94976ac7a014 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:14:43.932106 30829 consensus_peers.cc:485] T d0657f27fd474a41b0c33db55989daa3 P ce1496a47d084365ad786cd09e472a2c -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:43.969245 30960 consensus_peers.cc:485] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:14:43.981200 30742 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 08:14:43.993829 30695 consensus_peers.cc:485] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:14:44.001735 27937 consensus_peers.cc:485] T c8ca21ffe16e41ab80f5693dc4dbb682 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:14:44.005861 28328 consensus_peers.cc:485] T 720f36100d794158be13ff02595e6ddf P d09a11a03989421b9f28019dc1c14cc2 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:14:44.006527 31006 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 08:14:44.027628 31141 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 08:14:44.034673 27938 consensus_peers.cc:485] T 45f47478f49149b895060791655503ae P 833582ff68c94269be364ebd187a75db -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:14:44.046669 27983 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:14:44.058724 30874 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 08:14:44.080554 28373 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 08:14:44.098021 27937 consensus_peers.cc:485] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:14:44.114164 30829 consensus_peers.cc:485] T 62ca502a089f4a37999d2252776b741b P ce1496a47d084365ad786cd09e472a2c -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:44.129144 30828 consensus_peers.cc:485] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:14:44.145540 30829 consensus_peers.cc:485] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:44.145527 27938 consensus_peers.cc:485] T 417cedc3fa5c4a85a7a130005291d64b P 833582ff68c94269be364ebd187a75db -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:44.157322 28328 consensus_peers.cc:485] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:44.159310 31096 consensus_peers.cc:485] T 23fc18f9417148c2a6d451bcca395338 P b00f244c69df401997ea99d492218a98 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:44.161181 28327 consensus_peers.cc:485] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:14:44.165341 30828 consensus_peers.cc:462] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:44.169678 28328 consensus_peers.cc:485] T 1fca3d7f4b734eceb210e1186478eb72 P d09a11a03989421b9f28019dc1c14cc2 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:44.175081 30696 consensus_peers.cc:485] T 6c2824aa878845e098f600f1fcfea087 P 6d1d7de19459435fbde1999e3dfd226b -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:44.178900 30961 consensus_peers.cc:485] T 74f43feb3bc342a18723bc5a143bdd99 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:44.195957 27938 consensus_peers.cc:485] T c68032ba399547e3bd272f424927fbce P 833582ff68c94269be364ebd187a75db -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:44.203287 30961 consensus_peers.cc:485] T 50e4d2c569344dde915eec12f6f3d747 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:44.232616 27938 consensus_peers.cc:485] T 0ceedfb43869465bb63931c39c28583a P 833582ff68c94269be364ebd187a75db -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:44.235394 30696 consensus_peers.cc:485] T 607a503b64d8440a81cf66ab158ae273 P 6d1d7de19459435fbde1999e3dfd226b -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:44.235989 30696 consensus_peers.cc:462] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
W20240503 08:14:44.263100 28328 consensus_peers.cc:485] T 8bcbc022caa94bb08a2db4de9b060557 P d09a11a03989421b9f28019dc1c14cc2 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:44.263633 27937 consensus_peers.cc:485] T e0b904461c1c44ed93e1a227d93498eb P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:14:44.293484 30961 consensus_peers.cc:485] T 7df30eb244914dcabc6965324eced236 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:44.295033 28327 consensus_peers.cc:485] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:14:44.297093 30696 consensus_peers.cc:485] T 41e26e13355d4e0cb56bb526415a3998 P 6d1d7de19459435fbde1999e3dfd226b -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:44.311663 28328 consensus_peers.cc:485] T ab7b01c2e05a430a9873398268b1e41f P d09a11a03989421b9f28019dc1c14cc2 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:44.342253 28327 consensus_peers.cc:485] T bdfe68a127f043f08f04749843a38375 P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:14:44.352819 12654 raft_consensus.cc:482] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:44.352972 12654 raft_consensus.cc:504] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:14:44.353570 12654 leader_election.cc:288] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
W20240503 08:14:44.354431 27937 consensus_peers.cc:485] T 7c8c7659181040cabf20ae116fae78c8 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:14:44.355417 31095 leader_election.cc:334] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:44.356367 31096 leader_election.cc:334] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:44.356479 31096 leader_election.cc:302] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b00f244c69df401997ea99d492218a98; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:44.356676 12654 raft_consensus.cc:2726] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:14:44.358713 30828 consensus_peers.cc:485] T 045b3ce4b1d44f94a802b41a12e2704d P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:14:44.378283 27937 consensus_peers.cc:485] T 3d0498256b5f4b8fb1681759429f10ed P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:14:44.391206 28328 consensus_peers.cc:485] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:44.392887 30961 consensus_peers.cc:485] T 4ed8026daee840c6b31ca7d989ed25f0 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:44.396431 28328 consensus_peers.cc:485] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:44.422364 28327 consensus_peers.cc:485] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:14:44.426399 30696 consensus_peers.cc:485] T 2222ff7fb5184d6c86010e1475d6c1f7 P 6d1d7de19459435fbde1999e3dfd226b -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:44.428417 31095 consensus_peers.cc:485] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:14:44.428920 27938 consensus_peers.cc:485] T 3d0498256b5f4b8fb1681759429f10ed P 833582ff68c94269be364ebd187a75db -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:14:44.429713 30742 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 08:14:44.434872 12664 raft_consensus.cc:482] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:44.435030 12664 raft_consensus.cc:504] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:44.435638 12664 leader_election.cc:288] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
W20240503 08:14:44.437780 30960 leader_election.cc:334] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:44.443522 30961 leader_election.cc:334] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:44.443640 30961 leader_election.cc:302] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:44.443869 12664 raft_consensus.cc:2726] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 08:14:44.450186 31141 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 08:14:44.451392 28328 consensus_peers.cc:485] T 76d79d8be9e94e99a7ba51d903ca8a91 P d09a11a03989421b9f28019dc1c14cc2 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:44.452183 28327 consensus_peers.cc:485] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:14:44.466375 28373 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 08:14:44.471309 27983 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:14:44.472189 30874 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 08:14:44.477412 31006 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 08:14:44.504792 31095 consensus_peers.cc:485] T 2c595805729d48fd974d761ed8a60158 P b00f244c69df401997ea99d492218a98 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:14:44.520952 12716 raft_consensus.cc:482] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:44.521145 12716 raft_consensus.cc:504] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:14:44.521915 12716 leader_election.cc:288] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
W20240503 08:14:44.523377 30696 leader_election.cc:334] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
W20240503 08:14:44.525279 30695 leader_election.cc:334] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:44.525391 30695 leader_election.cc:302] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:44.525745 12716 raft_consensus.cc:2726] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:14:44.531610 27937 consensus_peers.cc:485] T 7f2743ab5fa24c9899b67ea52f48f9d8 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:14:44.555303 30828 consensus_peers.cc:485] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:14:44.561285 31096 consensus_peers.cc:485] T 2c595805729d48fd974d761ed8a60158 P b00f244c69df401997ea99d492218a98 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:44.573735 30960 consensus_peers.cc:485] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:14:44.574317 12664 raft_consensus.cc:482] T 658865ee3f8d424ca5cd35e12e6d7c77 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:44.574451 12664 raft_consensus.cc:504] T 658865ee3f8d424ca5cd35e12e6d7c77 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:44.575035 12664 leader_election.cc:288] T 658865ee3f8d424ca5cd35e12e6d7c77 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
W20240503 08:14:44.576753 30961 leader_election.cc:334] T 658865ee3f8d424ca5cd35e12e6d7c77 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
W20240503 08:14:44.577685 30960 leader_election.cc:334] T 658865ee3f8d424ca5cd35e12e6d7c77 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:44.577857 30960 leader_election.cc:302] T 658865ee3f8d424ca5cd35e12e6d7c77 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:44.579547 12664 raft_consensus.cc:2726] T 658865ee3f8d424ca5cd35e12e6d7c77 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 08:14:44.621379 30960 consensus_peers.cc:485] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:14:44.622135 28327 consensus_peers.cc:485] T 8425cfe5c6134a60852e12f9af55227b P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:14:44.628279 28327 consensus_peers.cc:485] T ba1fc93781714efb8eb7ec67c2fceb8a P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:14:44.634006 27938 consensus_peers.cc:485] T fa714cf4cc254703a2d356d2a536db8d P 833582ff68c94269be364ebd187a75db -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:14:44.650168 12830 raft_consensus.cc:482] T 2dce5b2aaea045ccb7424cbbd44576c2 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:44.650339 12830 raft_consensus.cc:504] T 2dce5b2aaea045ccb7424cbbd44576c2 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:44.650935 12830 leader_election.cc:288] T 2dce5b2aaea045ccb7424cbbd44576c2 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
W20240503 08:14:44.652271 31096 leader_election.cc:334] T 2dce5b2aaea045ccb7424cbbd44576c2 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
W20240503 08:14:44.652561 31095 leader_election.cc:334] T 2dce5b2aaea045ccb7424cbbd44576c2 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:44.652693 31095 leader_election.cc:302] T 2dce5b2aaea045ccb7424cbbd44576c2 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b00f244c69df401997ea99d492218a98; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:44.652933 12830 raft_consensus.cc:2726] T 2dce5b2aaea045ccb7424cbbd44576c2 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 08:14:44.697464 30829 consensus_peers.cc:485] T 7815cb2db9514c02b0ee3e3cd12b2212 P ce1496a47d084365ad786cd09e472a2c -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:44.746789 30828 consensus_peers.cc:462] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:44.752983 30828 consensus_peers.cc:485] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:14:44.753587 13421 raft_consensus.cc:482] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:44.753868 13421 raft_consensus.cc:504] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:44.755116 13421 leader_election.cc:288] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
W20240503 08:14:44.756421 30960 leader_election.cc:334] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:44.757004 30961 leader_election.cc:334] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:44.757114 30961 leader_election.cc:302] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:44.758829 13421 raft_consensus.cc:2726] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:14:44.766196 27937 consensus_peers.cc:485] T 0ceedfb43869465bb63931c39c28583a P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:14:44.769549 28327 consensus_peers.cc:485] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:14:44.792663 12840 raft_consensus.cc:482] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:44.792838 12840 raft_consensus.cc:504] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:44.793411 12840 leader_election.cc:288] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
W20240503 08:14:44.795027 28327 leader_election.cc:334] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:44.795262 28328 leader_election.cc:334] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:44.795398 28328 leader_election.cc:302] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d09a11a03989421b9f28019dc1c14cc2; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:44.796094 13061 raft_consensus.cc:2726] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240503 08:14:44.814124 30696 consensus_peers.cc:462] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
W20240503 08:14:44.845201 30696 consensus_peers.cc:485] T 1fcc779a649b461db4d14b5cd8b17744 P 6d1d7de19459435fbde1999e3dfd226b -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:44.886336 28327 consensus_peers.cc:485] T 45e7a03f838d4beaae2080d523319d58 P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:14:44.904573 27937 consensus_peers.cc:485] T d9098f3d36b24d578a3f8d7d3ccad052 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:14:44.913122 30742 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 08:14:44.920011 31141 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 08:14:44.925596 30874 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 08:14:44.926612 27983 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:14:44.945914 13337 raft_consensus.cc:482] T fa7e18041c294a239eb7b8f735b66c53 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:44.946084 13337 raft_consensus.cc:504] T fa7e18041c294a239eb7b8f735b66c53 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:44.946777 13337 leader_election.cc:288] T fa7e18041c294a239eb7b8f735b66c53 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
W20240503 08:14:44.952117 28327 consensus_peers.cc:485] T eeee6a265fc94e91bf9daa8301662e5f P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:14:44.953001 31006 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 08:14:44.963587 28327 consensus_peers.cc:485] T 8bcbc022caa94bb08a2db4de9b060557 P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:14:44.969779 13337 raft_consensus.cc:482] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:44.969949 13337 raft_consensus.cc:504] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:44.971948 13337 leader_election.cc:288] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
W20240503 08:14:44.972448 30828 leader_election.cc:334] T fa7e18041c294a239eb7b8f735b66c53 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:44.973239 30829 leader_election.cc:334] T fa7e18041c294a239eb7b8f735b66c53 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:44.973373 30829 leader_election.cc:302] T fa7e18041c294a239eb7b8f735b66c53 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:44.973824 13337 raft_consensus.cc:2726] T fa7e18041c294a239eb7b8f735b66c53 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 08:14:44.974613 30829 leader_election.cc:334] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
W20240503 08:14:44.975255 30828 leader_election.cc:334] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:44.975370 30828 leader_election.cc:302] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:44.975658 13337 raft_consensus.cc:2726] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 08:14:44.997486 28373 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 08:14:45.054900 13421 raft_consensus.cc:482] T bab5c25c31a64b578489b1006f631cc0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:45.055049 13421 raft_consensus.cc:504] T bab5c25c31a64b578489b1006f631cc0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:45.055858 13421 leader_election.cc:288] T bab5c25c31a64b578489b1006f631cc0 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
W20240503 08:14:45.057480 30960 leader_election.cc:334] T bab5c25c31a64b578489b1006f631cc0 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:45.058099 30961 leader_election.cc:334] T bab5c25c31a64b578489b1006f631cc0 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:45.058064 28389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab5c25c31a64b578489b1006f631cc0" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:14:45.058193 30961 leader_election.cc:302] T bab5c25c31a64b578489b1006f631cc0 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:45.058275 28389 raft_consensus.cc:2454] T bab5c25c31a64b578489b1006f631cc0 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 1.
I20240503 08:14:45.058429 13421 raft_consensus.cc:2726] T bab5c25c31a64b578489b1006f631cc0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:14:45.070848 12840 raft_consensus.cc:482] T d94f5824a71f454f8a1bfa13dd032684 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:45.071007 13452 raft_consensus.cc:482] T 099e37089b2e42b4839e681f8f37365d P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:45.071028 12840 raft_consensus.cc:504] T d94f5824a71f454f8a1bfa13dd032684 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:45.071192 13452 raft_consensus.cc:504] T 099e37089b2e42b4839e681f8f37365d P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:14:45.071776 12840 leader_election.cc:288] T d94f5824a71f454f8a1bfa13dd032684 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
I20240503 08:14:45.071931 13452 leader_election.cc:288] T 099e37089b2e42b4839e681f8f37365d P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
W20240503 08:14:45.073460 28328 leader_election.cc:334] T d94f5824a71f454f8a1bfa13dd032684 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
W20240503 08:14:45.074409 30696 leader_election.cc:334] T 099e37089b2e42b4839e681f8f37365d P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
W20240503 08:14:45.074474 28327 leader_election.cc:334] T d94f5824a71f454f8a1bfa13dd032684 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:45.074520 27999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099e37089b2e42b4839e681f8f37365d" candidate_uuid: "6d1d7de19459435fbde1999e3dfd226b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "833582ff68c94269be364ebd187a75db" is_pre_election: true
I20240503 08:14:45.074584 28327 leader_election.cc:302] T d94f5824a71f454f8a1bfa13dd032684 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d09a11a03989421b9f28019dc1c14cc2; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:45.074697 27999 raft_consensus.cc:2454] T 099e37089b2e42b4839e681f8f37365d P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1d7de19459435fbde1999e3dfd226b in term 1.
I20240503 08:14:45.074833 12840 raft_consensus.cc:2726] T d94f5824a71f454f8a1bfa13dd032684 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:14:45.075546 31025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d94f5824a71f454f8a1bfa13dd032684" candidate_uuid: "d09a11a03989421b9f28019dc1c14cc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" is_pre_election: true
I20240503 08:14:45.075752 31025 raft_consensus.cc:2454] T d94f5824a71f454f8a1bfa13dd032684 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09a11a03989421b9f28019dc1c14cc2 in term 1.
W20240503 08:14:45.076083 30695 leader_election.cc:334] T 099e37089b2e42b4839e681f8f37365d P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:45.076215 30695 leader_election.cc:302] T 099e37089b2e42b4839e681f8f37365d P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, 833582ff68c94269be364ebd187a75db; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:45.076475 13452 raft_consensus.cc:2726] T 099e37089b2e42b4839e681f8f37365d P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:14:45.082981 13421 raft_consensus.cc:482] T 978bf5783b0e47e3bfb30441adc600ae P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:45.083117 13421 raft_consensus.cc:504] T 978bf5783b0e47e3bfb30441adc600ae P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:45.083690 13421 leader_election.cc:288] T 978bf5783b0e47e3bfb30441adc600ae P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
W20240503 08:14:45.084941 30961 leader_election.cc:334] T 978bf5783b0e47e3bfb30441adc600ae P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
W20240503 08:14:45.085062 30960 leader_election.cc:334] T 978bf5783b0e47e3bfb30441adc600ae P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:45.085153 30960 leader_election.cc:302] T 978bf5783b0e47e3bfb30441adc600ae P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:45.085355 13421 raft_consensus.cc:2726] T 978bf5783b0e47e3bfb30441adc600ae P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 08:14:45.089368 13421 raft_consensus.cc:482] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:45.089498 13421 raft_consensus.cc:504] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:45.093365 13421 leader_election.cc:288] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
W20240503 08:14:45.094424 30961 leader_election.cc:334] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
W20240503 08:14:45.094601 30960 leader_election.cc:334] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:45.094709 30960 leader_election.cc:302] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:45.094938 13421 raft_consensus.cc:2726] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240503 08:14:45.287245 12963 raft_consensus.cc:482] T df90a0d9d338416b81e253fa23542cca P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:45.287438 12963 raft_consensus.cc:504] T df90a0d9d338416b81e253fa23542cca P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:14:45.288092 12963 leader_election.cc:288] T df90a0d9d338416b81e253fa23542cca P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
W20240503 08:14:45.289615 27938 leader_election.cc:334] T df90a0d9d338416b81e253fa23542cca P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
W20240503 08:14:45.290144 27937 leader_election.cc:334] T df90a0d9d338416b81e253fa23542cca P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:45.290274 27937 leader_election.cc:302] T df90a0d9d338416b81e253fa23542cca P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 833582ff68c94269be364ebd187a75db; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:45.290510 12963 raft_consensus.cc:2726] T df90a0d9d338416b81e253fa23542cca P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 08:14:45.295859 30828 consensus_peers.cc:462] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:45.335747 27983 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:14:45.337361 30742 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 08:14:45.353510 31141 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 08:14:45.361240 30874 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 08:14:45.369971 27938 consensus_peers.cc:485] T d0a06e419c6c4f3db6705cb2377e9336 P 833582ff68c94269be364ebd187a75db -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:45.377000 30696 consensus_peers.cc:462] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:45.383417 31006 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 08:14:45.402776 28373 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 08:14:45.406637 30960 consensus_peers.cc:485] T 41aa0e288ca4469ba34a07ff159ace91 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:14:45.521438 30895 consensus_queue.cc:237] T e3a6b72060804beb81a177988d287f63 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:14:45.523451 30761 raft_consensus.cc:1261] T e3a6b72060804beb81a177988d287f63 P 6d1d7de19459435fbde1999e3dfd226b [term 2 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:14:45.523932 28000 raft_consensus.cc:1261] T e3a6b72060804beb81a177988d287f63 P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Refusing update from remote peer ce1496a47d084365ad786cd09e472a2c: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:14:45.523970 12941 consensus_queue.cc:1035] T e3a6b72060804beb81a177988d287f63 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:45.524453 13336 consensus_queue.cc:1035] T e3a6b72060804beb81a177988d287f63 P ce1496a47d084365ad786cd09e472a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:45.538430 13336 raft_consensus.cc:2932] T e3a6b72060804beb81a177988d287f63 P ce1496a47d084365ad786cd09e472a2c [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:14:45.538936 30761 raft_consensus.cc:2932] T e3a6b72060804beb81a177988d287f63 P 6d1d7de19459435fbde1999e3dfd226b [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:14:45.545389 27856 catalog_manager.cc:5484] T e3a6b72060804beb81a177988d287f63 P ce1496a47d084365ad786cd09e472a2c reported cstate change: config changed from index 2 to 4, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) evicted. New cstate: current_term: 2 leader_uuid: "ce1496a47d084365ad786cd09e472a2c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:14:45.549228 28000 raft_consensus.cc:2932] T e3a6b72060804beb81a177988d287f63 P 833582ff68c94269be364ebd187a75db [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } }
I20240503 08:14:45.574074 28373 tablet_service.cc:1514] Processing DeleteTablet for tablet e3a6b72060804beb81a177988d287f63 with delete_type TABLET_DATA_TOMBSTONED (TS d09a11a03989421b9f28019dc1c14cc2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34786
I20240503 08:14:45.586108 13532 tablet_replica.cc:330] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2: stopping tablet replica
I20240503 08:14:45.586325 13532 raft_consensus.cc:2229] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:45.586496 13532 raft_consensus.cc:2258] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:45.587014 13532 ts_tablet_manager.cc:1875] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:45.591418 13532 ts_tablet_manager.cc:1888] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240503 08:14:45.591531 13532 log.cc:1196] T e3a6b72060804beb81a177988d287f63 P d09a11a03989421b9f28019dc1c14cc2: Deleting WAL directory at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/wals/e3a6b72060804beb81a177988d287f63
I20240503 08:14:45.592166 27849 catalog_manager.cc:4830] TS d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): tablet e3a6b72060804beb81a177988d287f63 (table rebalancer_extra_table_60 [id=2979b3f65625430db5d8039280a09b13]) successfully deleted
W20240503 08:14:45.716879 27938 consensus_peers.cc:485] T ab35ebabd6fe45fab890739a800794b1 P 833582ff68c94269be364ebd187a75db -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:45.724663 27937 consensus_peers.cc:485] T f1b2f3b8111649a7b5725647b8251698 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:14:45.731333 27937 consensus_peers.cc:485] T 8cc6ae30aa60495d8a4364a11d5bef79 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:14:45.748018 30829 consensus_peers.cc:485] T b52e4b59153d40378e2a37809b04a2c8 P ce1496a47d084365ad786cd09e472a2c -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:45.818315 30828 consensus_peers.cc:485] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:14:45.820748 30829 consensus_peers.cc:485] T 7a101bee3af34166848ee7fa7c733892 P ce1496a47d084365ad786cd09e472a2c -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:45.916276 30828 consensus_peers.cc:462] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:45.930958 30874 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 08:14:45.936496 30695 consensus_peers.cc:485] T 2222ff7fb5184d6c86010e1475d6c1f7 P 6d1d7de19459435fbde1999e3dfd226b -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:14:45.939882 30742 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 08:14:45.942365 28327 consensus_peers.cc:485] T cdad6ef53b6f413ebfda67f9b42c9c23 P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:14:45.947223 30696 consensus_peers.cc:462] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:45.949990 31141 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 08:14:45.954473 30828 consensus_peers.cc:485] T 0f25688be4e74d31aaab85b7a4329a34 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:14:45.967940 27938 consensus_peers.cc:485] T 35c62760e1de4c2f88d06b97d0286c4c P 833582ff68c94269be364ebd187a75db -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:14:45.970669 31006 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 08:14:45.971603 27983 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 08:14:45.977591 28373 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 08:14:46.027992 30696 consensus_peers.cc:485] T 7083414bc16c4d499c97b93ca4e3f631 P 6d1d7de19459435fbde1999e3dfd226b -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:46.080405 31095 consensus_peers.cc:485] T 6a96965691644db8b9461d146536a026 P b00f244c69df401997ea99d492218a98 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:14:46.213209 31095 consensus_peers.cc:485] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:14:46.281744 30960 consensus_peers.cc:485] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:14:46.295683 30696 consensus_peers.cc:485] T 91c573181144456aa7e27a3026a639c4 P 6d1d7de19459435fbde1999e3dfd226b -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:46.340932 30829 consensus_peers.cc:485] T d0657f27fd474a41b0c33db55989daa3 P ce1496a47d084365ad786cd09e472a2c -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:46.359556 30696 consensus_peers.cc:462] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
W20240503 08:14:46.366163 28327 consensus_peers.cc:485] T f46393257ce9471891698c36a07546ad P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:14:46.384933 28328 consensus_peers.cc:485] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:46.424105 30961 consensus_peers.cc:485] T 50e4d2c569344dde915eec12f6f3d747 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:46.427433 27937 consensus_peers.cc:485] T ca38671a760d4534ab903579e0487772 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:14:46.432051 12654 raft_consensus.cc:482] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:46.432205 12654 raft_consensus.cc:504] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:14:46.432866 12654 leader_election.cc:288] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
W20240503 08:14:46.434469 31095 leader_election.cc:334] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:46.442780 31096 leader_election.cc:334] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:46.442876 31096 leader_election.cc:302] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b00f244c69df401997ea99d492218a98; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:46.443084 12654 raft_consensus.cc:2726] T 8fc87f7f3c724cd4a270583ddebd87aa P b00f244c69df401997ea99d492218a98 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 08:14:46.466539 13421 raft_consensus.cc:482] T 658865ee3f8d424ca5cd35e12e6d7c77 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:46.466694 13421 raft_consensus.cc:504] T 658865ee3f8d424ca5cd35e12e6d7c77 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
W20240503 08:14:46.466928 31095 consensus_peers.cc:485] T 486e9843fd784148b9b49dfe27996dff P b00f244c69df401997ea99d492218a98 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:14:46.467288 13421 leader_election.cc:288] T 658865ee3f8d424ca5cd35e12e6d7c77 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
W20240503 08:14:46.468528 30961 leader_election.cc:334] T 658865ee3f8d424ca5cd35e12e6d7c77 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
W20240503 08:14:46.469025 30828 consensus_peers.cc:462] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:46.469462 30960 leader_election.cc:334] T 658865ee3f8d424ca5cd35e12e6d7c77 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:46.469569 30960 leader_election.cc:302] T 658865ee3f8d424ca5cd35e12e6d7c77 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:46.469798 13421 raft_consensus.cc:2726] T 658865ee3f8d424ca5cd35e12e6d7c77 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 08:14:46.474015 30828 consensus_peers.cc:485] T f83a4778ffe640ee907e0a73e58abb73 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:14:46.491689 30961 consensus_peers.cc:485] T fe4d26ee022b424195e836834bd2f051 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:14:46.494735 30695 consensus_peers.cc:485] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:14:46.495237 27937 consensus_peers.cc:485] T 08fb64977309479b831635344969c662 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:14:46.495980 27938 consensus_peers.cc:485] T 417cedc3fa5c4a85a7a130005291d64b P 833582ff68c94269be364ebd187a75db -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:46.509511 30696 consensus_peers.cc:485] T 6c2824aa878845e098f600f1fcfea087 P 6d1d7de19459435fbde1999e3dfd226b -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:14:46.511207 12654 raft_consensus.cc:482] T 2dce5b2aaea045ccb7424cbbd44576c2 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:46.511369 12654 raft_consensus.cc:504] T 2dce5b2aaea045ccb7424cbbd44576c2 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
W20240503 08:14:46.513264 31096 leader_election.cc:334] T 2dce5b2aaea045ccb7424cbbd44576c2 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:46.513794 12654 leader_election.cc:288] T 2dce5b2aaea045ccb7424cbbd44576c2 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
W20240503 08:14:46.514910 31095 leader_election.cc:334] T 2dce5b2aaea045ccb7424cbbd44576c2 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:46.514998 31095 leader_election.cc:302] T 2dce5b2aaea045ccb7424cbbd44576c2 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b00f244c69df401997ea99d492218a98; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:46.515198 12654 raft_consensus.cc:2726] T 2dce5b2aaea045ccb7424cbbd44576c2 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 08:14:46.558869 27937 consensus_peers.cc:485] T c8ca21ffe16e41ab80f5693dc4dbb682 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:14:46.559005 27937 consensus_peers.cc:485] T 46d27fa12990405cae5f94976ac7a014 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:14:46.579020 13452 raft_consensus.cc:482] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:46.579178 13452 raft_consensus.cc:504] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:14:46.579743 13452 leader_election.cc:288] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
W20240503 08:14:46.582605 30696 leader_election.cc:334] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:46.583374 13061 raft_consensus.cc:482] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:46.583509 13061 raft_consensus.cc:504] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:46.584038 13061 leader_election.cc:288] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
W20240503 08:14:46.585584 28328 leader_election.cc:334] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
W20240503 08:14:46.585883 30695 leader_election.cc:334] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:46.586163 30695 leader_election.cc:302] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:46.586575 13452 raft_consensus.cc:2726] T 1cac39d5831d4b6887250f7a9d63dcd5 P 6d1d7de19459435fbde1999e3dfd226b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:14:46.587971 28327 leader_election.cc:334] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:46.588153 28327 leader_election.cc:302] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d09a11a03989421b9f28019dc1c14cc2; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:46.588413 13061 raft_consensus.cc:2726] T 362590d3d0834d0b8a672ab552f8c209 P d09a11a03989421b9f28019dc1c14cc2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240503 08:14:46.606611 30829 consensus_peers.cc:485] T 8c714f29abb140379b57e08eb927e2b7 P ce1496a47d084365ad786cd09e472a2c -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:46.610764 27938 consensus_peers.cc:485] T 45f47478f49149b895060791655503ae P 833582ff68c94269be364ebd187a75db -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:46.612097 30961 consensus_peers.cc:485] T 7df30eb244914dcabc6965324eced236 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:46.612519 27938 consensus_peers.cc:485] T 0ceedfb43869465bb63931c39c28583a P 833582ff68c94269be364ebd187a75db -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:46.625195 30829 consensus_peers.cc:485] T 62ca502a089f4a37999d2252776b741b P ce1496a47d084365ad786cd09e472a2c -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:46.628145 28328 consensus_peers.cc:485] T 07866de372e541ec9dca8a3d42f8b0ab P d09a11a03989421b9f28019dc1c14cc2 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:14:46.633105 12963 raft_consensus.cc:482] T 099e37089b2e42b4839e681f8f37365d P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:46.633249 12963 raft_consensus.cc:504] T 099e37089b2e42b4839e681f8f37365d P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } }
I20240503 08:14:46.634016 12963 leader_election.cc:288] T 099e37089b2e42b4839e681f8f37365d P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725)
I20240503 08:14:46.634959 30760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099e37089b2e42b4839e681f8f37365d" candidate_uuid: "833582ff68c94269be364ebd187a75db" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d1d7de19459435fbde1999e3dfd226b" is_pre_election: true
I20240503 08:14:46.635161 30760 raft_consensus.cc:2454] T 099e37089b2e42b4839e681f8f37365d P 6d1d7de19459435fbde1999e3dfd226b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833582ff68c94269be364ebd187a75db in term 1.
W20240503 08:14:46.635640 27938 leader_election.cc:334] T 099e37089b2e42b4839e681f8f37365d P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
W20240503 08:14:46.635668 28327 consensus_peers.cc:485] T bdfe68a127f043f08f04749843a38375 P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:14:46.636847 27937 leader_election.cc:334] T 099e37089b2e42b4839e681f8f37365d P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:46.636945 27937 leader_election.cc:302] T 099e37089b2e42b4839e681f8f37365d P 833582ff68c94269be364ebd187a75db [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 6d1d7de19459435fbde1999e3dfd226b, 833582ff68c94269be364ebd187a75db; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:46.637159 12963 raft_consensus.cc:2726] T 099e37089b2e42b4839e681f8f37365d P 833582ff68c94269be364ebd187a75db [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1203: Failure
Value of: s.ok()
Actual: false
Expected: true
Runtime error: /tmp/dist-test-task8bBFGA/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 44
Maximum Replica Count | 53
Average Replica Count | 49.500000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 2
Maximum | 3
Average | 2.568182
stderr: W20240503 08:13:10.714499 576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:10.714813 576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:10.723385 576 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:13:12.267714 576 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 08:13:13.046983 576 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f65dccb36e67468c8e28e95e66b5e4ef: could not find any suitable replica to move from server aae860d7ae2b4002bf7794c278f9cc56 to server ce1496a47d084365ad786cd09e472a2c
W20240503 08:13:13.047251 576 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3cd9bb4078874a2e8da7dea4f43db999: could not find any suitable replica to move from server ce1496a47d084365ad786cd09e472a2c to server aae860d7ae2b4002bf7794c278f9cc56
W20240503 08:13:13.047565 576 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8e9aac1c8668451485cb809d15f7c27d: could not find any suitable replica to move from server d09a11a03989421b9f28019dc1c14cc2 to server 6d1d7de19459435fbde1999e3dfd226b
W20240503 08:13:13.048022 576 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 05d29869a30847b6a0c137521965898a: could not find any suitable replica to move from server d09a11a03989421b9f28019dc1c14cc2 to server 6d1d7de19459435fbde1999e3dfd226b
W20240503 08:13:13.048141 576 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 490e37ef740f4e96b915339369c454ff: could not find any suitable replica to move from server d09a11a03989421b9f28019dc1c14cc2 to server b00f244c69df401997ea99d492218a98
W20240503 08:13:13.048261 576 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6e8538506685489f8a1e931a2a781746: could not find any suitable replica to move from server 1b411c6bb25549368041773d7b710d48 to server ced21ca983b749d0b4d4f6a0e2426bf8
W20240503 08:13:13.048503 576 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e37b245cb78d400384ac439a89fb8706: could not find any suitable replica to move from server ce1496a47d084365ad786cd09e472a2c to server 1b411c6bb25549368041773d7b710d48
W20240503 08:13:13.049084 576 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 18d795e9c19046fdba033edcea369490: could not find any suitable replica to move from server aae860d7ae2b4002bf7794c278f9cc56 to server ce1496a47d084365ad786cd09e472a2c
W20240503 08:13:13.049341 576 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2af0a6bdbc404add8684cd11068a06f9: could not find any suitable replica to move from server 6d1d7de19459435fbde1999e3dfd226b to server aae860d7ae2b4002bf7794c278f9cc56
W20240503 08:13:13.049474 576 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 80792788b9164bd4af02769ad074bd3a: could not find any suitable replica to move from server ce1496a47d084365ad786cd09e472a2c to server b00f244c69df401997ea99d492218a98
W20240503 08:13:13.049593 576 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 261c96b04fc141ddb92650c24ce76fa8: could not find any suitable replica to move from server aae860d7ae2b4002bf7794c278f9cc56 to server 6d1d7de19459435fbde1999e3dfd226b
W20240503 08:13:13.049722 576 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7e5325b51bd7410e93410447bfd72f9c: could not find any suitable replica to move from server aae860d7ae2b4002bf7794c278f9cc56 to server ce1496a47d084365ad786cd09e472a2c
W20240503 08:13:13.049839 576 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table eeeb4ab7272f4686b013f5d8f74fc917: could not find any suitable replica to move from server d09a11a03989421b9f28019dc1c14cc2 to server ced21ca983b749d0b4d4f6a0e2426bf8
W20240503 08:13:13.050016 576 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6db67fd7274948c0858a325398f6acbc: could not find any suitable replica to move from server d09a11a03989421b9f28019dc1c14cc2 to server ced21ca983b749d0b4d4f6a0e2426bf8
W20240503 08:13:13.050105 576 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1b577ea991674f799fea7ba7f25e80ad: could not find any suitable replica to move from server 6d1d7de19459435fbde1999e3dfd226b to server d09a11a03989421b9f28019dc1c14cc2
W20240503 08:13:13.050308 576 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e00af55ec1c448bdb2c617816b10a90f: could not find any suitable replica to move from server 833582ff68c94269be364ebd187a75db to server 6d1d7de19459435fbde1999e3dfd226b
W20240503 08:13:13.050534 576 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 10122b5266cf4446bb541a06acf3a16f: could not find any suitable replica to move from server ced21ca983b749d0b4d4f6a0e2426bf8 to server ce1496a47d084365ad786cd09e472a2c
W20240503 08:13:13.050719 576 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 40dc8c7f01804911a18e5c62a4b6be48: could not find any suitable replica to move from server ced21ca983b749d0b4d4f6a0e2426bf8 to server d09a11a03989421b9f28019dc1c14cc2
I20240503 08:13:13.134330 576 rebalancer_tool.cc:1074] tablet 417cedc3fa5c4a85a7a130005291d64b: 'b00f244c69df401997ea99d492218a98' -> '1b411c6bb25549368041773d7b710d48' move scheduled
I20240503 08:13:13.198101 576 rebalancer_tool.cc:1074] tablet c9055668b86e4dc5aa321a4d29beaf74: 'ced21ca983b749d0b4d4f6a0e2426bf8' -> '833582ff68c94269be364ebd187a75db' move scheduled
I20240503 08:13:13.269548 576 rebalancer_tool.cc:1074] tablet 3042212758bf4aef8be5c5d12d10113c: 'aae860d7ae2b4002bf7794c278f9cc56' -> 'ce1496a47d084365ad786cd09e472a2c' move scheduled
I20240503 08:13:13.330229 576 rebalancer_tool.cc:1074] tablet 7f2743ab5fa24c9899b67ea52f48f9d8: '6d1d7de19459435fbde1999e3dfd226b' -> 'd09a11a03989421b9f28019dc1c14cc2' move scheduled
I20240503 08:13:13.393499 576 rebalancer_tool.cc:1074] tablet 2222ff7fb5184d6c86010e1475d6c1f7: 'b00f244c69df401997ea99d492218a98' -> '1b411c6bb25549368041773d7b710d48' move scheduled
I20240503 08:13:13.453121 576 rebalancer_tool.cc:1074] tablet f9b4f775b9914bdc98b9b582072cb1ab: 'ced21ca983b749d0b4d4f6a0e2426bf8' -> '833582ff68c94269be364ebd187a75db' move scheduled
I20240503 08:13:13.514194 576 rebalancer_tool.cc:1074] tablet d3da6397e0c34c8f96933963dd4fa65e: 'aae860d7ae2b4002bf7794c278f9cc56' -> 'ce1496a47d084365ad786cd09e472a2c' move scheduled
I20240503 08:13:13.600708 576 rebalancer_tool.cc:1074] tablet 013e00c67a1c4ae9b3c1a050b966e9a5: '6d1d7de19459435fbde1999e3dfd226b' -> 'd09a11a03989421b9f28019dc1c14cc2' move scheduled
I20240503 08:13:13.706696 576 rebalancer_tool.cc:1074] tablet 7df30eb244914dcabc6965324eced236: 'b00f244c69df401997ea99d492218a98' -> '1b411c6bb25549368041773d7b710d48' move scheduled
I20240503 08:13:13.783340 576 rebalancer_tool.cc:1074] tablet de5dfccb0359462dbb7da776293fa679: '833582ff68c94269be364ebd187a75db' -> 'ced21ca983b749d0b4d4f6a0e2426bf8' move scheduled
I20240503 08:13:13.854471 576 rebalancer_tool.cc:1074] tablet 7a101bee3af34166848ee7fa7c733892: 'aae860d7ae2b4002bf7794c278f9cc56' -> 'ce1496a47d084365ad786cd09e472a2c' move scheduled
I20240503 08:13:13.960078 576 rebalancer_tool.cc:1074] tablet 8cc6ae30aa60495d8a4364a11d5bef79: 'd09a11a03989421b9f28019dc1c14cc2' -> '6d1d7de19459435fbde1999e3dfd226b' move scheduled
I20240503 08:13:14.265337 576 rebalancer_tool.cc:1074] tablet 1cac39d5831d4b6887250f7a9d63dcd5: 'b00f244c69df401997ea99d492218a98' -> '1b411c6bb25549368041773d7b710d48' move scheduled
I20240503 08:13:14.428594 576 rebalancer_tool.cc:1074] tablet 8450e62352a74d2ea1471166d4cb1132: '833582ff68c94269be364ebd187a75db' -> 'ced21ca983b749d0b4d4f6a0e2426bf8' move scheduled
I20240503 08:13:14.593323 576 rebalancer_tool.cc:1074] tablet 362590d3d0834d0b8a672ab552f8c209: 'ce1496a47d084365ad786cd09e472a2c' -> 'aae860d7ae2b4002bf7794c278f9cc56' move scheduled
I20240503 08:13:14.695650 576 rebalancer_tool.cc:1074] tablet 8fc87f7f3c724cd4a270583ddebd87aa: 'd09a11a03989421b9f28019dc1c14cc2' -> 'b00f244c69df401997ea99d492218a98' move scheduled
I20240503 08:13:14.775908 576 rebalancer_tool.cc:1074] tablet 50e4d2c569344dde915eec12f6f3d747: '6d1d7de19459435fbde1999e3dfd226b' -> '1b411c6bb25549368041773d7b710d48' move scheduled
I20240503 08:13:14.855772 576 rebalancer_tool.cc:1074] tablet d808d27e71a5486f9773ad4b239d1cfe: 'ce1496a47d084365ad786cd09e472a2c' -> '833582ff68c94269be364ebd187a75db' move scheduled
I20240503 08:13:14.920801 576 rebalancer_tool.cc:1074] tablet 45e7a03f838d4beaae2080d523319d58: 'ced21ca983b749d0b4d4f6a0e2426bf8' -> 'aae860d7ae2b4002bf7794c278f9cc56' move scheduled
I20240503 08:13:28.855995 576 rebalancer_tool.cc:1136] tablet 50e4d2c569344dde915eec12f6f3d747: '6d1d7de19459435fbde1999e3dfd226b' -> '1b411c6bb25549368041773d7b710d48' move completed: OK
I20240503 08:13:28.866106 576 rebalancer_tool.cc:1136] tablet 45e7a03f838d4beaae2080d523319d58: 'ced21ca983b749d0b4d4f6a0e2426bf8' -> 'aae860d7ae2b4002bf7794c278f9cc56' move completed: OK
I20240503 08:13:28.875023 576 rebalancer_tool.cc:1136] tablet f9b4f775b9914bdc98b9b582072cb1ab: 'ced21ca983b749d0b4d4f6a0e2426bf8' -> '833582ff68c94269be364ebd187a75db' move completed: OK
I20240503 08:13:28.885982 576 rebalancer_tool.cc:1136] tablet 3042212758bf4aef8be5c5d12d10113c: 'aae860d7ae2b4002bf7794c278f9cc56' -> 'ce1496a47d084365ad786cd09e472a2c' move completed: OK
I20240503 08:13:28.894214 576 rebalancer_tool.cc:1136] tablet 013e00c67a1c4ae9b3c1a050b966e9a5: '6d1d7de19459435fbde1999e3dfd226b' -> 'd09a11a03989421b9f28019dc1c14cc2' move completed: OK
I20240503 08:13:28.902887 576 rebalancer_tool.cc:1136] tablet 1cac39d5831d4b6887250f7a9d63dcd5: 'b00f244c69df401997ea99d492218a98' -> '1b411c6bb25549368041773d7b710d48' move completed: OK
I20240503 08:13:29.507979 576 rebalancer_tool.cc:1136] tablet 417cedc3fa5c4a85a7a130005291d64b: 'b00f244c69df401997ea99d492218a98' -> '1b411c6bb25549368041773d7b710d48' move completed: OK
I20240503 08:13:29.527679 576 rebalancer_tool.cc:1136] tablet 7df30eb244914dcabc6965324eced236: 'b00f244c69df401997ea99d492218a98' -> '1b411c6bb25549368041773d7b710d48' move completed: OK
I20240503 08:13:29.558849 576 rebalancer_tool.cc:1136] tablet 2222ff7fb5184d6c86010e1475d6c1f7: 'b00f244c69df401997ea99d492218a98' -> '1b411c6bb25549368041773d7b710d48' move completed: OK
I20240503 08:13:29.575524 576 rebalancer_tool.cc:1136] tablet 8fc87f7f3c724cd4a270583ddebd87aa: 'd09a11a03989421b9f28019dc1c14cc2' -> 'b00f244c69df401997ea99d492218a98' move completed: OK
I20240503 08:13:29.584354 576 rebalancer_tool.cc:1136] tablet c9055668b86e4dc5aa321a4d29beaf74: 'ced21ca983b749d0b4d4f6a0e2426bf8' -> '833582ff68c94269be364ebd187a75db' move completed: OK
I20240503 08:13:29.592474 576 rebalancer_tool.cc:1136] tablet d3da6397e0c34c8f96933963dd4fa65e: 'aae860d7ae2b4002bf7794c278f9cc56' -> 'ce1496a47d084365ad786cd09e472a2c' move completed: OK
I20240503 08:13:29.602792 576 rebalancer_tool.cc:1136] tablet de5dfccb0359462dbb7da776293fa679: '833582ff68c94269be364ebd187a75db' -> 'ced21ca983b749d0b4d4f6a0e2426bf8' move completed: OK
I20240503 08:13:29.612149 576 rebalancer_tool.cc:1136] tablet 8450e62352a74d2ea1471166d4cb1132: '833582ff68c94269be364ebd187a75db' -> 'ced21ca983b749d0b4d4f6a0e2426bf8' move completed: OK
I20240503 08:13:29.622882 576 rebalancer_tool.cc:1136] tablet 7a101bee3af34166848ee7fa7c733892: 'aae860d7ae2b4002bf7794c278f9cc56' -> 'ce1496a47d084365ad786cd09e472a2c' move completed: OK
I20240503 08:13:29.630909 576 rebalancer_tool.cc:1136] tablet 8cc6ae30aa60495d8a4364a11d5bef79: 'd09a11a03989421b9f28019dc1c14cc2' -> '6d1d7de19459435fbde1999e3dfd226b' move completed: OK
I20240503 08:13:30.268200 576 rebalancer_tool.cc:1074] tablet ca38671a760d4534ab903579e0487772: '1b411c6bb25549368041773d7b710d48' -> 'b00f244c69df401997ea99d492218a98' move scheduled
I20240503 08:13:30.327112 576 rebalancer_tool.cc:1074] tablet f53acf725b62447382e9bae95e6e74d5: 'ced21ca983b749d0b4d4f6a0e2426bf8' -> 'aae860d7ae2b4002bf7794c278f9cc56' move scheduled
I20240503 08:13:30.369963 576 rebalancer_tool.cc:1074] tablet b12ceaf7f8d04fad8b07b36459da7116: '833582ff68c94269be364ebd187a75db' -> 'd09a11a03989421b9f28019dc1c14cc2' move scheduled
I20240503 08:13:30.416042 576 rebalancer_tool.cc:1074] tablet cdad6ef53b6f413ebfda67f9b42c9c23: '1b411c6bb25549368041773d7b710d48' -> '6d1d7de19459435fbde1999e3dfd226b' move scheduled
I20240503 08:13:30.477130 576 rebalancer_tool.cc:1074] tablet 8425cfe5c6134a60852e12f9af55227b: 'b00f244c69df401997ea99d492218a98' -> 'aae860d7ae2b4002bf7794c278f9cc56' move scheduled
I20240503 08:13:30.544298 576 rebalancer_tool.cc:1074] tablet 4a23c8186c4d4b23b62d7474088b45c1: 'ced21ca983b749d0b4d4f6a0e2426bf8' -> 'ce1496a47d084365ad786cd09e472a2c' move scheduled
I20240503 08:13:30.611140 576 rebalancer_tool.cc:1074] tablet df31bd9e544a44f280db2b0d021000c2: '833582ff68c94269be364ebd187a75db' -> 'd09a11a03989421b9f28019dc1c14cc2' move scheduled
I20240503 08:13:30.698051 576 rebalancer_tool.cc:1074] tablet 08fb64977309479b831635344969c662: '1b411c6bb25549368041773d7b710d48' -> '6d1d7de19459435fbde1999e3dfd226b' move scheduled
I20240503 08:13:30.791273 576 rebalancer_tool.cc:1074] tablet 17c6e3293aad4591a9c1395ebffff2e6: 'b00f244c69df401997ea99d492218a98' -> 'aae860d7ae2b4002bf7794c278f9cc56' move scheduled
I20240503 08:13:30.859951 576 rebalancer_tool.cc:1074] tablet 9fc39354ba064392be1af1769513f738: 'ce1496a47d084365ad786cd09e472a2c' -> 'ced21ca983b749d0b4d4f6a0e2426bf8' move scheduled
I20240503 08:13:31.081063 576 rebalancer_tool.cc:1074] tablet 4ed8026daee840c6b31ca7d989ed25f0: '833582ff68c94269be364ebd187a75db' -> 'd09a11a03989421b9f28019dc1c14cc2' move scheduled
I20240503 08:13:31.196532 576 rebalancer_tool.cc:1074] tablet 599cb18c58a74e93a3e0334cc43a73d4: '1b411c6bb25549368041773d7b710d48' -> 'b00f244c69df401997ea99d492218a98' move scheduled
I20240503 08:13:31.249200 576 rebalancer_tool.cc:1074] tablet b40a43ad7ed44c30a070189ebfe5aab6: 'ced21ca983b749d0b4d4f6a0e2426bf8' -> '6d1d7de19459435fbde1999e3dfd226b' move scheduled
I20240503 08:13:31.378422 576 rebalancer_tool.cc:1074] tablet 62ca502a089f4a37999d2252776b741b: 'aae860d7ae2b4002bf7794c278f9cc56' -> '1b411c6bb25549368041773d7b710d48' move scheduled
I20240503 08:13:31.429328 576 rebalancer_tool.cc:1074] tablet f9746769e2514c65ac8d48ca7df8bca7: 'ce1496a47d084365ad786cd09e472a2c' -> '833582ff68c94269be364ebd187a75db' move scheduled
I20240503 08:13:31.478571 576 rebalancer_tool.cc:1074] tablet 46d27fa12990405cae5f94976ac7a014: 'd09a11a03989421b9f28019dc1c14cc2' -> 'b00f244c69df401997ea99d492218a98' move scheduled
I20240503 08:13:42.833135 576 rebalancer_tool.cc:1136] tablet d808d27e71a5486f9773ad4b239d1cfe: 'ce1496a47d084365ad786cd09e472a2c' -> '833582ff68c94269be364ebd187a75db' move completed: OK
I20240503 08:13:42.840956 576 rebalancer_tool.cc:1136] tablet 7f2743ab5fa24c9899b67ea52f48f9d8: '6d1d7de19459435fbde1999e3dfd226b' -> 'd09a11a03989421b9f28019dc1c14cc2' move completed: OK
I20240503 08:13:42.849617 576 rebalancer_tool.cc:1136] tablet 362590d3d0834d0b8a672ab552f8c209: 'ce1496a47d084365ad786cd09e472a2c' -> 'aae860d7ae2b4002bf7794c278f9cc56' move completed: OK
I20240503 08:13:42.895545 576 rebalancer_tool.cc:1074] tablet 48075c4a45d24e84bb2b9760710de3a0: '6d1d7de19459435fbde1999e3dfd226b' -> 'ce1496a47d084365ad786cd09e472a2c' move scheduled
I20240503 08:13:42.942965 576 rebalancer_tool.cc:1074] tablet 200387e6edce4ee9bf11331380923005: 'ced21ca983b749d0b4d4f6a0e2426bf8' -> 'd09a11a03989421b9f28019dc1c14cc2' move scheduled
I20240503 08:13:42.995481 576 rebalancer_tool.cc:1074] tablet 58611f4ce8b04ac39064a51580d73f31: '6d1d7de19459435fbde1999e3dfd226b' -> '833582ff68c94269be364ebd187a75db' move scheduled
I20240503 08:13:44.934773 576 rebalancer_tool.cc:1136] tablet b40a43ad7ed44c30a070189ebfe5aab6: 'ced21ca983b749d0b4d4f6a0e2426bf8' -> '6d1d7de19459435fbde1999e3dfd226b' move completed: OK
I20240503 08:13:44.943486 576 rebalancer_tool.cc:1136] tablet 4ed8026daee840c6b31ca7d989ed25f0: '833582ff68c94269be364ebd187a75db' -> 'd09a11a03989421b9f28019dc1c14cc2' move completed: OK
I20240503 08:13:44.952633 576 rebalancer_tool.cc:1136] tablet 9fc39354ba064392be1af1769513f738: 'ce1496a47d084365ad786cd09e472a2c' -> 'ced21ca983b749d0b4d4f6a0e2426bf8' move completed: OK
I20240503 08:13:46.887980 576 rebalancer_tool.cc:1136] tablet 08fb64977309479b831635344969c662: '1b411c6bb25549368041773d7b710d48' -> '6d1d7de19459435fbde1999e3dfd226b' move completed: OK
I20240503 08:13:47.586251 576 rebalancer_tool.cc:1136] tablet 4a23c8186c4d4b23b62d7474088b45c1: 'ced21ca983b749d0b4d4f6a0e2426bf8' -> 'ce1496a47d084365ad786cd09e472a2c' move completed: OK
I20240503 08:13:48.207423 576 rebalancer_tool.cc:1136] tablet 8425cfe5c6134a60852e12f9af55227b: 'b00f244c69df401997ea99d492218a98' -> 'aae860d7ae2b4002bf7794c278f9cc56' move completed: OK
I20240503 08:13:48.215477 576 rebalancer_tool.cc:1136] tablet cdad6ef53b6f413ebfda67f9b42c9c23: '1b411c6bb25549368041773d7b710d48' -> '6d1d7de19459435fbde1999e3dfd226b' move completed: OK
I20240503 08:13:48.226359 576 rebalancer_tool.cc:1136] tablet b12ceaf7f8d04fad8b07b36459da7116: '833582ff68c94269be364ebd187a75db' -> 'd09a11a03989421b9f28019dc1c14cc2' move completed: OK
I20240503 08:13:48.251142 576 rebalancer_tool.cc:1136] tablet f53acf725b62447382e9bae95e6e74d5: 'ced21ca983b749d0b4d4f6a0e2426bf8' -> 'aae860d7ae2b4002bf7794c278f9cc56' move completed: OK
I20240503 08:13:48.261415 576 rebalancer_tool.cc:1136] tablet 599cb18c58a74e93a3e0334cc43a73d4: '1b411c6bb25549368041773d7b710d48' -> 'b00f244c69df401997ea99d492218a98' move completed: OK
I20240503 08:13:48.280545 576 rebalancer_tool.cc:1136] tablet ca38671a760d4534ab903579e0487772: '1b411c6bb25549368041773d7b710d48' -> 'b00f244c69df401997ea99d492218a98' move completed: OK
I20240503 08:13:48.289709 576 rebalancer_tool.cc:1136] tablet 62ca502a089f4a37999d2252776b741b: 'aae860d7ae2b4002bf7794c278f9cc56' -> '1b411c6bb25549368041773d7b710d48' move completed: OK
I20240503 08:13:48.336182 576 rebalancer_tool.cc:1074] tablet 41aa0e288ca4469ba34a07ff159ace91: '1b411c6bb25549368041773d7b710d48' -> 'ced21ca983b749d0b4d4f6a0e2426bf8' move scheduled
I20240503 08:13:48.374956 576 rebalancer_tool.cc:1074] tablet 1fca3d7f4b734eceb210e1186478eb72: 'aae860d7ae2b4002bf7794c278f9cc56' -> '1b411c6bb25549368041773d7b710d48' move scheduled
I20240503 08:13:48.431397 576 rebalancer_tool.cc:1074] tablet 98b3c906cfe54f49bb88ce5e00e3cc04: 'ce1496a47d084365ad786cd09e472a2c' -> '6d1d7de19459435fbde1999e3dfd226b' move scheduled
I20240503 08:13:48.485291 576 rebalancer_tool.cc:1074] tablet 3bc8f4968b9743baab5d25c8c7237c9c: 'b00f244c69df401997ea99d492218a98' -> 'aae860d7ae2b4002bf7794c278f9cc56' move scheduled
I20240503 08:13:48.561549 576 rebalancer_tool.cc:1074] tablet 76d79d8be9e94e99a7ba51d903ca8a91: 'ced21ca983b749d0b4d4f6a0e2426bf8' -> '1b411c6bb25549368041773d7b710d48' move scheduled
I20240503 08:13:48.622735 576 rebalancer_tool.cc:1074] tablet 49b145f6946647248c967cf4badf8b40: 'd09a11a03989421b9f28019dc1c14cc2' -> '833582ff68c94269be364ebd187a75db' move scheduled
I20240503 08:13:48.762197 576 rebalancer_tool.cc:1074] tablet 41e26e13355d4e0cb56bb526415a3998: 'ce1496a47d084365ad786cd09e472a2c' -> '1b411c6bb25549368041773d7b710d48' move scheduled
I20240503 08:13:48.820289 576 rebalancer_tool.cc:1074] tablet 9fbe2eaba5344dfaa29e78a0d73f224a: '6d1d7de19459435fbde1999e3dfd226b' -> 'aae860d7ae2b4002bf7794c278f9cc56' move scheduled
I20240503 08:13:48.967101 576 rebalancer_tool.cc:1074] tablet 01c4229dfd25458296e85e412eae13bf: 'b00f244c69df401997ea99d492218a98' -> 'd09a11a03989421b9f28019dc1c14cc2' move scheduled
I20240503 08:13:49.114216 576 rebalancer_tool.cc:1074] tablet ab7b01c2e05a430a9873398268b1e41f: 'ced21ca983b749d0b4d4f6a0e2426bf8' -> '1b411c6bb25549368041773d7b710d48' move scheduled
I20240503 08:13:49.257266 576 rebalancer_tool.cc:1074] tablet 35c62760e1de4c2f88d06b97d0286c4c: 'ce1496a47d084365ad786cd09e472a2c' -> '833582ff68c94269be364ebd187a75db' move scheduled
I20240503 08:13:49.335297 576 rebalancer_tool.cc:1074] tablet fa7e18041c294a239eb7b8f735b66c53: '6d1d7de19459435fbde1999e3dfd226b' -> 'aae860d7ae2b4002bf7794c278f9cc56' move scheduled
I20240503 08:13:57.201531 576 rebalancer_tool.cc:1136] tablet 46d27fa12990405cae5f94976ac7a014: 'd09a11a03989421b9f28019dc1c14cc2' -> 'b00f244c69df401997ea99d492218a98' move completed: OK
I20240503 08:13:57.209834 576 rebalancer_tool.cc:1136] tablet f9746769e2514c65ac8d48ca7df8bca7: 'ce1496a47d084365ad786cd09e472a2c' -> '833582ff68c94269be364ebd187a75db' move completed: OK
I20240503 08:13:58.350232 576 rebalancer_tool.cc:1136] tablet 17c6e3293aad4591a9c1395ebffff2e6: 'b00f244c69df401997ea99d492218a98' -> 'aae860d7ae2b4002bf7794c278f9cc56' move completed: OK
I20240503 08:13:58.367671 576 rebalancer_tool.cc:1136] tablet df31bd9e544a44f280db2b0d021000c2: '833582ff68c94269be364ebd187a75db' -> 'd09a11a03989421b9f28019dc1c14cc2' move completed: OK
I20240503 08:13:58.391937 576 rebalancer_tool.cc:1136] tablet 58611f4ce8b04ac39064a51580d73f31: '6d1d7de19459435fbde1999e3dfd226b' -> '833582ff68c94269be364ebd187a75db' move completed: OK
I20240503 08:13:58.462046 576 rebalancer_tool.cc:1074] tablet e3e58f895edb4095ba3d77653c9197af: 'b00f244c69df401997ea99d492218a98' -> '833582ff68c94269be364ebd187a75db' move scheduled
I20240503 08:13:58.532388 576 rebalancer_tool.cc:1074] tablet dcc026d6d9bf40059fd6298c40b04c1a: 'b00f244c69df401997ea99d492218a98' -> 'd09a11a03989421b9f28019dc1c14cc2' move scheduled
I20240503 08:13:58.623478 576 rebalancer_tool.cc:1074] tablet 68b1a80883614b1ca54fbd88bc769dff: 'ce1496a47d084365ad786cd09e472a2c' -> '833582ff68c94269be364ebd187a75db' move scheduled
I20240503 08:13:58.738538 576 rebalancer_tool.cc:1074] tablet e7b54127ad4245838c4cc05071378472: 'ced21ca983b749d0b4d4f6a0e2426bf8' -> 'aae860d7ae2b4002bf7794c278f9cc56' move scheduled
I20240503 08:13:58.798502 576 rebalancer_tool.cc:1074] tablet 4ebf45bdc432451e87b7c98d435a76d2: 'b00f244c69df401997ea99d492218a98' -> '833582ff68c94269be364ebd187a75db' move scheduled
I20240503 08:14:01.313236 576 rebalancer_tool.cc:1136] tablet ab7b01c2e05a430a9873398268b1e41f: 'ced21ca983b749d0b4d4f6a0e2426bf8' -> '1b411c6bb25549368041773d7b710d48' move completed: OK
I20240503 08:14:01.332551 576 rebalancer_tool.cc:1136] tablet 01c4229dfd25458296e85e412eae13bf: 'b00f244c69df401997ea99d492218a98' -> 'd09a11a03989421b9f28019dc1c14cc2' move completed: OK
I20240503 08:14:01.341647 576 rebalancer_tool.cc:1136] tablet 41e26e13355d4e0cb56bb526415a3998: 'ce1496a47d084365ad786cd09e472a2c' -> '1b411c6bb25549368041773d7b710d48' move completed: OK
I20240503 08:14:01.349511 576 rebalancer_tool.cc:1136] tablet 76d79d8be9e94e99a7ba51d903ca8a91: 'ced21ca983b749d0b4d4f6a0e2426bf8' -> '1b411c6bb25549368041773d7b710d48' move completed: OK
I20240503 08:14:01.946522 576 rebalancer_tool.cc:1136] tablet 3bc8f4968b9743baab5d25c8c7237c9c: 'b00f244c69df401997ea99d492218a98' -> 'aae860d7ae2b4002bf7794c278f9cc56' move completed: OK
I20240503 08:14:02.592569 576 rebalancer_tool.cc:1136] tablet 98b3c906cfe54f49bb88ce5e00e3cc04: 'ce1496a47d084365ad786cd09e472a2c' -> '6d1d7de19459435fbde1999e3dfd226b' move completed: OK
I20240503 08:14:02.610083 576 rebalancer_tool.cc:1136] tablet 41aa0e288ca4469ba34a07ff159ace91: '1b411c6bb25549368041773d7b710d48' -> 'ced21ca983b749d0b4d4f6a0e2426bf8' move completed: OK
I20240503 08:14:03.189782 576 rebalancer_tool.cc:1136] tablet 9fbe2eaba5344dfaa29e78a0d73f224a: '6d1d7de19459435fbde1999e3dfd226b' -> 'aae860d7ae2b4002bf7794c278f9cc56' move completed: OK
I20240503 08:14:03.214215 576 rebalancer_tool.cc:1136] tablet 1fca3d7f4b734eceb210e1186478eb72: 'aae860d7ae2b4002bf7794c278f9cc56' -> '1b411c6bb25549368041773d7b710d48' move completed: OK
I20240503 08:14:03.272526 576 rebalancer_tool.cc:1136] tablet 48075c4a45d24e84bb2b9760710de3a0: '6d1d7de19459435fbde1999e3dfd226b' -> 'ce1496a47d084365ad786cd09e472a2c' move completed: OK
I20240503 08:14:03.289546 576 rebalancer_tool.cc:1136] tablet fa7e18041c294a239eb7b8f735b66c53: '6d1d7de19459435fbde1999e3dfd226b' -> 'aae860d7ae2b4002bf7794c278f9cc56' move completed: OK
I20240503 08:14:03.299938 576 rebalancer_tool.cc:1136] tablet 200387e6edce4ee9bf11331380923005: 'ced21ca983b749d0b4d4f6a0e2426bf8' -> 'd09a11a03989421b9f28019dc1c14cc2' move completed: OK
I20240503 08:14:03.368921 576 rebalancer_tool.cc:1074] tablet f463dfe26ee94b5f9447c6dbdf8759c3: '1b411c6bb25549368041773d7b710d48' -> 'ced21ca983b749d0b4d4f6a0e2426bf8' move scheduled
I20240503 08:14:03.485725 576 rebalancer_tool.cc:1074] tablet 486e9843fd784148b9b49dfe27996dff: '6d1d7de19459435fbde1999e3dfd226b' -> 'aae860d7ae2b4002bf7794c278f9cc56' move scheduled
I20240503 08:14:03.577307 576 rebalancer_tool.cc:1074] tablet 775b8190036b4b828431fca19ce23ea9: '1b411c6bb25549368041773d7b710d48' -> 'ce1496a47d084365ad786cd09e472a2c' move scheduled
I20240503 08:14:03.645313 576 rebalancer_tool.cc:1074] tablet f83a4778ffe640ee907e0a73e58abb73: '6d1d7de19459435fbde1999e3dfd226b' -> 'aae860d7ae2b4002bf7794c278f9cc56' move scheduled
I20240503 08:14:03.746387 576 rebalancer_tool.cc:1074] tablet 978bf5783b0e47e3bfb30441adc600ae: 'd09a11a03989421b9f28019dc1c14cc2' -> 'ced21ca983b749d0b4d4f6a0e2426bf8' move scheduled
I20240503 08:14:03.822235 576 rebalancer_tool.cc:1074] tablet d9098f3d36b24d578a3f8d7d3ccad052: '1b411c6bb25549368041773d7b710d48' -> 'b00f244c69df401997ea99d492218a98' move scheduled
I20240503 08:14:03.905972 576 rebalancer_tool.cc:1074] tablet 393abed63f334ac4b71986573c86c4d4: '6d1d7de19459435fbde1999e3dfd226b' -> 'aae860d7ae2b4002bf7794c278f9cc56' move scheduled
I20240503 08:14:04.014817 576 rebalancer_tool.cc:1074] tablet 620b0cdb874d4efe8d49c971a48ee924: 'ce1496a47d084365ad786cd09e472a2c' -> 'd09a11a03989421b9f28019dc1c14cc2' move scheduled
I20240503 08:14:04.120223 576 rebalancer_tool.cc:1074] tablet f46393257ce9471891698c36a07546ad: '1b411c6bb25549368041773d7b710d48' -> 'ced21ca983b749d0b4d4f6a0e2426bf8' move scheduled
I20240503 08:14:04.223799 576 rebalancer_tool.cc:1074] tablet 658865ee3f8d424ca5cd35e12e6d7c77: '6d1d7de19459435fbde1999e3dfd226b' -> 'aae860d7ae2b4002bf7794c278f9cc56' move scheduled
I20240503 08:14:04.294410 576 rebalancer_tool.cc:1074] tablet 8c714f29abb140379b57e08eb927e2b7: 'b00f244c69df401997ea99d492218a98' -> '1b411c6bb25549368041773d7b710d48' move scheduled
I20240503 08:14:04.491227 576 rebalancer_tool.cc:1074] tablet fd6f4b81101b46588c3c7711f6bafcfa: 'ce1496a47d084365ad786cd09e472a2c' -> 'd09a11a03989421b9f28019dc1c14cc2' move scheduled
I20240503 08:14:15.068099 576 rebalancer_tool.cc:1136] tablet e7b54127ad4245838c4cc05071378472: 'ced21ca983b749d0b4d4f6a0e2426bf8' -> 'aae860d7ae2b4002bf7794c278f9cc56' move completed: OK
I20240503 08:14:15.076443 576 rebalancer_tool.cc:1136] tablet 68b1a80883614b1ca54fbd88bc769dff: 'ce1496a47d084365ad786cd09e472a2c' -> '833582ff68c94269be364ebd187a75db' move completed: OK
I20240503 08:14:15.085453 576 rebalancer_tool.cc:1136] tablet 35c62760e1de4c2f88d06b97d0286c4c: 'ce1496a47d084365ad786cd09e472a2c' -> '833582ff68c94269be364ebd187a75db' move completed: OK
I20240503 08:14:15.094347 576 rebalancer_tool.cc:1136] tablet 49b145f6946647248c967cf4badf8b40: 'd09a11a03989421b9f28019dc1c14cc2' -> '833582ff68c94269be364ebd187a75db' move completed: OK
I20240503 08:14:15.104120 576 rebalancer_tool.cc:1136] tablet dcc026d6d9bf40059fd6298c40b04c1a: 'b00f244c69df401997ea99d492218a98' -> 'd09a11a03989421b9f28019dc1c14cc2' move completed: OK
I20240503 08:14:15.113592 576 rebalancer_tool.cc:1136] tablet e3e58f895edb4095ba3d77653c9197af: 'b00f244c69df401997ea99d492218a98' -> '833582ff68c94269be364ebd187a75db' move completed: OK
I20240503 08:14:15.178776 576 rebalancer_tool.cc:1074] tablet 2401f8f101f94189aa3c0582f08cef88: 'ce1496a47d084365ad786cd09e472a2c' -> '833582ff68c94269be364ebd187a75db' move scheduled
I20240503 08:14:15.219065 576 rebalancer_tool.cc:1074] tablet 644bb089a4064ecfb88438d8b859bf01: 'd09a11a03989421b9f28019dc1c14cc2' -> '833582ff68c94269be364ebd187a75db' move scheduled
I20240503 08:14:15.302160 576 rebalancer_tool.cc:1074] tablet 03b02a4425784a30b1007b3a14139d8d: 'ced21ca983b749d0b4d4f6a0e2426bf8' -> 'aae860d7ae2b4002bf7794c278f9cc56' move scheduled
I20240503 08:14:15.355027 576 rebalancer_tool.cc:1074] tablet e30557ffd4594f7d9cda13025e4c5e60: '833582ff68c94269be364ebd187a75db' -> 'b00f244c69df401997ea99d492218a98' move scheduled
I20240503 08:14:15.420056 576 rebalancer_tool.cc:1074] tablet 71fa62206f744de0b97e3aa9ea753e37: 'd09a11a03989421b9f28019dc1c14cc2' -> 'ce1496a47d084365ad786cd09e472a2c' move scheduled
I20240503 08:14:18.963541 576 rebalancer_tool.cc:1136] tablet 658865ee3f8d424ca5cd35e12e6d7c77: '6d1d7de19459435fbde1999e3dfd226b' -> 'aae860d7ae2b4002bf7794c278f9cc56' move completed: OK
I20240503 08:14:18.976200 576 rebalancer_tool.cc:1136] tablet 620b0cdb874d4efe8d49c971a48ee924: 'ce1496a47d084365ad786cd09e472a2c' -> 'd09a11a03989421b9f28019dc1c14cc2' move completed: OK
I20240503 08:14:18.986521 576 rebalancer_tool.cc:1136] tablet fd6f4b81101b46588c3c7711f6bafcfa: 'ce1496a47d084365ad786cd09e472a2c' -> 'd09a11a03989421b9f28019dc1c14cc2' move completed: OK
I20240503 08:14:18.997169 576 rebalancer_tool.cc:1136] tablet 393abed63f334ac4b71986573c86c4d4: '6d1d7de19459435fbde1999e3dfd226b' -> 'aae860d7ae2b4002bf7794c278f9cc56' move completed: OK
I20240503 08:14:19.033681 576 rebalancer_tool.cc:1136] tablet d9098f3d36b24d578a3f8d7d3ccad052: '1b411c6bb25549368041773d7b710d48' -> 'b00f244c69df401997ea99d492218a98' move completed: OK
I20240503 08:14:19.044071 576 rebalancer_tool.cc:1136] tablet 978bf5783b0e47e3bfb30441adc600ae: 'd09a11a03989421b9f28019dc1c14cc2' -> 'ced21ca983b749d0b4d4f6a0e2426bf8' move completed: OK
I20240503 08:14:19.055047 576 rebalancer_tool.cc:1136] tablet f46393257ce9471891698c36a07546ad: '1b411c6bb25549368041773d7b710d48' -> 'ced21ca983b749d0b4d4f6a0e2426bf8' move completed: OK
I20240503 08:14:19.063576 576 rebalancer_tool.cc:1136] tablet f83a4778ffe640ee907e0a73e58abb73: '6d1d7de19459435fbde1999e3dfd226b' -> 'aae860d7ae2b4002bf7794c278f9cc56' move completed: OK
I20240503 08:14:19.073019 576 rebalancer_tool.cc:1136] tablet 775b8190036b4b828431fca19ce23ea9: '1b411c6bb25549368041773d7b710d48' -> 'ce1496a47d084365ad786cd09e472a2c' move completed: OK
I20240503 08:14:19.082160 576 rebalancer_tool.cc:1136] tablet 486e9843fd784148b9b49dfe27996dff: '6d1d7de19459435fbde1999e3dfd226b' -> 'aae860d7ae2b4002bf7794c278f9cc56' move completed: OK
I20240503 08:14:19.091667 576 rebalancer_tool.cc:1136] tablet f463dfe26ee94b5f9447c6dbdf8759c3: '1b411c6bb25549368041773d7b710d48' -> 'ced21ca983b749d0b4d4f6a0e2426bf8' move completed: OK
I20240503 08:14:20.743294 576 rebalancer_tool.cc:1136] tablet 4ebf45bdc432451e87b7c98d435a76d2: 'b00f244c69df401997ea99d492218a98' -> '833582ff68c94269be364ebd187a75db' move completed: OK
I20240503 08:14:20.810920 576 rebalancer_tool.cc:1074] tablet c68032ba399547e3bd272f424927fbce: 'aae860d7ae2b4002bf7794c278f9cc56' -> '6d1d7de19459435fbde1999e3dfd226b' move scheduled
I20240503 08:14:20.863950 576 rebalancer_tool.cc:1074] tablet 2e511a3ab0ee4eabb22240de99690603: '1b411c6bb25549368041773d7b710d48' -> 'd09a11a03989421b9f28019dc1c14cc2' move scheduled
I20240503 08:14:20.933521 576 rebalancer_tool.cc:1074] tablet fe4d26ee022b424195e836834bd2f051: 'aae860d7ae2b4002bf7794c278f9cc56' -> 'ced21ca983b749d0b4d4f6a0e2426bf8' move scheduled
I20240503 08:14:20.989688 576 rebalancer_tool.cc:1074] tablet 7083414bc16c4d499c97b93ca4e3f631: 'aae860d7ae2b4002bf7794c278f9cc56' -> '6d1d7de19459435fbde1999e3dfd226b' move scheduled
I20240503 08:14:21.068564 576 rebalancer_tool.cc:1074] tablet 6c2824aa878845e098f600f1fcfea087: 'ce1496a47d084365ad786cd09e472a2c' -> '1b411c6bb25549368041773d7b710d48' move scheduled
I20240503 08:14:21.133549 576 rebalancer_tool.cc:1074] tablet 720f36100d794158be13ff02595e6ddf: 'b00f244c69df401997ea99d492218a98' -> '1b411c6bb25549368041773d7b710d48' move scheduled
I20240503 08:14:21.192261 576 rebalancer_tool.cc:1074] tablet 45f47478f49149b895060791655503ae: 'ced21ca983b749d0b4d4f6a0e2426bf8' -> 'd09a11a03989421b9f28019dc1c14cc2' move scheduled
I20240503 08:14:21.308876 576 rebalancer_tool.cc:1074] tablet d61ad2e11c4544bfbdef7df1e92d2d11: 'aae860d7ae2b4002bf7794c278f9cc56' -> '6d1d7de19459435fbde1999e3dfd226b' move scheduled
I20240503 08:14:21.376514 576 rebalancer_tool.cc:1074] tablet 607a503b64d8440a81cf66ab158ae273: 'd09a11a03989421b9f28019dc1c14cc2' -> '833582ff68c94269be364ebd187a75db' move scheduled
I20240503 08:14:21.787930 576 rebalancer_tool.cc:1074] tablet ba1fc93781714efb8eb7ec67c2fceb8a: '1b411c6bb25549368041773d7b710d48' -> 'ce1496a47d084365ad786cd09e472a2c' move scheduled
I20240503 08:14:29.424487 576 rebalancer_tool.cc:1136] tablet e30557ffd4594f7d9cda13025e4c5e60: '833582ff68c94269be364ebd187a75db' -> 'b00f244c69df401997ea99d492218a98' move completed: OK
I20240503 08:14:30.097755 576 rebalancer_tool.cc:1136] tablet 03b02a4425784a30b1007b3a14139d8d: 'ced21ca983b749d0b4d4f6a0e2426bf8' -> 'aae860d7ae2b4002bf7794c278f9cc56' move completed: OK
I20240503 08:14:30.106868 576 rebalancer_tool.cc:1136] tablet 644bb089a4064ecfb88438d8b859bf01: 'd09a11a03989421b9f28019dc1c14cc2' -> '833582ff68c94269be364ebd187a75db' move completed: OK
I20240503 08:14:30.117344 576 rebalancer_tool.cc:1136] tablet 8c714f29abb140379b57e08eb927e2b7: 'b00f244c69df401997ea99d492218a98' -> '1b411c6bb25549368041773d7b710d48' move completed: OK
I20240503 08:14:30.821275 576 rebalancer_tool.cc:1074] tablet 97149ad1d03847a085b2142519135a17: 'b00f244c69df401997ea99d492218a98' -> 'aae860d7ae2b4002bf7794c278f9cc56' move scheduled
I20240503 08:14:30.885316 576 rebalancer_tool.cc:1074] tablet e3a6b72060804beb81a177988d287f63: 'd09a11a03989421b9f28019dc1c14cc2' -> 'ce1496a47d084365ad786cd09e472a2c' move scheduled
I20240503 08:14:33.445629 576 rebalancer_tool.cc:1136] tablet d61ad2e11c4544bfbdef7df1e92d2d11: 'aae860d7ae2b4002bf7794c278f9cc56' -> '6d1d7de19459435fbde1999e3dfd226b' move completed: OK
I20240503 08:14:33.453473 576 rebalancer_tool.cc:1136] tablet 45f47478f49149b895060791655503ae: 'ced21ca983b749d0b4d4f6a0e2426bf8' -> 'd09a11a03989421b9f28019dc1c14cc2' move completed: OK
I20240503 08:14:33.465044 576 rebalancer_tool.cc:1136] tablet 720f36100d794158be13ff02595e6ddf: 'b00f244c69df401997ea99d492218a98' -> '1b411c6bb25549368041773d7b710d48' move completed: OK
I20240503 08:14:35.096617 576 rebalancer_tool.cc:1136] tablet 6c2824aa878845e098f600f1fcfea087: 'ce1496a47d084365ad786cd09e472a2c' -> '1b411c6bb25549368041773d7b710d48' move completed: OK
I20240503 08:14:35.117213 576 rebalancer_tool.cc:1136] tablet 7083414bc16c4d499c97b93ca4e3f631: 'aae860d7ae2b4002bf7794c278f9cc56' -> '6d1d7de19459435fbde1999e3dfd226b' move completed: OK
I20240503 08:14:35.133956 576 rebalancer_tool.cc:1136] tablet fe4d26ee022b424195e836834bd2f051: 'aae860d7ae2b4002bf7794c278f9cc56' -> 'ced21ca983b749d0b4d4f6a0e2426bf8' move completed: OK
I20240503 08:14:35.153131 576 rebalancer_tool.cc:1136] tablet 2e511a3ab0ee4eabb22240de99690603: '1b411c6bb25549368041773d7b710d48' -> 'd09a11a03989421b9f28019dc1c14cc2' move completed: OK
I20240503 08:14:35.162961 576 rebalancer_tool.cc:1136] tablet c68032ba399547e3bd272f424927fbce: 'aae860d7ae2b4002bf7794c278f9cc56' -> '6d1d7de19459435fbde1999e3dfd226b' move completed: OK
I20240503 08:14:35.172441 576 rebalancer_tool.cc:1136] tablet 71fa62206f744de0b97e3aa9ea753e37: 'd09a11a03989421b9f28019dc1c14cc2' -> 'ce1496a47d084365ad786cd09e472a2c' move completed: OK
I20240503 08:14:35.183490 576 rebalancer_tool.cc:1136] tablet ba1fc93781714efb8eb7ec67c2fceb8a: '1b411c6bb25549368041773d7b710d48' -> 'ce1496a47d084365ad786cd09e472a2c' move completed: OK
I20240503 08:14:35.191643 576 rebalancer_tool.cc:1136] tablet 2401f8f101f94189aa3c0582f08cef88: 'ce1496a47d084365ad786cd09e472a2c' -> '833582ff68c94269be364ebd187a75db' move completed: OK
I20240503 08:14:36.219348 576 rebalancer_tool.cc:1074] tablet d94f5824a71f454f8a1bfa13dd032684: 'aae860d7ae2b4002bf7794c278f9cc56' -> 'ced21ca983b749d0b4d4f6a0e2426bf8' move scheduled
I20240503 08:14:37.279166 576 rebalancer_tool.cc:1074] tablet 099e37089b2e42b4839e681f8f37365d: 'aae860d7ae2b4002bf7794c278f9cc56' -> '6d1d7de19459435fbde1999e3dfd226b' move scheduled
I20240503 08:14:38.392443 576 rebalancer_tool.cc:1074] tablet 33f0f4bc5722485a8f978a02367ca80e: '1b411c6bb25549368041773d7b710d48' -> 'ce1496a47d084365ad786cd09e472a2c' move scheduled
I20240503 08:14:38.500846 576 rebalancer_tool.cc:1074] tablet f1b2f3b8111649a7b5725647b8251698: 'b00f244c69df401997ea99d492218a98' -> 'aae860d7ae2b4002bf7794c278f9cc56' move scheduled
I20240503 08:14:39.548880 576 rebalancer_tool.cc:1074] tablet bab5c25c31a64b578489b1006f631cc0: 'aae860d7ae2b4002bf7794c278f9cc56' -> 'ced21ca983b749d0b4d4f6a0e2426bf8' move scheduled
I20240503 08:14:39.637657 576 rebalancer_tool.cc:1074] tablet 23fc18f9417148c2a6d451bcca395338: '1b411c6bb25549368041773d7b710d48' -> 'd09a11a03989421b9f28019dc1c14cc2' move scheduled
I20240503 08:14:40.712441 576 rebalancer_tool.cc:1074] tablet 07866de372e541ec9dca8a3d42f8b0ab: '1b411c6bb25549368041773d7b710d48' -> 'ce1496a47d084365ad786cd09e472a2c' move scheduled
I20240503 08:14:40.830382 576 rebalancer_tool.cc:1074] tablet 91c573181144456aa7e27a3026a639c4: 'd09a11a03989421b9f28019dc1c14cc2' -> '1b411c6bb25549368041773d7b710d48' move scheduled
I20240503 08:14:40.883234 576 rebalancer_tool.cc:1074] tablet 2c77993083a049668875b1ecd5a893ec: 'd09a11a03989421b9f28019dc1c14cc2' -> '1b411c6bb25549368041773d7b710d48' move scheduled
I20240503 08:14:43.283193 576 rebalancer_tool.cc:1123] tablet bab5c25c31a64b578489b1006f631cc0: 'aae860d7ae2b4002bf7794c278f9cc56' -> 'ced21ca983b749d0b4d4f6a0e2426bf8' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:43.716557 576 rebalancer_tool.cc:1123] tablet 099e37089b2e42b4839e681f8f37365d: 'aae860d7ae2b4002bf7794c278f9cc56' -> '6d1d7de19459435fbde1999e3dfd226b' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:43.720515 576 rebalancer_tool.cc:1123] tablet d94f5824a71f454f8a1bfa13dd032684: 'aae860d7ae2b4002bf7794c278f9cc56' -> 'ced21ca983b749d0b4d4f6a0e2426bf8' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:45.544862 576 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): unacceptable health status UNAVAILABLE
I20240503 08:14:46.653385 21401 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8bBFGA/build/asan/bin/kudu pid 27921
I20240503 08:14:46.655954 13421 raft_consensus.cc:482] T d94f5824a71f454f8a1bfa13dd032684 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:46.656118 13421 raft_consensus.cc:504] T d94f5824a71f454f8a1bfa13dd032684 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
W20240503 08:14:46.656515 28327 consensus_peers.cc:485] T 599cb18c58a74e93a3e0334cc43a73d4 P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:14:46.656911 13421 leader_election.cc:288] T d94f5824a71f454f8a1bfa13dd032684 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:14:46.657316 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d94f5824a71f454f8a1bfa13dd032684" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:14:46.657490 28391 raft_consensus.cc:2454] T d94f5824a71f454f8a1bfa13dd032684 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 1.
W20240503 08:14:46.658663 30960 leader_election.cc:334] T d94f5824a71f454f8a1bfa13dd032684 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:46.659947 30961 leader_election.cc:334] T d94f5824a71f454f8a1bfa13dd032684 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:46.660050 30961 leader_election.cc:302] T d94f5824a71f454f8a1bfa13dd032684 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:46.660303 13421 raft_consensus.cc:2726] T d94f5824a71f454f8a1bfa13dd032684 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 08:14:46.670526 30828 consensus_peers.cc:485] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:14:46.677170 28043 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:14:46.692543 13336 raft_consensus.cc:482] T fa7e18041c294a239eb7b8f735b66c53 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:46.692698 13336 raft_consensus.cc:504] T fa7e18041c294a239eb7b8f735b66c53 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:46.693310 13336 leader_election.cc:288] T fa7e18041c294a239eb7b8f735b66c53 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
W20240503 08:14:46.694456 30829 leader_election.cc:334] T fa7e18041c294a239eb7b8f735b66c53 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
W20240503 08:14:46.694862 30828 leader_election.cc:334] T fa7e18041c294a239eb7b8f735b66c53 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:46.694967 30828 leader_election.cc:302] T fa7e18041c294a239eb7b8f735b66c53 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:46.695191 13336 raft_consensus.cc:2726] T fa7e18041c294a239eb7b8f735b66c53 P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 08:14:46.720085 28328 consensus_peers.cc:485] T 1fca3d7f4b734eceb210e1186478eb72 P d09a11a03989421b9f28019dc1c14cc2 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:46.737715 28327 consensus_peers.cc:485] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:14:46.740324 30696 consensus_peers.cc:462] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
W20240503 08:14:46.787953 30696 consensus_peers.cc:485] T 41e26e13355d4e0cb56bb526415a3998 P 6d1d7de19459435fbde1999e3dfd226b -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:14:46.801198 13421 raft_consensus.cc:482] T bab5c25c31a64b578489b1006f631cc0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:46.801378 13421 raft_consensus.cc:504] T bab5c25c31a64b578489b1006f631cc0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:46.802124 13421 leader_election.cc:288] T bab5c25c31a64b578489b1006f631cc0 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:14:46.803400 28391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab5c25c31a64b578489b1006f631cc0" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d09a11a03989421b9f28019dc1c14cc2" is_pre_election: true
I20240503 08:14:46.803613 28391 raft_consensus.cc:2454] T bab5c25c31a64b578489b1006f631cc0 P d09a11a03989421b9f28019dc1c14cc2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced21ca983b749d0b4d4f6a0e2426bf8 in term 1.
W20240503 08:14:46.804139 30960 leader_election.cc:334] T bab5c25c31a64b578489b1006f631cc0 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:46.804512 30961 leader_election.cc:334] T bab5c25c31a64b578489b1006f631cc0 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:46.804623 30961 leader_election.cc:302] T bab5c25c31a64b578489b1006f631cc0 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8, d09a11a03989421b9f28019dc1c14cc2; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:46.804831 13421 raft_consensus.cc:2726] T bab5c25c31a64b578489b1006f631cc0 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 08:14:46.807917 30696 consensus_peers.cc:485] T 607a503b64d8440a81cf66ab158ae273 P 6d1d7de19459435fbde1999e3dfd226b -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:46.824178 31096 consensus_peers.cc:485] T 23fc18f9417148c2a6d451bcca395338 P b00f244c69df401997ea99d492218a98 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:46.841184 30828 consensus_peers.cc:485] T 045b3ce4b1d44f94a802b41a12e2704d P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:14:46.853358 28328 consensus_peers.cc:485] T 76d79d8be9e94e99a7ba51d903ca8a91 P d09a11a03989421b9f28019dc1c14cc2 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:46.888341 28328 consensus_peers.cc:485] T 720f36100d794158be13ff02595e6ddf P d09a11a03989421b9f28019dc1c14cc2 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:46.892017 30696 consensus_peers.cc:485] T 2222ff7fb5184d6c86010e1475d6c1f7 P 6d1d7de19459435fbde1999e3dfd226b -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:46.899262 28328 consensus_peers.cc:485] T ab7b01c2e05a430a9873398268b1e41f P d09a11a03989421b9f28019dc1c14cc2 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:46.942262 28328 consensus_peers.cc:485] T 03b02a4425784a30b1007b3a14139d8d P d09a11a03989421b9f28019dc1c14cc2 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:46.950367 30961 consensus_peers.cc:485] T 4ed8026daee840c6b31ca7d989ed25f0 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:14:46.967217 13601 raft_consensus.cc:482] T 978bf5783b0e47e3bfb30441adc600ae P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:46.967391 13601 raft_consensus.cc:504] T 978bf5783b0e47e3bfb30441adc600ae P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:46.968339 13601 leader_election.cc:288] T 978bf5783b0e47e3bfb30441adc600ae P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
W20240503 08:14:46.969542 30960 leader_election.cc:334] T 978bf5783b0e47e3bfb30441adc600ae P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:46.969727 30961 leader_election.cc:334] T 978bf5783b0e47e3bfb30441adc600ae P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
I20240503 08:14:46.969847 30961 leader_election.cc:302] T 978bf5783b0e47e3bfb30441adc600ae P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:46.970204 13601 raft_consensus.cc:2726] T 978bf5783b0e47e3bfb30441adc600ae P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:14:46.982792 28328 consensus_peers.cc:485] T 8bcbc022caa94bb08a2db4de9b060557 P d09a11a03989421b9f28019dc1c14cc2 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:46.988809 28327 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111) [suppressed 231 similar messages]
W20240503 08:14:46.989569 28327 consensus_peers.cc:485] T 775b8190036b4b828431fca19ce23ea9 P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:14:46.995019 30828 consensus_peers.cc:485] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:14:47.014719 30961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111) [suppressed 127 similar messages]
W20240503 08:14:47.015667 30961 consensus_peers.cc:485] T 74f43feb3bc342a18723bc5a143bdd99 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:47.036640 30696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111) [suppressed 115 similar messages]
W20240503 08:14:47.037269 30828 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111) [suppressed 152 similar messages]
W20240503 08:14:47.038154 30828 consensus_peers.cc:462] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:47.050669 31095 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240503 08:14:47.051534 31095 consensus_peers.cc:485] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:14:47.089668 27937 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111) [suppressed 221 similar messages]
W20240503 08:14:47.092206 31095 consensus_peers.cc:485] T 2c595805729d48fd974d761ed8a60158 P b00f244c69df401997ea99d492218a98 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:14:47.094877 27937 consensus_peers.cc:485] T 7c8c7659181040cabf20ae116fae78c8 P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:14:47.097374 27937 consensus_peers.cc:485] T e0b904461c1c44ed93e1a227d93498eb P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:14:47.097483 27937 consensus_peers.cc:485] T 9fbe2eaba5344dfaa29e78a0d73f224a P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:14:47.098760 27937 consensus_peers.cc:485] T 3d0498256b5f4b8fb1681759429f10ed P 833582ff68c94269be364ebd187a75db -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:14:47.104035 30960 consensus_peers.cc:485] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:14:47.104631 31096 consensus_peers.cc:485] T 2c595805729d48fd974d761ed8a60158 P b00f244c69df401997ea99d492218a98 -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:47.106180 27938 consensus_peers.cc:485] T c68032ba399547e3bd272f424927fbce P 833582ff68c94269be364ebd187a75db -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:14:47.112614 21401 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8bBFGA/build/asan/bin/kudu with pid 27921
W20240503 08:14:47.142783 28327 consensus_peers.cc:485] T 33f0f4bc5722485a8f978a02367ca80e P d09a11a03989421b9f28019dc1c14cc2 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:14:47.155162 30696 consensus_peers.cc:462] T 2c77993083a049668875b1ecd5a893ec P 6d1d7de19459435fbde1999e3dfd226b -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
W20240503 08:14:47.169034 30697 connection.cc:535] client connection to 127.20.230.65:39081 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:14:47.169567 30830 connection.cc:535] client connection to 127.20.230.65:39081 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:14:47.169941 21401 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8bBFGA/build/asan/bin/kudu pid 28311
W20240503 08:14:47.170519 30697 consensus_peers.cc:485] T f9b4f775b9914bdc98b9b582072cb1ab P 6d1d7de19459435fbde1999e3dfd226b -> Peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Couldn't send request to peer 833582ff68c94269be364ebd187a75db. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.65:39081: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:47.170644 30697 consensus_peers.cc:485] T e3e58f895edb4095ba3d77653c9197af P 6d1d7de19459435fbde1999e3dfd226b -> Peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Couldn't send request to peer 833582ff68c94269be364ebd187a75db. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.65:39081: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:47.170527 30830 consensus_peers.cc:485] T c9055668b86e4dc5aa321a4d29beaf74 P ce1496a47d084365ad786cd09e472a2c -> Peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Couldn't send request to peer 833582ff68c94269be364ebd187a75db. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.65:39081: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:47.171025 30830 consensus_peers.cc:485] T b016e31e76854807af0cfaed6ec49071 P ce1496a47d084365ad786cd09e472a2c -> Peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Couldn't send request to peer 833582ff68c94269be364ebd187a75db. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.65:39081: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:47.171113 30830 consensus_peers.cc:485] T 0f25688be4e74d31aaab85b7a4329a34 P ce1496a47d084365ad786cd09e472a2c -> Peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Couldn't send request to peer 833582ff68c94269be364ebd187a75db. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.65:39081: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:47.175503 28433 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 08:14:47.183684 30828 consensus_peers.cc:485] T f53acf725b62447382e9bae95e6e74d5 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:14:47.206610 30829 consensus_peers.cc:485] T 7815cb2db9514c02b0ee3e3cd12b2212 P ce1496a47d084365ad786cd09e472a2c -> Peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Couldn't send request to peer 1b411c6bb25549368041773d7b710d48. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:14:47.214982 30697 consensus_peers.cc:485] T f9746769e2514c65ac8d48ca7df8bca7 P 6d1d7de19459435fbde1999e3dfd226b -> Peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Couldn't send request to peer 833582ff68c94269be364ebd187a75db. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.65:39081: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:47.242666 31097 consensus_peers.cc:485] T 644bb089a4064ecfb88438d8b859bf01 P b00f244c69df401997ea99d492218a98 -> Peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Couldn't send request to peer 833582ff68c94269be364ebd187a75db. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.65:39081: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:47.255451 30960 consensus_peers.cc:485] T 393abed63f334ac4b71986573c86c4d4 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:14:47.303246 30830 consensus_peers.cc:485] T e7b54127ad4245838c4cc05071378472 P ce1496a47d084365ad786cd09e472a2c -> Peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Couldn't send request to peer 833582ff68c94269be364ebd187a75db. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.65:39081: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:47.322911 31097 consensus_peers.cc:485] T b40a43ad7ed44c30a070189ebfe5aab6 P b00f244c69df401997ea99d492218a98 -> Peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Couldn't send request to peer 833582ff68c94269be364ebd187a75db. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.65:39081: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:47.325861 30962 consensus_peers.cc:485] T fe4d26ee022b424195e836834bd2f051 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Couldn't send request to peer 833582ff68c94269be364ebd187a75db. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.65:39081: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:47.394620 30830 consensus_peers.cc:485] T e3a6b72060804beb81a177988d287f63 P ce1496a47d084365ad786cd09e472a2c -> Peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Couldn't send request to peer 833582ff68c94269be364ebd187a75db. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.65:39081: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:47.414359 30697 consensus_peers.cc:485] T 01c4229dfd25458296e85e412eae13bf P 6d1d7de19459435fbde1999e3dfd226b -> Peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Couldn't send request to peer 833582ff68c94269be364ebd187a75db. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.65:39081: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:47.425930 13601 raft_consensus.cc:482] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:47.426082 13601 raft_consensus.cc:504] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:47.426625 13601 leader_election.cc:288] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
W20240503 08:14:47.427891 30961 leader_election.cc:334] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
W20240503 08:14:47.428067 30960 leader_election.cc:334] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:47.428194 30960 leader_election.cc:302] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:47.428427 13601 raft_consensus.cc:2726] T de5dfccb0359462dbb7da776293fa679 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:14:47.447530 31097 consensus_peers.cc:485] T 6a96965691644db8b9461d146536a026 P b00f244c69df401997ea99d492218a98 -> Peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Couldn't send request to peer 833582ff68c94269be364ebd187a75db. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.65:39081: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:47.471820 30697 consensus_peers.cc:485] T 4ebf45bdc432451e87b7c98d435a76d2 P 6d1d7de19459435fbde1999e3dfd226b -> Peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Couldn't send request to peer 833582ff68c94269be364ebd187a75db. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.65:39081: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:47.521245 13421 raft_consensus.cc:482] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:47.521436 13421 raft_consensus.cc:504] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:47.522291 13421 leader_election.cc:288] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
W20240503 08:14:47.524133 30828 consensus_peers.cc:462] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:47.524221 30961 leader_election.cc:334] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
W20240503 08:14:47.525027 30960 leader_election.cc:334] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:47.525152 30960 leader_election.cc:302] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:47.525522 13601 raft_consensus.cc:2726] T 17c6e3293aad4591a9c1395ebffff2e6 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240503 08:14:47.527387 30830 consensus_peers.cc:485] T 49b145f6946647248c967cf4badf8b40 P ce1496a47d084365ad786cd09e472a2c -> Peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Couldn't send request to peer 833582ff68c94269be364ebd187a75db. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.65:39081: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:47.564723 21401 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8bBFGA/build/asan/bin/kudu with pid 28311
W20240503 08:14:47.567545 30697 consensus_peers.cc:485] T 607a503b64d8440a81cf66ab158ae273 P 6d1d7de19459435fbde1999e3dfd226b -> Peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Couldn't send request to peer 833582ff68c94269be364ebd187a75db. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.65:39081: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:47.585608 31097 consensus_peers.cc:485] T 2401f8f101f94189aa3c0582f08cef88 P b00f244c69df401997ea99d492218a98 -> Peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Couldn't send request to peer 833582ff68c94269be364ebd187a75db. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.65:39081: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:47.611726 30962 consensus_peers.cc:485] T 1b53494abdd349158c897af8379ac0b0 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Couldn't send request to peer 833582ff68c94269be364ebd187a75db. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.65:39081: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:47.620265 30697 connection.cc:535] server connection from 127.20.230.68:50161 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:14:47.620288 30962 connection.cc:535] client connection to 127.20.230.68:46211 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:14:47.620862 30830 connection.cc:535] server connection from 127.20.230.68:45821 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:14:47.621225 21401 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8bBFGA/build/asan/bin/kudu pid 30677
W20240503 08:14:47.622186 30962 consensus_peers.cc:485] T 1e8dd968fc6c4416ae57cfcf7d8b4693 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): Couldn't send request to peer d09a11a03989421b9f28019dc1c14cc2. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.68:46211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:47.622308 30962 consensus_peers.cc:485] T 82591d648f3f4a55b32ed8e09490953d P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): Couldn't send request to peer d09a11a03989421b9f28019dc1c14cc2. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.68:46211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:47.622411 30962 consensus_peers.cc:485] T 41aa0e288ca4469ba34a07ff159ace91 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): Couldn't send request to peer d09a11a03989421b9f28019dc1c14cc2. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.68:46211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:47.622489 30962 consensus_peers.cc:485] T b12ceaf7f8d04fad8b07b36459da7116 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): Couldn't send request to peer d09a11a03989421b9f28019dc1c14cc2. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.68:46211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:47.622584 30962 consensus_peers.cc:485] T 4ed8026daee840c6b31ca7d989ed25f0 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): Couldn't send request to peer d09a11a03989421b9f28019dc1c14cc2. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.68:46211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:47.622695 30962 consensus_peers.cc:485] T 2c8df43c895d49cfa758eb04382261d4 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): Couldn't send request to peer d09a11a03989421b9f28019dc1c14cc2. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.68:46211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:47.622884 31097 consensus_peers.cc:485] T 620b0cdb874d4efe8d49c971a48ee924 P b00f244c69df401997ea99d492218a98 -> Peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): Couldn't send request to peer d09a11a03989421b9f28019dc1c14cc2. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.68:46211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:47.622592 30961 connection.cc:535] server connection from 127.20.230.68:37753 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:14:47.623028 31097 consensus_peers.cc:485] T fd6f4b81101b46588c3c7711f6bafcfa P b00f244c69df401997ea99d492218a98 -> Peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): Couldn't send request to peer d09a11a03989421b9f28019dc1c14cc2. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.68:46211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:47.623095 30697 consensus_peers.cc:485] T 01c4229dfd25458296e85e412eae13bf P 6d1d7de19459435fbde1999e3dfd226b -> Peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): Couldn't send request to peer d09a11a03989421b9f28019dc1c14cc2. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.68:46211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:47.623149 31097 consensus_peers.cc:485] T 200387e6edce4ee9bf11331380923005 P b00f244c69df401997ea99d492218a98 -> Peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): Couldn't send request to peer d09a11a03989421b9f28019dc1c14cc2. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.68:46211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:47.623220 30697 consensus_peers.cc:485] T 607a503b64d8440a81cf66ab158ae273 P 6d1d7de19459435fbde1999e3dfd226b -> Peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): Couldn't send request to peer d09a11a03989421b9f28019dc1c14cc2. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.68:46211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:47.623247 31097 consensus_peers.cc:485] T 013e00c67a1c4ae9b3c1a050b966e9a5 P b00f244c69df401997ea99d492218a98 -> Peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): Couldn't send request to peer d09a11a03989421b9f28019dc1c14cc2. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.68:46211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:47.623328 30697 consensus_peers.cc:485] T 2e511a3ab0ee4eabb22240de99690603 P 6d1d7de19459435fbde1999e3dfd226b -> Peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): Couldn't send request to peer d09a11a03989421b9f28019dc1c14cc2. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.68:46211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:47.623333 31097 consensus_peers.cc:485] T f463dfe26ee94b5f9447c6dbdf8759c3 P b00f244c69df401997ea99d492218a98 -> Peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): Couldn't send request to peer d09a11a03989421b9f28019dc1c14cc2. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.68:46211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:47.623420 31097 consensus_peers.cc:485] T 23fc18f9417148c2a6d451bcca395338 P b00f244c69df401997ea99d492218a98 -> Peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): Couldn't send request to peer d09a11a03989421b9f28019dc1c14cc2. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.68:46211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:47.627583 30803 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:14:47.640125 13601 raft_consensus.cc:482] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:47.640295 13601 raft_consensus.cc:504] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:47.640859 13601 leader_election.cc:288] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
W20240503 08:14:47.641947 30961 leader_election.cc:334] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
W20240503 08:14:47.642297 30960 leader_election.cc:334] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:47.642401 30960 leader_election.cc:302] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:47.642650 13601 raft_consensus.cc:2726] T 8450e62352a74d2ea1471166d4cb1132 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:14:47.643882 30830 consensus_peers.cc:485] T 62ca502a089f4a37999d2252776b741b P ce1496a47d084365ad786cd09e472a2c -> Peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Couldn't send request to peer 833582ff68c94269be364ebd187a75db. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.65:39081: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:47.753854 30830 consensus_peers.cc:485] T 7a101bee3af34166848ee7fa7c733892 P ce1496a47d084365ad786cd09e472a2c -> Peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): Couldn't send request to peer d09a11a03989421b9f28019dc1c14cc2. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.68:46211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:47.770251 30830 consensus_peers.cc:485] T c9055668b86e4dc5aa321a4d29beaf74 P ce1496a47d084365ad786cd09e472a2c -> Peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): Couldn't send request to peer d09a11a03989421b9f28019dc1c14cc2. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.68:46211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:47.822790 30830 consensus_peers.cc:485] T d61ad2e11c4544bfbdef7df1e92d2d11 P ce1496a47d084365ad786cd09e472a2c -> Peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): Couldn't send request to peer d09a11a03989421b9f28019dc1c14cc2. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.68:46211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:47.878433 30830 consensus_peers.cc:485] T 3131a638a8bc4030ab93c9b4b9173b94 P ce1496a47d084365ad786cd09e472a2c -> Peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): Couldn't send request to peer d09a11a03989421b9f28019dc1c14cc2. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.68:46211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:47.889165 30830 consensus_peers.cc:485] T d8c0fcef461446eabc1e55f434a2ce9f P ce1496a47d084365ad786cd09e472a2c -> Peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): Couldn't send request to peer d09a11a03989421b9f28019dc1c14cc2. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.68:46211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:47.926050 30830 consensus_peers.cc:485] T df31bd9e544a44f280db2b0d021000c2 P ce1496a47d084365ad786cd09e472a2c -> Peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): Couldn't send request to peer d09a11a03989421b9f28019dc1c14cc2. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.68:46211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:47.946976 30960 consensus_peers.cc:485] T 41aa0e288ca4469ba34a07ff159ace91 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:14:47.965756 21401 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8bBFGA/build/asan/bin/kudu with pid 30677
W20240503 08:14:47.972276 30828 consensus_peers.cc:462] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:48.012645 13599 raft_consensus.cc:482] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:48.012806 13599 raft_consensus.cc:504] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:48.013389 13599 leader_election.cc:288] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
W20240503 08:14:48.014950 30829 leader_election.cc:334] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
W20240503 08:14:48.015112 30828 leader_election.cc:334] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:48.015273 30828 leader_election.cc:302] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce1496a47d084365ad786cd09e472a2c; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:48.015554 13599 raft_consensus.cc:2726] T f7a8c95ac81942dabef98a07caa84082 P ce1496a47d084365ad786cd09e472a2c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:14:48.017799 30829 connection.cc:535] client connection to 127.20.230.69:34725 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:14:48.018576 21401 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8bBFGA/build/asan/bin/kudu pid 30812
W20240503 08:14:48.018724 30829 consensus_peers.cc:485] T 97149ad1d03847a085b2142519135a17 P ce1496a47d084365ad786cd09e472a2c -> Peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): Couldn't send request to peer 6d1d7de19459435fbde1999e3dfd226b. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.69:34725: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:48.018833 30829 consensus_peers.cc:485] T 3bc8f4968b9743baab5d25c8c7237c9c P ce1496a47d084365ad786cd09e472a2c -> Peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): Couldn't send request to peer 6d1d7de19459435fbde1999e3dfd226b. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.69:34725: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:48.018916 30829 consensus_peers.cc:485] T d8c0fcef461446eabc1e55f434a2ce9f P ce1496a47d084365ad786cd09e472a2c -> Peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): Couldn't send request to peer 6d1d7de19459435fbde1999e3dfd226b. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.69:34725: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:48.019002 30829 consensus_peers.cc:485] T df31bd9e544a44f280db2b0d021000c2 P ce1496a47d084365ad786cd09e472a2c -> Peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): Couldn't send request to peer 6d1d7de19459435fbde1999e3dfd226b. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.69:34725: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:48.019102 30829 consensus_peers.cc:485] T 7815cb2db9514c02b0ee3e3cd12b2212 P ce1496a47d084365ad786cd09e472a2c -> Peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): Couldn't send request to peer 6d1d7de19459435fbde1999e3dfd226b. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.69:34725: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:48.019196 30829 consensus_peers.cc:485] T d61ad2e11c4544bfbdef7df1e92d2d11 P ce1496a47d084365ad786cd09e472a2c -> Peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): Couldn't send request to peer 6d1d7de19459435fbde1999e3dfd226b. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.69:34725: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:48.019291 30829 consensus_peers.cc:485] T ebe1a0b781404070a5326a80f8f9bfb4 P ce1496a47d084365ad786cd09e472a2c -> Peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): Couldn't send request to peer 6d1d7de19459435fbde1999e3dfd226b. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.69:34725: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:48.019394 30829 consensus_peers.cc:485] T a829381b44c3494ab29ddcf21df2f9e1 P ce1496a47d084365ad786cd09e472a2c -> Peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): Couldn't send request to peer 6d1d7de19459435fbde1999e3dfd226b. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.69:34725: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:48.019488 30829 consensus_peers.cc:485] T e3a6b72060804beb81a177988d287f63 P ce1496a47d084365ad786cd09e472a2c -> Peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): Couldn't send request to peer 6d1d7de19459435fbde1999e3dfd226b. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.69:34725: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:48.019702 31096 connection.cc:535] client connection to 127.20.230.69:34725 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:14:48.020633 31096 consensus_peers.cc:485] T fd6f4b81101b46588c3c7711f6bafcfa P b00f244c69df401997ea99d492218a98 -> Peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): Couldn't send request to peer 6d1d7de19459435fbde1999e3dfd226b. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.69:34725: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:48.020757 31096 consensus_peers.cc:485] T 549f64ad9ef64a3da619c5df5a411e1c P b00f244c69df401997ea99d492218a98 -> Peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): Couldn't send request to peer 6d1d7de19459435fbde1999e3dfd226b. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.69:34725: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:48.020860 31096 consensus_peers.cc:485] T e8bc656988c44600a198850c604e0b64 P b00f244c69df401997ea99d492218a98 -> Peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): Couldn't send request to peer 6d1d7de19459435fbde1999e3dfd226b. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.69:34725: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:48.020967 31096 consensus_peers.cc:485] T b40a43ad7ed44c30a070189ebfe5aab6 P b00f244c69df401997ea99d492218a98 -> Peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): Couldn't send request to peer 6d1d7de19459435fbde1999e3dfd226b. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.69:34725: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:48.021070 31096 consensus_peers.cc:485] T 9fc39354ba064392be1af1769513f738 P b00f244c69df401997ea99d492218a98 -> Peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): Couldn't send request to peer 6d1d7de19459435fbde1999e3dfd226b. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.69:34725: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:48.025403 30961 consensus_peers.cc:485] T 1b53494abdd349158c897af8379ac0b0 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): Couldn't send request to peer 6d1d7de19459435fbde1999e3dfd226b. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.69:34725: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:48.026520 30934 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 08:14:48.052683 31096 consensus_peers.cc:485] T 98b3c906cfe54f49bb88ce5e00e3cc04 P b00f244c69df401997ea99d492218a98 -> Peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): Couldn't send request to peer 6d1d7de19459435fbde1999e3dfd226b. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.69:34725: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:48.082180 13503 raft_consensus.cc:482] T d9098f3d36b24d578a3f8d7d3ccad052 P b00f244c69df401997ea99d492218a98 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 833582ff68c94269be364ebd187a75db)
I20240503 08:14:48.082355 13503 raft_consensus.cc:504] T d9098f3d36b24d578a3f8d7d3ccad052 P b00f244c69df401997ea99d492218a98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:14:48.082969 13503 leader_election.cc:288] T d9098f3d36b24d578a3f8d7d3ccad052 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
W20240503 08:14:48.084146 31095 leader_election.cc:334] T d9098f3d36b24d578a3f8d7d3ccad052 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:48.084297 31097 leader_election.cc:334] T d9098f3d36b24d578a3f8d7d3ccad052 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Network error: Client connection negotiation failed: client connection to 127.20.230.65:39081: connect: Connection refused (error 111)
I20240503 08:14:48.084409 31097 leader_election.cc:302] T d9098f3d36b24d578a3f8d7d3ccad052 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b00f244c69df401997ea99d492218a98; no voters: 833582ff68c94269be364ebd187a75db, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:48.084669 13503 raft_consensus.cc:2726] T d9098f3d36b24d578a3f8d7d3ccad052 P b00f244c69df401997ea99d492218a98 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 08:14:48.132119 13421 raft_consensus.cc:482] T dcc026d6d9bf40059fd6298c40b04c1a P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 833582ff68c94269be364ebd187a75db)
I20240503 08:14:48.132310 13421 raft_consensus.cc:504] T dcc026d6d9bf40059fd6298c40b04c1a P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:14:48.132992 13421 leader_election.cc:288] T dcc026d6d9bf40059fd6298c40b04c1a P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
W20240503 08:14:48.134361 30962 leader_election.cc:334] T dcc026d6d9bf40059fd6298c40b04c1a P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Network error: Client connection negotiation failed: client connection to 127.20.230.65:39081: connect: Connection refused (error 111)
W20240503 08:14:48.134548 30962 leader_election.cc:334] T dcc026d6d9bf40059fd6298c40b04c1a P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): Network error: Client connection negotiation failed: client connection to 127.20.230.68:46211: connect: Connection refused (error 111)
I20240503 08:14:48.134652 30962 leader_election.cc:302] T dcc026d6d9bf40059fd6298c40b04c1a P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8; no voters: 833582ff68c94269be364ebd187a75db, d09a11a03989421b9f28019dc1c14cc2
I20240503 08:14:48.134877 13421 raft_consensus.cc:2726] T dcc026d6d9bf40059fd6298c40b04c1a P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:14:48.163007 13421 raft_consensus.cc:482] T 7e0350bcc5fa4051bd10cc3de09639aa P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 833582ff68c94269be364ebd187a75db)
I20240503 08:14:48.163180 13421 raft_consensus.cc:504] T 7e0350bcc5fa4051bd10cc3de09639aa P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:14:48.163861 13421 leader_election.cc:288] T 7e0350bcc5fa4051bd10cc3de09639aa P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:14:48.164197 31161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e0350bcc5fa4051bd10cc3de09639aa" candidate_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b00f244c69df401997ea99d492218a98" is_pre_election: true
W20240503 08:14:48.165163 30962 leader_election.cc:334] T 7e0350bcc5fa4051bd10cc3de09639aa P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Network error: Client connection negotiation failed: client connection to 127.20.230.65:39081: connect: Connection refused (error 111)
I20240503 08:14:48.165271 30962 leader_election.cc:302] T 7e0350bcc5fa4051bd10cc3de09639aa P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8; no voters: 833582ff68c94269be364ebd187a75db, b00f244c69df401997ea99d492218a98
I20240503 08:14:48.165513 13421 raft_consensus.cc:2726] T 7e0350bcc5fa4051bd10cc3de09639aa P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:14:48.307130 13421 raft_consensus.cc:482] T d94f5824a71f454f8a1bfa13dd032684 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b411c6bb25549368041773d7b710d48)
I20240503 08:14:48.307358 13421 raft_consensus.cc:504] T d94f5824a71f454f8a1bfa13dd032684 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:48.308075 13421 leader_election.cc:288] T d94f5824a71f454f8a1bfa13dd032684 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
W20240503 08:14:48.309196 30961 leader_election.cc:334] T d94f5824a71f454f8a1bfa13dd032684 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143): Network error: Client connection negotiation failed: client connection to 127.20.230.66:37143: connect: Connection refused (error 111)
W20240503 08:14:48.309381 30960 leader_election.cc:334] T d94f5824a71f454f8a1bfa13dd032684 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:48.309557 30960 leader_election.cc:302] T d94f5824a71f454f8a1bfa13dd032684 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ced21ca983b749d0b4d4f6a0e2426bf8; no voters: 1b411c6bb25549368041773d7b710d48, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:48.309872 13421 raft_consensus.cc:2726] T d94f5824a71f454f8a1bfa13dd032684 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 08:14:48.309926 30962 leader_election.cc:334] T d94f5824a71f454f8a1bfa13dd032684 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211): Network error: Client connection negotiation failed: client connection to 127.20.230.68:46211: connect: Connection refused (error 111)
I20240503 08:14:48.364173 13608 raft_consensus.cc:482] T d3da6397e0c34c8f96933963dd4fa65e P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 833582ff68c94269be364ebd187a75db)
I20240503 08:14:48.364341 13608 raft_consensus.cc:504] T d3da6397e0c34c8f96933963dd4fa65e P ce1496a47d084365ad786cd09e472a2c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:14:48.364964 13608 leader_election.cc:288] T d3da6397e0c34c8f96933963dd4fa65e P ce1496a47d084365ad786cd09e472a2c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:14:48.365409 21401 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8bBFGA/build/asan/bin/kudu with pid 30812
W20240503 08:14:48.368903 31095 consensus_peers.cc:485] T 6a96965691644db8b9461d146536a026 P b00f244c69df401997ea99d492218a98 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:14:48.415772 30960 connection.cc:535] server connection from 127.20.230.70:57987 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:14:48.416378 21401 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8bBFGA/build/asan/bin/kudu pid 30943
W20240503 08:14:48.416488 30959 consensus_peers.cc:485] T 48075c4a45d24e84bb2b9760710de3a0 P ced21ca983b749d0b4d4f6a0e2426bf8 -> Peer ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): Couldn't send request to peer ce1496a47d084365ad786cd09e472a2c. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.70:33747: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:48.416848 31094 consensus_peers.cc:485] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98 -> Peer ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): Couldn't send request to peer ce1496a47d084365ad786cd09e472a2c. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.70:33747: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:48.421500 31067 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 08:14:48.430086 31094 consensus_peers.cc:485] T 200387e6edce4ee9bf11331380923005 P b00f244c69df401997ea99d492218a98 -> Peer ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): Couldn't send request to peer ce1496a47d084365ad786cd09e472a2c. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.70:33747: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:48.595489 13613 raft_consensus.cc:482] T 7e0350bcc5fa4051bd10cc3de09639aa P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 833582ff68c94269be364ebd187a75db)
I20240503 08:14:48.595665 13613 raft_consensus.cc:504] T 7e0350bcc5fa4051bd10cc3de09639aa P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:14:48.596226 13613 leader_election.cc:288] T 7e0350bcc5fa4051bd10cc3de09639aa P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:14:48.596925 13613 raft_consensus.cc:482] T 46d27fa12990405cae5f94976ac7a014 P b00f244c69df401997ea99d492218a98 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 833582ff68c94269be364ebd187a75db)
I20240503 08:14:48.597092 13613 raft_consensus.cc:504] T 46d27fa12990405cae5f94976ac7a014 P b00f244c69df401997ea99d492218a98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
W20240503 08:14:48.597345 31097 leader_election.cc:334] T 7e0350bcc5fa4051bd10cc3de09639aa P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Network error: Client connection negotiation failed: client connection to 127.20.230.65:39081: connect: Connection refused (error 111)
I20240503 08:14:48.597467 13503 raft_consensus.cc:482] T ca38671a760d4534ab903579e0487772 P b00f244c69df401997ea99d492218a98 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 833582ff68c94269be364ebd187a75db)
I20240503 08:14:48.597568 13636 raft_consensus.cc:482] T f1b2f3b8111649a7b5725647b8251698 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 833582ff68c94269be364ebd187a75db)
I20240503 08:14:48.597610 13503 raft_consensus.cc:504] T ca38671a760d4534ab903579e0487772 P b00f244c69df401997ea99d492218a98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:14:48.597726 13613 leader_election.cc:288] T 46d27fa12990405cae5f94976ac7a014 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:14:48.597694 13636 raft_consensus.cc:504] T f1b2f3b8111649a7b5725647b8251698 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:48.598220 13503 leader_election.cc:288] T ca38671a760d4534ab903579e0487772 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:14:48.598681 13636 leader_election.cc:288] T f1b2f3b8111649a7b5725647b8251698 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
W20240503 08:14:48.599478 31095 leader_election.cc:334] T 46d27fa12990405cae5f94976ac7a014 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:48.599610 31097 leader_election.cc:334] T 46d27fa12990405cae5f94976ac7a014 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Network error: Client connection negotiation failed: client connection to 127.20.230.65:39081: connect: Connection refused (error 111)
W20240503 08:14:48.599701 31095 leader_election.cc:334] T ca38671a760d4534ab903579e0487772 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:48.599733 31097 leader_election.cc:302] T 46d27fa12990405cae5f94976ac7a014 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b00f244c69df401997ea99d492218a98; no voters: 833582ff68c94269be364ebd187a75db, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:48.600127 13636 raft_consensus.cc:2726] T 46d27fa12990405cae5f94976ac7a014 P b00f244c69df401997ea99d492218a98 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:14:48.600260 31095 leader_election.cc:334] T f1b2f3b8111649a7b5725647b8251698 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
W20240503 08:14:48.600495 31097 leader_election.cc:334] T ca38671a760d4534ab903579e0487772 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Network error: Client connection negotiation failed: client connection to 127.20.230.65:39081: connect: Connection refused (error 111)
I20240503 08:14:48.600602 31097 leader_election.cc:302] T ca38671a760d4534ab903579e0487772 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b00f244c69df401997ea99d492218a98; no voters: 833582ff68c94269be364ebd187a75db, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:48.600817 13636 raft_consensus.cc:2726] T ca38671a760d4534ab903579e0487772 P b00f244c69df401997ea99d492218a98 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:14:48.601294 31097 leader_election.cc:334] T f1b2f3b8111649a7b5725647b8251698 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 833582ff68c94269be364ebd187a75db (127.20.230.65:39081): Network error: Client connection negotiation failed: client connection to 127.20.230.65:39081: connect: Connection refused (error 111)
I20240503 08:14:48.601414 31097 leader_election.cc:302] T f1b2f3b8111649a7b5725647b8251698 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b00f244c69df401997ea99d492218a98; no voters: 833582ff68c94269be364ebd187a75db, aae860d7ae2b4002bf7794c278f9cc56
I20240503 08:14:48.601621 13636 raft_consensus.cc:2726] T f1b2f3b8111649a7b5725647b8251698 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 08:14:48.646972 31094 consensus_peers.cc:485] T e8bc656988c44600a198850c604e0b64 P b00f244c69df401997ea99d492218a98 -> Peer ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): Couldn't send request to peer ce1496a47d084365ad786cd09e472a2c. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.70:33747: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:48.712939 13636 raft_consensus.cc:482] T 71fa62206f744de0b97e3aa9ea753e37 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d1d7de19459435fbde1999e3dfd226b)
I20240503 08:14:48.713093 13636 raft_consensus.cc:504] T 71fa62206f744de0b97e3aa9ea753e37 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } }
I20240503 08:14:48.713696 13636 leader_election.cc:288] T 71fa62206f744de0b97e3aa9ea753e37 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747)
W20240503 08:14:48.714871 31096 leader_election.cc:334] T 71fa62206f744de0b97e3aa9ea753e37 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): Network error: Client connection negotiation failed: client connection to 127.20.230.69:34725: connect: Connection refused (error 111)
W20240503 08:14:48.715045 31094 leader_election.cc:334] T 71fa62206f744de0b97e3aa9ea753e37 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): Network error: Client connection negotiation failed: client connection to 127.20.230.70:33747: connect: Connection refused (error 111)
I20240503 08:14:48.715165 31094 leader_election.cc:302] T 71fa62206f744de0b97e3aa9ea753e37 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b00f244c69df401997ea99d492218a98; no voters: 6d1d7de19459435fbde1999e3dfd226b, ce1496a47d084365ad786cd09e472a2c
I20240503 08:14:48.715378 13636 raft_consensus.cc:2726] T 71fa62206f744de0b97e3aa9ea753e37 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 08:14:48.730710 31095 consensus_peers.cc:485] T 4a23c8186c4d4b23b62d7474088b45c1 P b00f244c69df401997ea99d492218a98 -> Peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Couldn't send request to peer aae860d7ae2b4002bf7794c278f9cc56. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:14:48.734172 13421 raft_consensus.cc:482] T fe4cf85b2a5d46fea0bb09ccda5e6351 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d09a11a03989421b9f28019dc1c14cc2)
I20240503 08:14:48.734911 13421 raft_consensus.cc:504] T fe4cf85b2a5d46fea0bb09ccda5e6351 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "ce1496a47d084365ad786cd09e472a2c" member_type: VOTER last_known_addr { host: "127.20.230.70" port: 33747 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:14:48.735535 13421 leader_election.cc:288] T fe4cf85b2a5d46fea0bb09ccda5e6351 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:14:48.737731 13640 raft_consensus.cc:482] T ab35ebabd6fe45fab890739a800794b1 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 833582ff68c94269be364ebd187a75db)
I20240503 08:14:48.737895 13640 raft_consensus.cc:504] T ab35ebabd6fe45fab890739a800794b1 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:14:48.738444 13640 leader_election.cc:288] T ab35ebabd6fe45fab890739a800794b1 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:14:48.746114 13421 raft_consensus.cc:482] T 9fbe2eaba5344dfaa29e78a0d73f224a P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 833582ff68c94269be364ebd187a75db)
I20240503 08:14:48.746269 13421 raft_consensus.cc:504] T 9fbe2eaba5344dfaa29e78a0d73f224a P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:48.746832 13421 leader_election.cc:288] T 9fbe2eaba5344dfaa29e78a0d73f224a P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
W20240503 08:14:48.748507 30960 leader_election.cc:334] T 9fbe2eaba5344dfaa29e78a0d73f224a P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919): Network error: Client connection negotiation failed: client connection to 127.20.230.67:39919: connect: Connection refused (error 111)
I20240503 08:14:48.749505 13421 raft_consensus.cc:482] T 91c573181144456aa7e27a3026a639c4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d1d7de19459435fbde1999e3dfd226b)
I20240503 08:14:48.749655 13421 raft_consensus.cc:504] T 91c573181144456aa7e27a3026a639c4 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "1b411c6bb25549368041773d7b710d48" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 37143 } }
I20240503 08:14:48.750447 13421 leader_election.cc:288] T 91c573181144456aa7e27a3026a639c4 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), 1b411c6bb25549368041773d7b710d48 (127.20.230.66:37143)
I20240503 08:14:48.750581 13601 raft_consensus.cc:482] T f46393257ce9471891698c36a07546ad P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d09a11a03989421b9f28019dc1c14cc2)
I20240503 08:14:48.750581 13421 raft_consensus.cc:482] T f1b2f3b8111649a7b5725647b8251698 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 833582ff68c94269be364ebd187a75db)
I20240503 08:14:48.750756 13601 raft_consensus.cc:504] T f46393257ce9471891698c36a07546ad P ced21ca983b749d0b4d4f6a0e2426bf8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:48.750775 13421 raft_consensus.cc:504] T f1b2f3b8111649a7b5725647b8251698 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } }
I20240503 08:14:48.751297 13642 raft_consensus.cc:482] T c8ca21ffe16e41ab80f5693dc4dbb682 P ced21ca983b749d0b4d4f6a0e2426bf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 833582ff68c94269be364ebd187a75db)
I20240503 08:14:48.751464 13421 leader_election.cc:288] T f1b2f3b8111649a7b5725647b8251698 P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b00f244c69df401997ea99d492218a98 (127.20.230.72:40893), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:14:48.751298 13601 leader_election.cc:288] T f46393257ce9471891698c36a07546ad P ced21ca983b749d0b4d4f6a0e2426bf8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919), d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211)
I20240503 08:14:48.751658 21401 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8bBFGA/build/asan/bin/kudu with pid 30943
I20240503 08:14:48.770392 13636 raft_consensus.cc:482] T 8d0be44e70ce4c15aaf845fd898e01c3 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d1d7de19459435fbde1999e3dfd226b)
I20240503 08:14:48.770851 13636 raft_consensus.cc:504] T 8d0be44e70ce4c15aaf845fd898e01c3 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } }
I20240503 08:14:48.771863 13636 leader_election.cc:288] T 8d0be44e70ce4c15aaf845fd898e01c3 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915)
W20240503 08:14:48.772989 31096 leader_election.cc:334] T 8d0be44e70ce4c15aaf845fd898e01c3 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725): Network error: Client connection negotiation failed: client connection to 127.20.230.69:34725: connect: Connection refused (error 111)
W20240503 08:14:48.795692 31095 connection.cc:535] client connection to 127.20.230.71:41915 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:14:48.796347 21401 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8bBFGA/build/asan/bin/kudu pid 31076
W20240503 08:14:48.796780 31095 consensus_peers.cc:485] T f463dfe26ee94b5f9447c6dbdf8759c3 P b00f244c69df401997ea99d492218a98 -> Peer ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): Couldn't send request to peer ced21ca983b749d0b4d4f6a0e2426bf8. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.71:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:48.796922 31095 leader_election.cc:334] T f1b2f3b8111649a7b5725647b8251698 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): Network error: Client connection negotiation failed: client connection to 127.20.230.71:41915: connect: Connection refused (error 111)
W20240503 08:14:48.797202 31095 consensus_peers.cc:485] T 013e00c67a1c4ae9b3c1a050b966e9a5 P b00f244c69df401997ea99d492218a98 -> Peer ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): Couldn't send request to peer ced21ca983b749d0b4d4f6a0e2426bf8. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.71:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:48.797291 31095 leader_election.cc:334] T 8d0be44e70ce4c15aaf845fd898e01c3 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): Network error: Client connection negotiation failed: client connection to 127.20.230.71:41915: connect: Connection refused (error 111)
I20240503 08:14:48.797381 31095 leader_election.cc:302] T 8d0be44e70ce4c15aaf845fd898e01c3 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b00f244c69df401997ea99d492218a98; no voters: 6d1d7de19459435fbde1999e3dfd226b, ced21ca983b749d0b4d4f6a0e2426bf8
I20240503 08:14:48.797605 13636 raft_consensus.cc:2726] T 8d0be44e70ce4c15aaf845fd898e01c3 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 08:14:48.797662 31095 consensus_peers.cc:485] T 644bb089a4064ecfb88438d8b859bf01 P b00f244c69df401997ea99d492218a98 -> Peer ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): Couldn't send request to peer ced21ca983b749d0b4d4f6a0e2426bf8. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.71:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:48.797783 31095 consensus_peers.cc:485] T 2401f8f101f94189aa3c0582f08cef88 P b00f244c69df401997ea99d492218a98 -> Peer ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): Couldn't send request to peer ced21ca983b749d0b4d4f6a0e2426bf8. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.71:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:48.797874 31095 consensus_peers.cc:485] T 98b3c906cfe54f49bb88ce5e00e3cc04 P b00f244c69df401997ea99d492218a98 -> Peer ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): Couldn't send request to peer ced21ca983b749d0b4d4f6a0e2426bf8. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.71:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:48.797955 31095 leader_election.cc:334] T 7e0350bcc5fa4051bd10cc3de09639aa P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ced21ca983b749d0b4d4f6a0e2426bf8 (127.20.230.71:41915): Network error: Client connection negotiation failed: client connection to 127.20.230.71:41915: connect: Connection refused (error 111)
I20240503 08:14:48.798046 31095 leader_election.cc:302] T 7e0350bcc5fa4051bd10cc3de09639aa P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b00f244c69df401997ea99d492218a98; no voters: 833582ff68c94269be364ebd187a75db, ced21ca983b749d0b4d4f6a0e2426bf8
I20240503 08:14:48.798261 13636 raft_consensus.cc:2726] T 7e0350bcc5fa4051bd10cc3de09639aa P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 08:14:48.800076 31094 consensus_peers.cc:485] T 549f64ad9ef64a3da619c5df5a411e1c P b00f244c69df401997ea99d492218a98 -> Peer ce1496a47d084365ad786cd09e472a2c (127.20.230.70:33747): Couldn't send request to peer ce1496a47d084365ad786cd09e472a2c. Status: Network error: Client connection negotiation failed: client connection to 127.20.230.70:33747: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:48.803305 31201 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:14:49.117934 13636 raft_consensus.cc:482] T f9746769e2514c65ac8d48ca7df8bca7 P b00f244c69df401997ea99d492218a98 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d1d7de19459435fbde1999e3dfd226b)
I20240503 08:14:49.118122 13636 raft_consensus.cc:504] T f9746769e2514c65ac8d48ca7df8bca7 P b00f244c69df401997ea99d492218a98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d1d7de19459435fbde1999e3dfd226b" member_type: VOTER last_known_addr { host: "127.20.230.69" port: 34725 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "833582ff68c94269be364ebd187a75db" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 39081 } }
I20240503 08:14:49.118752 13636 leader_election.cc:288] T f9746769e2514c65ac8d48ca7df8bca7 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d1d7de19459435fbde1999e3dfd226b (127.20.230.69:34725), 833582ff68c94269be364ebd187a75db (127.20.230.65:39081)
I20240503 08:14:49.119697 13647 raft_consensus.cc:482] T 599cb18c58a74e93a3e0334cc43a73d4 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d09a11a03989421b9f28019dc1c14cc2)
I20240503 08:14:49.119841 13647 raft_consensus.cc:504] T 599cb18c58a74e93a3e0334cc43a73d4 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } } peers { permanent_uuid: "aae860d7ae2b4002bf7794c278f9cc56" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 39919 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } }
I20240503 08:14:49.120455 13647 leader_election.cc:288] T 599cb18c58a74e93a3e0334cc43a73d4 P b00f244c69df401997ea99d492218a98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d09a11a03989421b9f28019dc1c14cc2 (127.20.230.68:46211), aae860d7ae2b4002bf7794c278f9cc56 (127.20.230.67:39919)
I20240503 08:14:49.120581 13647 raft_consensus.cc:482] T d24e437f369b4df49e6363f3f492dc46 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d09a11a03989421b9f28019dc1c14cc2)
I20240503 08:14:49.120697 13647 raft_consensus.cc:504] T d24e437f369b4df49e6363f3f492dc46 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced21ca983b749d0b4d4f6a0e2426bf8" member_type: VOTER last_known_addr { host: "127.20.230.71" port: 41915 } } peers { permanent_uuid: "b00f244c69df401997ea99d492218a98" member_type: VOTER last_known_addr { host: "127.20.230.72" port: 40893 } } peers { permanent_uuid: "d09a11a03989421b9f28019dc1c14cc2" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 46211 } }
I20240503 08:14:49.120857 21401 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8bBFGA/build/asan/bin/kudu with pid 31076
I20240503 08:14:49.120865 13613 raft_consensus.cc:482] T 6c2824aa878845e098f600f1fcfea087 P b00f244c69df401997ea99d492218a98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d1d7de19459435fbde1999e3dfd226b)
I20240503 08:14:49.164891 21401 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8bBFGA/build/asan/bin/kudu pid 27831
I20240503 08:14:49.170603 27892 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:14:49.470522 21401 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8bBFGA/build/asan/bin/kudu with pid 27831
2024-05-03T08:14:49Z chronyd exiting
I20240503 08:14:49.532210 21401 test_util.cc:175] -----------------------------------------------
I20240503 08:14:49.532290 21401 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714723866460737-21401-0
[ FAILED ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/0, where GetParam() = 4-byte object <00-00 00-00> (169280 ms)
[----------] 1 test from DDLDuringRebalancingTest (169280 ms total)
[----------] 1 test from RebalancingDuringElectionStormTest
[ RUN ] RebalancingDuringElectionStormTest.RoundRobin/0
I20240503 08:14:49.534355 21401 test_util.cc:268] Using random seed: 1346279279
I20240503 08:14:49.535996 21401 ts_itest-base.cc:112] Starting cluster with:
I20240503 08:14:49.536067 21401 ts_itest-base.cc:113] --------------
I20240503 08:14:49.536113 21401 ts_itest-base.cc:114] 8 tablet servers
I20240503 08:14:49.536178 21401 ts_itest-base.cc:115] 3 replicas per TS
I20240503 08:14:49.536231 21401 ts_itest-base.cc:116] --------------
2024-05-03T08:14:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:14:49Z Disabled control of system clock
I20240503 08:14:49.563829 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.230.126:38135
--webserver_interface=127.20.230.126
--webserver_port=0
--builtin_ntp_servers=127.20.230.84:40715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.230.126:38135
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240503 08:14:49.875203 13658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:49.875487 13658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:49.875605 13658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:49.882858 13658 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:14:49.882969 13658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:14:49.883044 13658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:49.883112 13658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:14:49.883177 13658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:14:49.892022 13658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:40715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.20.230.126:38135
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.230.126:38135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.230.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:14:49.893316 13658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:49.895074 13658 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:49.905732 13663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:49.906242 13664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:49.906713 13666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:49.907508 13658 server_base.cc:1034] running on GCE node
I20240503 08:14:49.908170 13658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:49.909440 13658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:49.910638 13658 hybrid_clock.cc:648] HybridClock initialized: now 1714724089910595 us; error 34 us; skew 500 ppm
I20240503 08:14:49.910928 13658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:14:49.916471 13658 webserver.cc:460] Webserver started at http://127.20.230.126:38549/ using document root <none> and password file <none>
I20240503 08:14:49.917086 13658 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:49.917186 13658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:49.917444 13658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:49.921885 13658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "9e8bf16ab9984571aa834aef777d1065"
format_stamp: "Formatted at 2024-05-03 08:14:49 on dist-test-slave-6vp8"
I20240503 08:14:49.922595 13658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "9e8bf16ab9984571aa834aef777d1065"
format_stamp: "Formatted at 2024-05-03 08:14:49 on dist-test-slave-6vp8"
I20240503 08:14:49.927246 13658 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 08:14:49.930317 13672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:49.931507 13658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:14:49.931640 13658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/wal
uuid: "9e8bf16ab9984571aa834aef777d1065"
format_stamp: "Formatted at 2024-05-03 08:14:49 on dist-test-slave-6vp8"
I20240503 08:14:49.931790 13658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:49.950961 13658 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:49.951632 13658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:49.951853 13658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:49.967811 13658 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.126:38135
I20240503 08:14:49.967839 13723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.126:38135 every 8 connection(s)
I20240503 08:14:49.968986 13658 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 08:14:49.971046 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 13658
I20240503 08:14:49.971251 21401 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 08:14:49.972352 13724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:49.983017 13724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e8bf16ab9984571aa834aef777d1065: Bootstrap starting.
I20240503 08:14:49.985929 13724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e8bf16ab9984571aa834aef777d1065: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:49.986819 13724 log.cc:824] T 00000000000000000000000000000000 P 9e8bf16ab9984571aa834aef777d1065: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:49.989033 13724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e8bf16ab9984571aa834aef777d1065: No bootstrap required, opened a new log
I20240503 08:14:49.995677 13724 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e8bf16ab9984571aa834aef777d1065 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e8bf16ab9984571aa834aef777d1065" member_type: VOTER last_known_addr { host: "127.20.230.126" port: 38135 } }
I20240503 08:14:49.995981 13724 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e8bf16ab9984571aa834aef777d1065 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:49.996083 13724 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e8bf16ab9984571aa834aef777d1065 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e8bf16ab9984571aa834aef777d1065, State: Initialized, Role: FOLLOWER
I20240503 08:14:49.996620 13724 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e8bf16ab9984571aa834aef777d1065 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e8bf16ab9984571aa834aef777d1065" member_type: VOTER last_known_addr { host: "127.20.230.126" port: 38135 } }
I20240503 08:14:49.996800 13724 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e8bf16ab9984571aa834aef777d1065 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:14:49.996910 13724 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e8bf16ab9984571aa834aef777d1065 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:14:49.997045 13724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e8bf16ab9984571aa834aef777d1065 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:49.999013 13724 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e8bf16ab9984571aa834aef777d1065 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e8bf16ab9984571aa834aef777d1065" member_type: VOTER last_known_addr { host: "127.20.230.126" port: 38135 } }
I20240503 08:14:49.999455 13724 leader_election.cc:302] T 00000000000000000000000000000000 P 9e8bf16ab9984571aa834aef777d1065 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e8bf16ab9984571aa834aef777d1065; no voters:
I20240503 08:14:50.000087 13724 leader_election.cc:288] T 00000000000000000000000000000000 P 9e8bf16ab9984571aa834aef777d1065 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:14:50.000285 13729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e8bf16ab9984571aa834aef777d1065 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:50.001076 13729 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e8bf16ab9984571aa834aef777d1065 [term 1 LEADER]: Becoming Leader. State: Replica: 9e8bf16ab9984571aa834aef777d1065, State: Running, Role: LEADER
I20240503 08:14:50.001659 13729 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e8bf16ab9984571aa834aef777d1065 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e8bf16ab9984571aa834aef777d1065" member_type: VOTER last_known_addr { host: "127.20.230.126" port: 38135 } }
I20240503 08:14:50.002149 13724 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e8bf16ab9984571aa834aef777d1065 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:14:50.004619 13730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e8bf16ab9984571aa834aef777d1065 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e8bf16ab9984571aa834aef777d1065" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e8bf16ab9984571aa834aef777d1065" member_type: VOTER last_known_addr { host: "127.20.230.126" port: 38135 } } }
I20240503 08:14:50.004725 13731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e8bf16ab9984571aa834aef777d1065 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e8bf16ab9984571aa834aef777d1065. Latest consensus state: current_term: 1 leader_uuid: "9e8bf16ab9984571aa834aef777d1065" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e8bf16ab9984571aa834aef777d1065" member_type: VOTER last_known_addr { host: "127.20.230.126" port: 38135 } } }
I20240503 08:14:50.004946 13730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e8bf16ab9984571aa834aef777d1065 [sys.catalog]: This master's current role is: LEADER
I20240503 08:14:50.005014 13731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e8bf16ab9984571aa834aef777d1065 [sys.catalog]: This master's current role is: LEADER
I20240503 08:14:50.006291 13736 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:14:50.011045 13736 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:14:50.019070 13736 catalog_manager.cc:1347] Generated new cluster ID: bf12ddca530f4942ae0baa2716fb102c
I20240503 08:14:50.019171 13736 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:14:50.037341 13736 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:14:50.038455 13736 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:14:50.053831 13736 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 9e8bf16ab9984571aa834aef777d1065: Generated new TSK 0
I20240503 08:14:50.054822 13736 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:14:50.066994 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.65:0
--local_ip_for_outbound_sockets=127.20.230.65
--webserver_interface=127.20.230.65
--webserver_port=0
--tserver_master_addrs=127.20.230.126:38135
--builtin_ntp_servers=127.20.230.84:40715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 08:14:50.375476 13748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:50.375721 13748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:50.375839 13748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:50.382875 13748 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:14:50.382987 13748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:50.383141 13748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.65
I20240503 08:14:50.391891 13748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:40715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.230.65
--webserver_port=0
--tserver_master_addrs=127.20.230.126:38135
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.65
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:14:50.393014 13748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:50.394798 13748 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:50.406239 13756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:50.406666 13753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:50.407259 13754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:50.407907 13748 server_base.cc:1034] running on GCE node
I20240503 08:14:50.408402 13748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:50.409178 13748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:50.410372 13748 hybrid_clock.cc:648] HybridClock initialized: now 1714724090410328 us; error 32 us; skew 500 ppm
I20240503 08:14:50.410656 13748 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:14:50.412607 13748 webserver.cc:460] Webserver started at http://127.20.230.65:40557/ using document root <none> and password file <none>
I20240503 08:14:50.413265 13748 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:50.413355 13748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:50.413692 13748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:50.416077 13748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "5a198ad76e4f462c8be0859184aeb2fc"
format_stamp: "Formatted at 2024-05-03 08:14:50 on dist-test-slave-6vp8"
I20240503 08:14:50.416785 13748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "5a198ad76e4f462c8be0859184aeb2fc"
format_stamp: "Formatted at 2024-05-03 08:14:50 on dist-test-slave-6vp8"
I20240503 08:14:50.421285 13748 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20240503 08:14:50.424249 13762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:50.425469 13748 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 08:14:50.425624 13748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "5a198ad76e4f462c8be0859184aeb2fc"
format_stamp: "Formatted at 2024-05-03 08:14:50 on dist-test-slave-6vp8"
I20240503 08:14:50.425820 13748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:50.446549 13748 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:50.447269 13748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:50.447505 13748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:50.448338 13748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:50.449661 13748 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:50.449751 13748 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:50.449880 13748 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:50.449951 13748 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:50.477511 13748 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.65:44925
I20240503 08:14:50.477556 13874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.65:44925 every 8 connection(s)
I20240503 08:14:50.478710 13748 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 08:14:50.485571 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 13748
I20240503 08:14:50.485735 21401 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 08:14:50.488726 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.66:0
--local_ip_for_outbound_sockets=127.20.230.66
--webserver_interface=127.20.230.66
--webserver_port=0
--tserver_master_addrs=127.20.230.126:38135
--builtin_ntp_servers=127.20.230.84:40715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:14:50.490307 13875 heartbeater.cc:347] Connected to a master server at 127.20.230.126:38135
I20240503 08:14:50.490547 13875 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:50.491168 13875 heartbeater.cc:511] Master 127.20.230.126:38135 requested a full tablet report, sending...
I20240503 08:14:50.493155 13689 ts_manager.cc:194] Registered new tserver with Master: 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:14:50.495020 13689 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.65:57813
W20240503 08:14:50.789834 13878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:50.790115 13878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:50.790244 13878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:50.797309 13878 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:14:50.797437 13878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:50.797585 13878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.66
I20240503 08:14:50.806200 13878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:40715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.230.66
--webserver_port=0
--tserver_master_addrs=127.20.230.126:38135
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.66
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:14:50.807417 13878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:50.809046 13878 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:50.820026 13883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:50.820576 13886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:50.821347 13884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:50.822044 13878 server_base.cc:1034] running on GCE node
I20240503 08:14:50.822567 13878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:50.823364 13878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:50.824554 13878 hybrid_clock.cc:648] HybridClock initialized: now 1714724090824509 us; error 37 us; skew 500 ppm
I20240503 08:14:50.824847 13878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:14:50.826798 13878 webserver.cc:460] Webserver started at http://127.20.230.66:39725/ using document root <none> and password file <none>
I20240503 08:14:50.827474 13878 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:50.827579 13878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:50.827862 13878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:50.830293 13878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
format_stamp: "Formatted at 2024-05-03 08:14:50 on dist-test-slave-6vp8"
I20240503 08:14:50.830983 13878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
format_stamp: "Formatted at 2024-05-03 08:14:50 on dist-test-slave-6vp8"
I20240503 08:14:50.835598 13878 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20240503 08:14:50.838752 13892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:50.839921 13878 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.003s
I20240503 08:14:50.840065 13878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
format_stamp: "Formatted at 2024-05-03 08:14:50 on dist-test-slave-6vp8"
I20240503 08:14:50.840256 13878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:50.853256 13878 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:50.853960 13878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:50.854212 13878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:50.855012 13878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:50.856308 13878 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:50.856400 13878 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:50.856496 13878 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:50.856562 13878 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:50.883563 13878 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.66:44781
I20240503 08:14:50.883617 14004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.66:44781 every 8 connection(s)
I20240503 08:14:50.884683 13878 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 08:14:50.888103 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 13878
I20240503 08:14:50.888298 21401 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 08:14:50.891116 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.67:0
--local_ip_for_outbound_sockets=127.20.230.67
--webserver_interface=127.20.230.67
--webserver_port=0
--tserver_master_addrs=127.20.230.126:38135
--builtin_ntp_servers=127.20.230.84:40715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:14:50.896317 14005 heartbeater.cc:347] Connected to a master server at 127.20.230.126:38135
I20240503 08:14:50.896579 14005 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:50.897229 14005 heartbeater.cc:511] Master 127.20.230.126:38135 requested a full tablet report, sending...
I20240503 08:14:50.898476 13689 ts_manager.cc:194] Registered new tserver with Master: 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:14:50.899454 13689 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.66:51893
W20240503 08:14:51.196259 14008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:51.196518 14008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:51.196609 14008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:51.203871 14008 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:14:51.203974 14008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:51.204087 14008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.67
I20240503 08:14:51.212677 14008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:40715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.230.67
--webserver_port=0
--tserver_master_addrs=127.20.230.126:38135
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.67
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:14:51.213871 14008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:51.215518 14008 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:51.226374 14013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:51.226905 14014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:51.227608 14016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:51.228268 14008 server_base.cc:1034] running on GCE node
I20240503 08:14:51.228787 14008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:51.229697 14008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:51.230907 14008 hybrid_clock.cc:648] HybridClock initialized: now 1714724091230866 us; error 41 us; skew 500 ppm
I20240503 08:14:51.231181 14008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:14:51.233217 14008 webserver.cc:460] Webserver started at http://127.20.230.67:37959/ using document root <none> and password file <none>
I20240503 08:14:51.233927 14008 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:51.234035 14008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:51.234381 14008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:51.236910 14008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
format_stamp: "Formatted at 2024-05-03 08:14:51 on dist-test-slave-6vp8"
I20240503 08:14:51.237639 14008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
format_stamp: "Formatted at 2024-05-03 08:14:51 on dist-test-slave-6vp8"
I20240503 08:14:51.242437 14008 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20240503 08:14:51.245558 14022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:51.246747 14008 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 08:14:51.246884 14008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
format_stamp: "Formatted at 2024-05-03 08:14:51 on dist-test-slave-6vp8"
I20240503 08:14:51.247051 14008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:51.261932 14008 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:51.262609 14008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:51.262840 14008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:51.263639 14008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:51.264940 14008 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:51.265029 14008 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:51.265131 14008 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:51.265197 14008 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:51.292905 14008 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.67:34647
I20240503 08:14:51.292948 14134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.67:34647 every 8 connection(s)
I20240503 08:14:51.294147 14008 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:14:51.300233 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 14008
I20240503 08:14:51.300384 21401 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 08:14:51.303848 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.68:0
--local_ip_for_outbound_sockets=127.20.230.68
--webserver_interface=127.20.230.68
--webserver_port=0
--tserver_master_addrs=127.20.230.126:38135
--builtin_ntp_servers=127.20.230.84:40715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:14:51.305917 14135 heartbeater.cc:347] Connected to a master server at 127.20.230.126:38135
I20240503 08:14:51.306156 14135 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:51.306805 14135 heartbeater.cc:511] Master 127.20.230.126:38135 requested a full tablet report, sending...
I20240503 08:14:51.307986 13689 ts_manager.cc:194] Registered new tserver with Master: abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:14:51.308722 13689 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.67:59005
I20240503 08:14:51.497948 13875 heartbeater.cc:503] Master 127.20.230.126:38135 was elected leader, sending a full tablet report...
W20240503 08:14:51.607177 14138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:51.607434 14138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:51.607537 14138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:51.614518 14138 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:14:51.614610 14138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:51.614722 14138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.68
I20240503 08:14:51.623270 14138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:40715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.230.68
--webserver_port=0
--tserver_master_addrs=127.20.230.126:38135
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.68
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:14:51.624358 14138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:51.625916 14138 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:51.637089 14143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:51.637706 14144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:51.638106 14146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:51.638645 14138 server_base.cc:1034] running on GCE node
I20240503 08:14:51.639118 14138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:51.639900 14138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:51.641067 14138 hybrid_clock.cc:648] HybridClock initialized: now 1714724091641020 us; error 45 us; skew 500 ppm
I20240503 08:14:51.641333 14138 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:14:51.643301 14138 webserver.cc:460] Webserver started at http://127.20.230.68:32901/ using document root <none> and password file <none>
I20240503 08:14:51.643916 14138 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:51.644008 14138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:51.644316 14138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:51.646651 14138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
format_stamp: "Formatted at 2024-05-03 08:14:51 on dist-test-slave-6vp8"
I20240503 08:14:51.647342 14138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
format_stamp: "Formatted at 2024-05-03 08:14:51 on dist-test-slave-6vp8"
I20240503 08:14:51.651952 14138 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 08:14:51.654948 14152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:51.656066 14138 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:14:51.656211 14138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
format_stamp: "Formatted at 2024-05-03 08:14:51 on dist-test-slave-6vp8"
I20240503 08:14:51.656380 14138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:51.672923 14138 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:51.673576 14138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:51.673840 14138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:51.674657 14138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:51.675940 14138 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:51.676028 14138 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:51.676126 14138 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:51.676193 14138 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:51.704115 14138 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.68:33713
I20240503 08:14:51.704195 14264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.68:33713 every 8 connection(s)
I20240503 08:14:51.705263 14138 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:14:51.711840 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 14138
I20240503 08:14:51.712042 21401 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 08:14:51.715251 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.69:0
--local_ip_for_outbound_sockets=127.20.230.69
--webserver_interface=127.20.230.69
--webserver_port=0
--tserver_master_addrs=127.20.230.126:38135
--builtin_ntp_servers=127.20.230.84:40715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:14:51.717084 14265 heartbeater.cc:347] Connected to a master server at 127.20.230.126:38135
I20240503 08:14:51.717391 14265 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:51.718200 14265 heartbeater.cc:511] Master 127.20.230.126:38135 requested a full tablet report, sending...
I20240503 08:14:51.719434 13689 ts_manager.cc:194] Registered new tserver with Master: 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:14:51.720175 13689 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.68:58895
I20240503 08:14:51.901533 14005 heartbeater.cc:503] Master 127.20.230.126:38135 was elected leader, sending a full tablet report...
W20240503 08:14:52.019593 14268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:52.019851 14268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:52.019980 14268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:52.027034 14268 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:14:52.027155 14268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:52.027295 14268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.69
I20240503 08:14:52.035960 14268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:40715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.230.69
--webserver_port=0
--tserver_master_addrs=127.20.230.126:38135
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.69
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:14:52.037115 14268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:52.038738 14268 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:52.049757 14273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:52.050320 14276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:52.050938 14274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:52.051666 14268 server_base.cc:1034] running on GCE node
I20240503 08:14:52.052124 14268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:52.052893 14268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:52.054073 14268 hybrid_clock.cc:648] HybridClock initialized: now 1714724092054033 us; error 31 us; skew 500 ppm
I20240503 08:14:52.054348 14268 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:14:52.056264 14268 webserver.cc:460] Webserver started at http://127.20.230.69:40385/ using document root <none> and password file <none>
I20240503 08:14:52.056879 14268 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:52.056977 14268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:52.057268 14268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:52.059630 14268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "027334ce0b324689b9697ad372f38d8c"
format_stamp: "Formatted at 2024-05-03 08:14:52 on dist-test-slave-6vp8"
I20240503 08:14:52.060318 14268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "027334ce0b324689b9697ad372f38d8c"
format_stamp: "Formatted at 2024-05-03 08:14:52 on dist-test-slave-6vp8"
I20240503 08:14:52.065200 14268 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20240503 08:14:52.068431 14282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:52.069567 14268 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:52.069703 14268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "027334ce0b324689b9697ad372f38d8c"
format_stamp: "Formatted at 2024-05-03 08:14:52 on dist-test-slave-6vp8"
I20240503 08:14:52.069900 14268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:52.094434 14268 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:52.095157 14268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:52.095408 14268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:52.096251 14268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:52.097571 14268 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:52.097663 14268 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:52.097796 14268 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:52.097873 14268 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20240503 08:14:52.125849 14268 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.69:33543
I20240503 08:14:52.125886 14394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.69:33543 every 8 connection(s)
I20240503 08:14:52.126986 14268 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:14:52.139266 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 14268
I20240503 08:14:52.139428 21401 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 08:14:52.140817 14395 heartbeater.cc:347] Connected to a master server at 127.20.230.126:38135
I20240503 08:14:52.141063 14395 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:52.141906 14395 heartbeater.cc:511] Master 127.20.230.126:38135 requested a full tablet report, sending...
I20240503 08:14:52.142586 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.70:0
--local_ip_for_outbound_sockets=127.20.230.70
--webserver_interface=127.20.230.70
--webserver_port=0
--tserver_master_addrs=127.20.230.126:38135
--builtin_ntp_servers=127.20.230.84:40715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:14:52.143308 13689 ts_manager.cc:194] Registered new tserver with Master: 027334ce0b324689b9697ad372f38d8c (127.20.230.69:33543)
I20240503 08:14:52.144272 13689 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.69:40771
I20240503 08:14:52.310863 14135 heartbeater.cc:503] Master 127.20.230.126:38135 was elected leader, sending a full tablet report...
W20240503 08:14:52.448333 14398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:52.448596 14398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:52.448719 14398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:52.455744 14398 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:14:52.455854 14398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:52.455997 14398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.70
I20240503 08:14:52.464282 14398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:40715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.20.230.70
--webserver_port=0
--tserver_master_addrs=127.20.230.126:38135
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.70
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:14:52.465365 14398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:52.466923 14398 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:52.477710 14403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:52.478302 14406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:52.479053 14404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:52.479720 14398 server_base.cc:1034] running on GCE node
I20240503 08:14:52.480228 14398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:52.481050 14398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:52.482259 14398 hybrid_clock.cc:648] HybridClock initialized: now 1714724092482211 us; error 44 us; skew 500 ppm
I20240503 08:14:52.482566 14398 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:14:52.484506 14398 webserver.cc:460] Webserver started at http://127.20.230.70:42161/ using document root <none> and password file <none>
I20240503 08:14:52.485173 14398 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:52.485275 14398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:52.485569 14398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:52.487946 14398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "fed845d7e14a4c07a62eeb5a383532ee"
format_stamp: "Formatted at 2024-05-03 08:14:52 on dist-test-slave-6vp8"
I20240503 08:14:52.488647 14398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "fed845d7e14a4c07a62eeb5a383532ee"
format_stamp: "Formatted at 2024-05-03 08:14:52 on dist-test-slave-6vp8"
I20240503 08:14:52.493232 14398 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20240503 08:14:52.496207 14412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:52.497359 14398 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 08:14:52.497498 14398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "fed845d7e14a4c07a62eeb5a383532ee"
format_stamp: "Formatted at 2024-05-03 08:14:52 on dist-test-slave-6vp8"
I20240503 08:14:52.497663 14398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:52.515775 14398 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:52.516448 14398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:52.516667 14398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:52.517503 14398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:52.518849 14398 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:52.518939 14398 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:52.519033 14398 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:52.519098 14398 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:52.546881 14398 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.70:45427
I20240503 08:14:52.546924 14524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.70:45427 every 8 connection(s)
I20240503 08:14:52.547998 14398 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 08:14:52.550558 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 14398
I20240503 08:14:52.550722 21401 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 08:14:52.553710 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.71:0
--local_ip_for_outbound_sockets=127.20.230.71
--webserver_interface=127.20.230.71
--webserver_port=0
--tserver_master_addrs=127.20.230.126:38135
--builtin_ntp_servers=127.20.230.84:40715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:14:52.559800 14525 heartbeater.cc:347] Connected to a master server at 127.20.230.126:38135
I20240503 08:14:52.560055 14525 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:52.560734 14525 heartbeater.cc:511] Master 127.20.230.126:38135 requested a full tablet report, sending...
I20240503 08:14:52.562004 13689 ts_manager.cc:194] Registered new tserver with Master: fed845d7e14a4c07a62eeb5a383532ee (127.20.230.70:45427)
I20240503 08:14:52.562920 13689 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.70:60579
I20240503 08:14:52.722555 14265 heartbeater.cc:503] Master 127.20.230.126:38135 was elected leader, sending a full tablet report...
W20240503 08:14:52.862428 14528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:52.862682 14528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:52.862804 14528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:52.869975 14528 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:14:52.870079 14528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:52.870239 14528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.71
I20240503 08:14:52.878846 14528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:40715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.20.230.71
--webserver_port=0
--tserver_master_addrs=127.20.230.126:38135
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.71
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:14:52.880012 14528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:52.881585 14528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:52.892654 14534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:52.893128 14533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:52.893893 14536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:52.894512 14528 server_base.cc:1034] running on GCE node
I20240503 08:14:52.895012 14528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:52.895826 14528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:52.897003 14528 hybrid_clock.cc:648] HybridClock initialized: now 1714724092896957 us; error 42 us; skew 500 ppm
I20240503 08:14:52.897291 14528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:14:52.899363 14528 webserver.cc:460] Webserver started at http://127.20.230.71:42117/ using document root <none> and password file <none>
I20240503 08:14:52.900035 14528 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:52.900137 14528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:52.900434 14528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:52.903049 14528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "2a4c6b598a44453bb25cf87f51b0a159"
format_stamp: "Formatted at 2024-05-03 08:14:52 on dist-test-slave-6vp8"
I20240503 08:14:52.903913 14528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "2a4c6b598a44453bb25cf87f51b0a159"
format_stamp: "Formatted at 2024-05-03 08:14:52 on dist-test-slave-6vp8"
I20240503 08:14:52.909106 14528 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240503 08:14:52.912264 14542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:52.913470 14528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 08:14:52.913605 14528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "2a4c6b598a44453bb25cf87f51b0a159"
format_stamp: "Formatted at 2024-05-03 08:14:52 on dist-test-slave-6vp8"
I20240503 08:14:52.913812 14528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:52.936287 14528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:52.937032 14528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:52.937280 14528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:52.938230 14528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:52.939632 14528 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:52.939718 14528 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:52.939822 14528 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:52.939888 14528 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:52.968024 14528 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.71:44051
I20240503 08:14:52.968120 14654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.71:44051 every 8 connection(s)
I20240503 08:14:52.969238 14528 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 08:14:52.972963 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 14528
I20240503 08:14:52.973124 21401 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 08:14:52.976119 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.72:0
--local_ip_for_outbound_sockets=127.20.230.72
--webserver_interface=127.20.230.72
--webserver_port=0
--tserver_master_addrs=127.20.230.126:38135
--builtin_ntp_servers=127.20.230.84:40715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:14:52.982018 14655 heartbeater.cc:347] Connected to a master server at 127.20.230.126:38135
I20240503 08:14:52.982309 14655 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:52.982981 14655 heartbeater.cc:511] Master 127.20.230.126:38135 requested a full tablet report, sending...
I20240503 08:14:52.984293 13689 ts_manager.cc:194] Registered new tserver with Master: 2a4c6b598a44453bb25cf87f51b0a159 (127.20.230.71:44051)
I20240503 08:14:52.985244 13689 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.71:57169
I20240503 08:14:53.146844 14395 heartbeater.cc:503] Master 127.20.230.126:38135 was elected leader, sending a full tablet report...
W20240503 08:14:53.292310 14658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:53.292644 14658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:53.292847 14658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:53.301177 14658 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:14:53.301282 14658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:53.301406 14658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.72
I20240503 08:14:53.310417 14658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:40715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.20.230.72
--webserver_port=0
--tserver_master_addrs=127.20.230.126:38135
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.72
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:14:53.311621 14658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:53.313282 14658 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:53.324970 14663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:53.325515 14666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:53.326192 14664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:53.326944 14658 server_base.cc:1034] running on GCE node
I20240503 08:14:53.327518 14658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:53.328424 14658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:53.329648 14658 hybrid_clock.cc:648] HybridClock initialized: now 1714724093329587 us; error 50 us; skew 500 ppm
I20240503 08:14:53.330006 14658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:14:53.332242 14658 webserver.cc:460] Webserver started at http://127.20.230.72:42803/ using document root <none> and password file <none>
I20240503 08:14:53.332960 14658 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:53.333066 14658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:53.333393 14658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:53.336117 14658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "cab9267f70cb419dbc54ba686883abcb"
format_stamp: "Formatted at 2024-05-03 08:14:53 on dist-test-slave-6vp8"
I20240503 08:14:53.337020 14658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "cab9267f70cb419dbc54ba686883abcb"
format_stamp: "Formatted at 2024-05-03 08:14:53 on dist-test-slave-6vp8"
I20240503 08:14:53.342118 14658 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20240503 08:14:53.345482 14672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:53.346987 14658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:14:53.347146 14658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "cab9267f70cb419dbc54ba686883abcb"
format_stamp: "Formatted at 2024-05-03 08:14:53 on dist-test-slave-6vp8"
I20240503 08:14:53.347339 14658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:53.379133 14658 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:53.379972 14658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:53.380235 14658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:53.381148 14658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:53.382615 14658 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:53.382709 14658 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:53.382813 14658 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:53.382879 14658 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:53.411674 14658 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.72:43911
I20240503 08:14:53.411779 14784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.72:43911 every 8 connection(s)
I20240503 08:14:53.412948 14658 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 08:14:53.417538 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 14658
I20240503 08:14:53.417737 21401 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 08:14:53.425261 14785 heartbeater.cc:347] Connected to a master server at 127.20.230.126:38135
I20240503 08:14:53.425540 14785 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:53.426297 14785 heartbeater.cc:511] Master 127.20.230.126:38135 requested a full tablet report, sending...
I20240503 08:14:53.427562 13689 ts_manager.cc:194] Registered new tserver with Master: cab9267f70cb419dbc54ba686883abcb (127.20.230.72:43911)
I20240503 08:14:53.428334 13689 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.72:42891
I20240503 08:14:53.433643 21401 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 08:14:53.449815 21401 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 08:14:53.449939 21401 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8bBFGA/build/asan/bin/kudu pid 14268
I20240503 08:14:53.457552 14390 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:14:53.536471 21401 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8bBFGA/build/asan/bin/kudu with pid 14268
I20240503 08:14:53.549603 21401 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8bBFGA/build/asan/bin/kudu pid 14398
I20240503 08:14:53.557471 14520 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:14:53.634253 14525 heartbeater.cc:503] Master 127.20.230.126:38135 was elected leader, sending a full tablet report...
I20240503 08:14:53.635816 21401 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8bBFGA/build/asan/bin/kudu with pid 14398
W20240503 08:14:53.648763 13676 connection.cc:535] server connection from 127.20.230.70:60579 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:14:53.649048 21401 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8bBFGA/build/asan/bin/kudu pid 14528
I20240503 08:14:53.656502 14650 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:14:53.734993 21401 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8bBFGA/build/asan/bin/kudu with pid 14528
I20240503 08:14:53.747536 21401 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8bBFGA/build/asan/bin/kudu pid 14658
I20240503 08:14:53.755460 14780 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:14:53.832880 21401 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8bBFGA/build/asan/bin/kudu with pid 14658
I20240503 08:14:57.597638 13689 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44232:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:14:57.631440 14069 tablet_service.cc:1467] Processing CreateTablet for tablet 4b680592b1774f09ba79789afa55a0f1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d98f711a8cd1404b9a01fa843b9a9c7f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:57.632943 14069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b680592b1774f09ba79789afa55a0f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:57.634570 13940 tablet_service.cc:1467] Processing CreateTablet for tablet cdf9885a1e7c4b8dbb160e370ca0f60c (DEFAULT_TABLE table=rebalance_test_table_0 [id=d98f711a8cd1404b9a01fa843b9a9c7f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:57.636013 13939 tablet_service.cc:1467] Processing CreateTablet for tablet 769c4490062b4206951af2f4b9a6da05 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d98f711a8cd1404b9a01fa843b9a9c7f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:57.636456 14199 tablet_service.cc:1467] Processing CreateTablet for tablet 4b680592b1774f09ba79789afa55a0f1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d98f711a8cd1404b9a01fa843b9a9c7f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:57.636451 13810 tablet_service.cc:1467] Processing CreateTablet for tablet cdf9885a1e7c4b8dbb160e370ca0f60c (DEFAULT_TABLE table=rebalance_test_table_0 [id=d98f711a8cd1404b9a01fa843b9a9c7f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:57.636857 14200 tablet_service.cc:1467] Processing CreateTablet for tablet 769c4490062b4206951af2f4b9a6da05 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d98f711a8cd1404b9a01fa843b9a9c7f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:57.637627 14070 tablet_service.cc:1467] Processing CreateTablet for tablet cdf9885a1e7c4b8dbb160e370ca0f60c (DEFAULT_TABLE table=rebalance_test_table_0 [id=d98f711a8cd1404b9a01fa843b9a9c7f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:57.637670 13809 tablet_service.cc:1467] Processing CreateTablet for tablet 769c4490062b4206951af2f4b9a6da05 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d98f711a8cd1404b9a01fa843b9a9c7f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:57.638115 14070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdf9885a1e7c4b8dbb160e370ca0f60c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:57.638231 13808 tablet_service.cc:1467] Processing CreateTablet for tablet 4b680592b1774f09ba79789afa55a0f1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d98f711a8cd1404b9a01fa843b9a9c7f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:57.638607 13939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 769c4490062b4206951af2f4b9a6da05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:57.637749 13810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdf9885a1e7c4b8dbb160e370ca0f60c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:57.639462 13808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b680592b1774f09ba79789afa55a0f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:57.639783 13809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 769c4490062b4206951af2f4b9a6da05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:57.640825 13940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdf9885a1e7c4b8dbb160e370ca0f60c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:57.643517 14199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b680592b1774f09ba79789afa55a0f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:57.644560 14200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 769c4490062b4206951af2f4b9a6da05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:57.654386 14814 tablet_bootstrap.cc:492] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0: Bootstrap starting.
I20240503 08:14:57.657143 14815 tablet_bootstrap.cc:492] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f: Bootstrap starting.
I20240503 08:14:57.659747 14814 tablet_bootstrap.cc:654] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:57.660398 14815 tablet_bootstrap.cc:654] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:57.661237 14814 log.cc:824] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:57.661589 14815 log.cc:824] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:57.665374 14817 tablet_bootstrap.cc:492] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc: Bootstrap starting.
I20240503 08:14:57.668759 14817 tablet_bootstrap.cc:654] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:57.669435 14819 tablet_bootstrap.cc:492] T 769c4490062b4206951af2f4b9a6da05 P 5761f1165461459f9aaa8cd5d8e9b0b9: Bootstrap starting.
I20240503 08:14:57.670814 14817 log.cc:824] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:57.671063 14815 tablet_bootstrap.cc:492] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f: No bootstrap required, opened a new log
I20240503 08:14:57.671376 14815 ts_tablet_manager.cc:1367] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent bootstrapping tablet: real 0.014s user 0.013s sys 0.000s
I20240503 08:14:57.671581 14814 tablet_bootstrap.cc:492] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0: No bootstrap required, opened a new log
I20240503 08:14:57.671875 14814 ts_tablet_manager.cc:1367] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.013s
I20240503 08:14:57.672573 14819 tablet_bootstrap.cc:654] T 769c4490062b4206951af2f4b9a6da05 P 5761f1165461459f9aaa8cd5d8e9b0b9: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:57.673785 14819 log.cc:824] T 769c4490062b4206951af2f4b9a6da05 P 5761f1165461459f9aaa8cd5d8e9b0b9: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:57.679661 14815 raft_consensus.cc:348] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:14:57.680141 14815 raft_consensus.cc:374] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:57.680272 14815 raft_consensus.cc:729] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Initialized, Role: FOLLOWER
I20240503 08:14:57.680861 14814 raft_consensus.cc:348] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:14:57.681038 14815 consensus_queue.cc:260] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:14:57.681516 14814 raft_consensus.cc:374] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:57.681658 14814 raft_consensus.cc:729] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Initialized, Role: FOLLOWER
I20240503 08:14:57.682354 14814 consensus_queue.cc:260] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:14:57.683679 14819 tablet_bootstrap.cc:492] T 769c4490062b4206951af2f4b9a6da05 P 5761f1165461459f9aaa8cd5d8e9b0b9: No bootstrap required, opened a new log
I20240503 08:14:57.683713 14815 ts_tablet_manager.cc:1398] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20240503 08:14:57.684067 14815 tablet_bootstrap.cc:492] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f: Bootstrap starting.
I20240503 08:14:57.684801 14817 tablet_bootstrap.cc:492] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc: No bootstrap required, opened a new log
I20240503 08:14:57.685084 14817 ts_tablet_manager.cc:1367] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.004s
I20240503 08:14:57.685443 14815 tablet_bootstrap.cc:654] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:57.689056 14819 ts_tablet_manager.cc:1367] T 769c4490062b4206951af2f4b9a6da05 P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.011s
I20240503 08:14:57.690867 14814 ts_tablet_manager.cc:1398] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0: Time spent starting tablet: real 0.019s user 0.006s sys 0.010s
I20240503 08:14:57.691285 14814 tablet_bootstrap.cc:492] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0: Bootstrap starting.
I20240503 08:14:57.692755 14814 tablet_bootstrap.cc:654] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:57.693698 14817 raft_consensus.cc:348] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:14:57.695152 14815 tablet_bootstrap.cc:492] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f: No bootstrap required, opened a new log
I20240503 08:14:57.695331 14815 ts_tablet_manager.cc:1367] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20240503 08:14:57.696271 14815 raft_consensus.cc:348] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:14:57.696566 14817 raft_consensus.cc:374] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:57.696700 14817 raft_consensus.cc:729] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Initialized, Role: FOLLOWER
I20240503 08:14:57.697364 14817 consensus_queue.cc:260] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:14:57.697654 14815 raft_consensus.cc:374] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:57.697791 14815 raft_consensus.cc:729] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Initialized, Role: FOLLOWER
I20240503 08:14:57.698033 14815 consensus_queue.cc:260] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:14:57.698779 14815 ts_tablet_manager.cc:1398] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:14:57.698832 14814 tablet_bootstrap.cc:492] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0: No bootstrap required, opened a new log
I20240503 08:14:57.699040 14814 ts_tablet_manager.cc:1367] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20240503 08:14:57.698909 14819 raft_consensus.cc:348] T 769c4490062b4206951af2f4b9a6da05 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:14:57.699337 14819 raft_consensus.cc:374] T 769c4490062b4206951af2f4b9a6da05 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:57.699472 14819 raft_consensus.cc:729] T 769c4490062b4206951af2f4b9a6da05 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Initialized, Role: FOLLOWER
I20240503 08:14:57.700083 14819 consensus_queue.cc:260] T 769c4490062b4206951af2f4b9a6da05 P 5761f1165461459f9aaa8cd5d8e9b0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:14:57.701233 14814 raft_consensus.cc:348] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:14:57.701498 14814 raft_consensus.cc:374] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:57.701639 14814 raft_consensus.cc:729] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Initialized, Role: FOLLOWER
I20240503 08:14:57.702425 14819 ts_tablet_manager.cc:1398] T 769c4490062b4206951af2f4b9a6da05 P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent starting tablet: real 0.013s user 0.008s sys 0.001s
I20240503 08:14:57.701992 14814 consensus_queue.cc:260] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:14:57.703217 14819 tablet_bootstrap.cc:492] T 4b680592b1774f09ba79789afa55a0f1 P 5761f1165461459f9aaa8cd5d8e9b0b9: Bootstrap starting.
I20240503 08:14:57.704074 14814 ts_tablet_manager.cc:1398] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 08:14:57.704818 14817 ts_tablet_manager.cc:1398] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc: Time spent starting tablet: real 0.020s user 0.008s sys 0.004s
I20240503 08:14:57.705221 14817 tablet_bootstrap.cc:492] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc: Bootstrap starting.
I20240503 08:14:57.705590 14819 tablet_bootstrap.cc:654] T 4b680592b1774f09ba79789afa55a0f1 P 5761f1165461459f9aaa8cd5d8e9b0b9: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:57.706801 14817 tablet_bootstrap.cc:654] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc: Neither blocks nor log segments found. Creating new log.
W20240503 08:14:57.710374 14266 tablet.cc:2369] T 769c4490062b4206951af2f4b9a6da05 P 5761f1165461459f9aaa8cd5d8e9b0b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:14:57.710906 14819 tablet_bootstrap.cc:492] T 4b680592b1774f09ba79789afa55a0f1 P 5761f1165461459f9aaa8cd5d8e9b0b9: No bootstrap required, opened a new log
I20240503 08:14:57.711030 14819 ts_tablet_manager.cc:1367] T 4b680592b1774f09ba79789afa55a0f1 P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240503 08:14:57.711727 14817 tablet_bootstrap.cc:492] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc: No bootstrap required, opened a new log
I20240503 08:14:57.711848 14817 ts_tablet_manager.cc:1367] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20240503 08:14:57.711875 14819 raft_consensus.cc:348] T 4b680592b1774f09ba79789afa55a0f1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:14:57.712049 14819 raft_consensus.cc:374] T 4b680592b1774f09ba79789afa55a0f1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:57.712121 14819 raft_consensus.cc:729] T 4b680592b1774f09ba79789afa55a0f1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Initialized, Role: FOLLOWER
I20240503 08:14:57.712319 14819 consensus_queue.cc:260] T 4b680592b1774f09ba79789afa55a0f1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:14:57.712563 14817 raft_consensus.cc:348] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:14:57.712739 14817 raft_consensus.cc:374] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:57.712744 14819 ts_tablet_manager.cc:1398] T 4b680592b1774f09ba79789afa55a0f1 P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:14:57.712822 14817 raft_consensus.cc:729] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Initialized, Role: FOLLOWER
I20240503 08:14:57.713053 14817 consensus_queue.cc:260] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:14:57.713564 14817 ts_tablet_manager.cc:1398] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:14:57.714202 14817 tablet_bootstrap.cc:492] T cdf9885a1e7c4b8dbb160e370ca0f60c P 5a198ad76e4f462c8be0859184aeb2fc: Bootstrap starting.
I20240503 08:14:57.715523 14817 tablet_bootstrap.cc:654] T cdf9885a1e7c4b8dbb160e370ca0f60c P 5a198ad76e4f462c8be0859184aeb2fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:57.719372 14817 tablet_bootstrap.cc:492] T cdf9885a1e7c4b8dbb160e370ca0f60c P 5a198ad76e4f462c8be0859184aeb2fc: No bootstrap required, opened a new log
I20240503 08:14:57.719486 14817 ts_tablet_manager.cc:1367] T cdf9885a1e7c4b8dbb160e370ca0f60c P 5a198ad76e4f462c8be0859184aeb2fc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 08:14:57.719920 14817 raft_consensus.cc:348] T cdf9885a1e7c4b8dbb160e370ca0f60c P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:14:57.720068 14817 raft_consensus.cc:374] T cdf9885a1e7c4b8dbb160e370ca0f60c P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:57.720119 14817 raft_consensus.cc:729] T cdf9885a1e7c4b8dbb160e370ca0f60c P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Initialized, Role: FOLLOWER
I20240503 08:14:57.720283 14817 consensus_queue.cc:260] T cdf9885a1e7c4b8dbb160e370ca0f60c P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:14:57.720685 14817 ts_tablet_manager.cc:1398] T cdf9885a1e7c4b8dbb160e370ca0f60c P 5a198ad76e4f462c8be0859184aeb2fc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:14:57.725493 14826 raft_consensus.cc:482] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:57.725653 14826 raft_consensus.cc:504] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:14:57.727061 14826 leader_election.cc:288] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:14:57.728142 14822 raft_consensus.cc:482] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:57.728381 14822 raft_consensus.cc:504] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:14:57.729933 14822 leader_election.cc:288] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
W20240503 08:14:57.734794 13876 tablet.cc:2369] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:14:57.737705 14220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b680592b1774f09ba79789afa55a0f1" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" is_pre_election: true
I20240503 08:14:57.738116 14220 raft_consensus.cc:2454] T 4b680592b1774f09ba79789afa55a0f1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 0.
I20240503 08:14:57.738727 13765 leader_election.cc:302] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:14:57.739296 14826 raft_consensus.cc:2781] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:57.739151 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b680592b1774f09ba79789afa55a0f1" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" is_pre_election: true
I20240503 08:14:57.739445 14826 raft_consensus.cc:482] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:57.739542 14826 raft_consensus.cc:3037] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:57.739666 14090 raft_consensus.cc:2454] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 0.
I20240503 08:14:57.742408 14826 raft_consensus.cc:504] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:14:57.743110 14826 leader_election.cc:288] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 1 election: Requested vote from peers abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:14:57.743799 14220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b680592b1774f09ba79789afa55a0f1" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:14:57.743974 14220 raft_consensus.cc:3037] T 4b680592b1774f09ba79789afa55a0f1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:57.743959 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b680592b1774f09ba79789afa55a0f1" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:14:57.744153 14090 raft_consensus.cc:3037] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:57.744205 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdf9885a1e7c4b8dbb160e370ca0f60c" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc" is_pre_election: true
I20240503 08:14:57.744524 13830 raft_consensus.cc:2454] T cdf9885a1e7c4b8dbb160e370ca0f60c P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 0.
I20240503 08:14:57.744797 14089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdf9885a1e7c4b8dbb160e370ca0f60c" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" is_pre_election: true
I20240503 08:14:57.745080 14089 raft_consensus.cc:2454] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 0.
I20240503 08:14:57.745126 13894 leader_election.cc:302] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:14:57.745452 14822 raft_consensus.cc:2781] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:57.745570 14822 raft_consensus.cc:482] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:57.745653 14822 raft_consensus.cc:3037] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:57.746634 14090 raft_consensus.cc:2454] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 1.
I20240503 08:14:57.746767 14220 raft_consensus.cc:2454] T 4b680592b1774f09ba79789afa55a0f1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 1.
I20240503 08:14:57.747000 13766 leader_election.cc:302] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:14:57.747242 14826 raft_consensus.cc:2781] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:57.747629 14826 raft_consensus.cc:686] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:14:57.748068 14826 consensus_queue.cc:237] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:14:57.748157 14822 raft_consensus.cc:504] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:14:57.748651 14822 leader_election.cc:288] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 1 election: Requested vote from peers abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:14:57.749001 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdf9885a1e7c4b8dbb160e370ca0f60c" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:14:57.749073 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdf9885a1e7c4b8dbb160e370ca0f60c" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:14:57.749176 14090 raft_consensus.cc:3037] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:57.749231 13830 raft_consensus.cc:3037] T cdf9885a1e7c4b8dbb160e370ca0f60c P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:57.751546 14090 raft_consensus.cc:2454] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 1.
I20240503 08:14:57.751957 13830 raft_consensus.cc:2454] T cdf9885a1e7c4b8dbb160e370ca0f60c P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 1.
I20240503 08:14:57.751982 13896 leader_election.cc:302] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:14:57.752311 14822 raft_consensus.cc:2781] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:57.752748 14822 raft_consensus.cc:686] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:14:57.752538 13689 catalog_manager.cc:5484] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 1 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } }
I20240503 08:14:57.753242 14822 consensus_queue.cc:237] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:14:57.754700 14838 raft_consensus.cc:482] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:57.754822 14838 raft_consensus.cc:504] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:14:57.756789 14838 leader_election.cc:288] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:14:57.757181 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "769c4490062b4206951af2f4b9a6da05" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc" is_pre_election: true
I20240503 08:14:57.757371 13830 raft_consensus.cc:2454] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 0.
I20240503 08:14:57.757652 13894 leader_election.cc:302] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:14:57.757866 14838 raft_consensus.cc:2781] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:57.757956 14838 raft_consensus.cc:482] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:57.757751 13689 catalog_manager.cc:5484] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 1 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } }
I20240503 08:14:57.758033 14838 raft_consensus.cc:3037] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:57.760218 14838 raft_consensus.cc:504] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:14:57.760813 14838 leader_election.cc:288] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 1 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:14:57.761173 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "769c4490062b4206951af2f4b9a6da05" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:14:57.761333 13830 raft_consensus.cc:3037] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:57.763008 13830 raft_consensus.cc:2454] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 1.
I20240503 08:14:57.763273 14220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "769c4490062b4206951af2f4b9a6da05" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" is_pre_election: true
I20240503 08:14:57.763425 13894 leader_election.cc:302] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:14:57.763504 14220 raft_consensus.cc:2454] T 769c4490062b4206951af2f4b9a6da05 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 0.
I20240503 08:14:57.763379 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "769c4490062b4206951af2f4b9a6da05" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:14:57.763662 14219 raft_consensus.cc:3037] T 769c4490062b4206951af2f4b9a6da05 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:57.763689 14838 raft_consensus.cc:2781] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:57.763803 14838 raft_consensus.cc:686] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:14:57.764047 14838 consensus_queue.cc:237] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:14:57.765964 14219 raft_consensus.cc:2454] T 769c4490062b4206951af2f4b9a6da05 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 1.
I20240503 08:14:57.766321 13687 catalog_manager.cc:5484] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 1 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } }
W20240503 08:14:57.799546 14136 tablet.cc:2369] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:14:57.890691 14006 tablet.cc:2369] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:14:58.093639 14841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:58.093945 14841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:58.100836 14841 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 08:14:58.149391 14838 consensus_queue.cc:1035] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:58.156406 14822 consensus_queue.cc:1035] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:58.242153 13830 raft_consensus.cc:1261] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Refusing update from remote peer 296f3a209cf5447aaff2b8ea96a3bc4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:58.242528 14219 raft_consensus.cc:1261] T 769c4490062b4206951af2f4b9a6da05 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Refusing update from remote peer 296f3a209cf5447aaff2b8ea96a3bc4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:58.243362 14838 consensus_queue.cc:1035] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:58.244155 14838 consensus_queue.cc:1035] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:58.245026 14090 raft_consensus.cc:1261] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Refusing update from remote peer 5a198ad76e4f462c8be0859184aeb2fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:58.245658 14826 consensus_queue.cc:1035] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:58.252923 14220 raft_consensus.cc:1261] T 4b680592b1774f09ba79789afa55a0f1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Refusing update from remote peer 5a198ad76e4f462c8be0859184aeb2fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:58.257505 14826 consensus_queue.cc:1035] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:58.362679 14881 mvcc.cc:205] Tried to move back new op lower bound from 7023509906397442048 to 7023509904380190720. Current Snapshot: MvccSnapshot[applied={T|T < 7023509906397442048}]
W20240503 08:14:58.368221 13896 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.67:34647, user_credentials={real_user=slave}} blocked reactor thread for 73870.1us
INSERT report
rows total: 2000
time total: 261.806 ms
time per row: 0.130903 ms
I20240503 08:14:58.503803 13685 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44232:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:14:58.512794 14069 tablet_service.cc:1467] Processing CreateTablet for tablet d69d33529e1140e5bc3a0a561012dfa1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ef4b9e3af84a44598853463143377f4c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:58.512856 13940 tablet_service.cc:1467] Processing CreateTablet for tablet d69d33529e1140e5bc3a0a561012dfa1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ef4b9e3af84a44598853463143377f4c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:58.513305 14069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d69d33529e1140e5bc3a0a561012dfa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:58.513322 13940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d69d33529e1140e5bc3a0a561012dfa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:58.513585 14070 tablet_service.cc:1467] Processing CreateTablet for tablet d09e60e9c309425da9d27ae1153e4510 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ef4b9e3af84a44598853463143377f4c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:58.513994 14070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d09e60e9c309425da9d27ae1153e4510. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:58.514675 13809 tablet_service.cc:1467] Processing CreateTablet for tablet 32e141d0103848c1a2d473429cffcca9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ef4b9e3af84a44598853463143377f4c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:58.515071 13809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32e141d0103848c1a2d473429cffcca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:58.518180 14200 tablet_service.cc:1467] Processing CreateTablet for tablet d69d33529e1140e5bc3a0a561012dfa1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ef4b9e3af84a44598853463143377f4c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:58.518596 14200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d69d33529e1140e5bc3a0a561012dfa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:58.522131 13939 tablet_service.cc:1467] Processing CreateTablet for tablet d09e60e9c309425da9d27ae1153e4510 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ef4b9e3af84a44598853463143377f4c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:58.522306 13938 tablet_service.cc:1467] Processing CreateTablet for tablet 32e141d0103848c1a2d473429cffcca9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ef4b9e3af84a44598853463143377f4c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:58.522392 14199 tablet_service.cc:1467] Processing CreateTablet for tablet d09e60e9c309425da9d27ae1153e4510 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ef4b9e3af84a44598853463143377f4c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:58.522569 13939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d09e60e9c309425da9d27ae1153e4510. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:58.522699 14199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d09e60e9c309425da9d27ae1153e4510. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:58.522795 13938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32e141d0103848c1a2d473429cffcca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:58.525378 14910 tablet_bootstrap.cc:492] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc: Bootstrap starting.
I20240503 08:14:58.527220 14068 tablet_service.cc:1467] Processing CreateTablet for tablet 32e141d0103848c1a2d473429cffcca9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ef4b9e3af84a44598853463143377f4c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:58.527592 14910 tablet_bootstrap.cc:654] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:58.527659 14068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32e141d0103848c1a2d473429cffcca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:58.530203 14908 tablet_bootstrap.cc:492] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f: Bootstrap starting.
I20240503 08:14:58.531917 14908 tablet_bootstrap.cc:654] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:58.540395 14909 tablet_bootstrap.cc:492] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0: Bootstrap starting.
I20240503 08:14:58.543977 14909 tablet_bootstrap.cc:654] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:58.548529 14913 tablet_bootstrap.cc:492] T d69d33529e1140e5bc3a0a561012dfa1 P 5761f1165461459f9aaa8cd5d8e9b0b9: Bootstrap starting.
I20240503 08:14:58.551352 14913 tablet_bootstrap.cc:654] T d69d33529e1140e5bc3a0a561012dfa1 P 5761f1165461459f9aaa8cd5d8e9b0b9: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:58.553604 14908 tablet_bootstrap.cc:492] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f: No bootstrap required, opened a new log
I20240503 08:14:58.553797 14908 ts_tablet_manager.cc:1367] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent bootstrapping tablet: real 0.024s user 0.012s sys 0.011s
I20240503 08:14:58.553813 14910 tablet_bootstrap.cc:492] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc: No bootstrap required, opened a new log
I20240503 08:14:58.553972 14910 ts_tablet_manager.cc:1367] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.017s
I20240503 08:14:58.554725 14913 tablet_bootstrap.cc:492] T d69d33529e1140e5bc3a0a561012dfa1 P 5761f1165461459f9aaa8cd5d8e9b0b9: No bootstrap required, opened a new log
I20240503 08:14:58.554764 14910 raft_consensus.cc:348] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:14:58.554876 14913 ts_tablet_manager.cc:1367] T d69d33529e1140e5bc3a0a561012dfa1 P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 08:14:58.554955 14910 raft_consensus.cc:374] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:58.555042 14910 raft_consensus.cc:729] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Initialized, Role: FOLLOWER
I20240503 08:14:58.555101 14908 raft_consensus.cc:348] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:14:58.555336 14908 raft_consensus.cc:374] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:58.555287 14910 consensus_queue.cc:260] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:14:58.555434 14908 raft_consensus.cc:729] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Initialized, Role: FOLLOWER
I20240503 08:14:58.555661 14913 raft_consensus.cc:348] T d69d33529e1140e5bc3a0a561012dfa1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:14:58.555662 14908 consensus_queue.cc:260] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:14:58.555845 14913 raft_consensus.cc:374] T d69d33529e1140e5bc3a0a561012dfa1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:58.555925 14913 raft_consensus.cc:729] T d69d33529e1140e5bc3a0a561012dfa1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Initialized, Role: FOLLOWER
I20240503 08:14:58.555972 14910 ts_tablet_manager.cc:1398] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:14:58.556175 14908 ts_tablet_manager.cc:1398] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:14:58.556181 14913 consensus_queue.cc:260] T d69d33529e1140e5bc3a0a561012dfa1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:14:58.556666 14913 ts_tablet_manager.cc:1398] T d69d33529e1140e5bc3a0a561012dfa1 P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:58.556955 14913 tablet_bootstrap.cc:492] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9: Bootstrap starting.
I20240503 08:14:58.558326 14908 tablet_bootstrap.cc:492] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f: Bootstrap starting.
I20240503 08:14:58.558514 14913 tablet_bootstrap.cc:654] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:58.559886 14908 tablet_bootstrap.cc:654] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:58.559973 14909 tablet_bootstrap.cc:492] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0: No bootstrap required, opened a new log
I20240503 08:14:58.560132 14909 ts_tablet_manager.cc:1367] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.003s
I20240503 08:14:58.561084 14909 raft_consensus.cc:348] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:14:58.561305 14909 raft_consensus.cc:374] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:58.561393 14909 raft_consensus.cc:729] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Initialized, Role: FOLLOWER
I20240503 08:14:58.561637 14909 consensus_queue.cc:260] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:14:58.562228 14909 ts_tablet_manager.cc:1398] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:58.562544 14909 tablet_bootstrap.cc:492] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0: Bootstrap starting.
I20240503 08:14:58.564066 14909 tablet_bootstrap.cc:654] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:58.567663 14908 tablet_bootstrap.cc:492] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f: No bootstrap required, opened a new log
I20240503 08:14:58.567782 14913 tablet_bootstrap.cc:492] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9: No bootstrap required, opened a new log
I20240503 08:14:58.567831 14908 ts_tablet_manager.cc:1367] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20240503 08:14:58.567978 14913 ts_tablet_manager.cc:1367] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20240503 08:14:58.567171 14909 tablet_bootstrap.cc:492] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0: No bootstrap required, opened a new log
I20240503 08:14:58.568204 14909 ts_tablet_manager.cc:1367] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20240503 08:14:58.568605 14908 raft_consensus.cc:348] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:14:58.568794 14908 raft_consensus.cc:374] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:58.568734 14913 raft_consensus.cc:348] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:14:58.568861 14908 raft_consensus.cc:729] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Initialized, Role: FOLLOWER
I20240503 08:14:58.568949 14913 raft_consensus.cc:374] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:58.569020 14913 raft_consensus.cc:729] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Initialized, Role: FOLLOWER
I20240503 08:14:58.569125 14908 consensus_queue.cc:260] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:14:58.569231 14909 raft_consensus.cc:348] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:14:58.569301 14913 consensus_queue.cc:260] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:14:58.569806 14908 ts_tablet_manager.cc:1398] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:14:58.569895 14913 ts_tablet_manager.cc:1398] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 08:14:58.570112 14909 raft_consensus.cc:374] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:58.570204 14909 raft_consensus.cc:729] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Initialized, Role: FOLLOWER
I20240503 08:14:58.570289 14908 tablet_bootstrap.cc:492] T d09e60e9c309425da9d27ae1153e4510 P 296f3a209cf5447aaff2b8ea96a3bc4f: Bootstrap starting.
I20240503 08:14:58.570444 14909 consensus_queue.cc:260] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:14:58.570923 14909 ts_tablet_manager.cc:1398] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:14:58.571200 14909 tablet_bootstrap.cc:492] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0: Bootstrap starting.
I20240503 08:14:58.571862 14908 tablet_bootstrap.cc:654] T d09e60e9c309425da9d27ae1153e4510 P 296f3a209cf5447aaff2b8ea96a3bc4f: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:58.572770 14909 tablet_bootstrap.cc:654] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:58.576136 14909 tablet_bootstrap.cc:492] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0: No bootstrap required, opened a new log
I20240503 08:14:58.576144 14908 tablet_bootstrap.cc:492] T d09e60e9c309425da9d27ae1153e4510 P 296f3a209cf5447aaff2b8ea96a3bc4f: No bootstrap required, opened a new log
I20240503 08:14:58.576265 14909 ts_tablet_manager.cc:1367] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240503 08:14:58.576273 14908 ts_tablet_manager.cc:1367] T d09e60e9c309425da9d27ae1153e4510 P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20240503 08:14:58.576831 14908 raft_consensus.cc:348] T d09e60e9c309425da9d27ae1153e4510 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:14:58.576828 14909 raft_consensus.cc:348] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:14:58.576984 14909 raft_consensus.cc:374] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:58.576992 14908 raft_consensus.cc:374] T d09e60e9c309425da9d27ae1153e4510 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:58.577039 14909 raft_consensus.cc:729] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Initialized, Role: FOLLOWER
I20240503 08:14:58.577085 14908 raft_consensus.cc:729] T d09e60e9c309425da9d27ae1153e4510 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Initialized, Role: FOLLOWER
I20240503 08:14:58.577260 14909 consensus_queue.cc:260] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:14:58.577296 14908 consensus_queue.cc:260] T d09e60e9c309425da9d27ae1153e4510 P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:14:58.577733 14909 ts_tablet_manager.cc:1398] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:14:58.577908 14908 ts_tablet_manager.cc:1398] T d09e60e9c309425da9d27ae1153e4510 P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:14:58.590600 14875 raft_consensus.cc:482] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:58.590811 14875 raft_consensus.cc:504] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:14:58.592366 14875 leader_election.cc:288] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:14:58.599439 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d09e60e9c309425da9d27ae1153e4510" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" is_pre_election: true
I20240503 08:14:58.599855 13960 raft_consensus.cc:2454] T d09e60e9c309425da9d27ae1153e4510 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5761f1165461459f9aaa8cd5d8e9b0b9 in term 0.
I20240503 08:14:58.599929 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d09e60e9c309425da9d27ae1153e4510" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" is_pre_election: true
I20240503 08:14:58.600162 14090 raft_consensus.cc:2454] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5761f1165461459f9aaa8cd5d8e9b0b9 in term 0.
I20240503 08:14:58.600525 14153 leader_election.cc:302] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5761f1165461459f9aaa8cd5d8e9b0b9; no voters:
I20240503 08:14:58.600888 14875 raft_consensus.cc:2781] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:58.601002 14875 raft_consensus.cc:482] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:58.601090 14875 raft_consensus.cc:3037] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:58.603473 14875 raft_consensus.cc:504] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:14:58.604068 14875 leader_election.cc:288] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 1 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:14:58.604522 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d09e60e9c309425da9d27ae1153e4510" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:14:58.604583 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d09e60e9c309425da9d27ae1153e4510" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:14:58.604734 13960 raft_consensus.cc:3037] T d09e60e9c309425da9d27ae1153e4510 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:58.604770 14090 raft_consensus.cc:3037] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:58.607600 14090 raft_consensus.cc:2454] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5761f1165461459f9aaa8cd5d8e9b0b9 in term 1.
I20240503 08:14:58.607620 13960 raft_consensus.cc:2454] T d09e60e9c309425da9d27ae1153e4510 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5761f1165461459f9aaa8cd5d8e9b0b9 in term 1.
I20240503 08:14:58.608073 14153 leader_election.cc:302] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5761f1165461459f9aaa8cd5d8e9b0b9; no voters:
I20240503 08:14:58.608415 14875 raft_consensus.cc:2781] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:58.608822 14875 raft_consensus.cc:686] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 LEADER]: Becoming Leader. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Running, Role: LEADER
I20240503 08:14:58.609302 14875 consensus_queue.cc:237] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:14:58.612493 13683 catalog_manager.cc:5484] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68). New cstate: current_term: 1 leader_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:14:58.624945 14887 raft_consensus.cc:482] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:58.625108 14887 raft_consensus.cc:504] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:14:58.625744 14887 leader_election.cc:288] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:14:58.626199 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d69d33529e1140e5bc3a0a561012dfa1" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" is_pre_election: true
I20240503 08:14:58.626197 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d69d33529e1140e5bc3a0a561012dfa1" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" is_pre_election: true
I20240503 08:14:58.626402 14090 raft_consensus.cc:2454] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 0.
I20240503 08:14:58.626394 14219 raft_consensus.cc:2454] T d69d33529e1140e5bc3a0a561012dfa1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 0.
I20240503 08:14:58.626780 13895 leader_election.cc:302] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5761f1165461459f9aaa8cd5d8e9b0b9, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:14:58.627048 14887 raft_consensus.cc:2781] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:58.627154 14887 raft_consensus.cc:482] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:58.627249 14887 raft_consensus.cc:3037] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:58.629089 14887 raft_consensus.cc:504] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:14:58.629606 14887 leader_election.cc:288] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 1 election: Requested vote from peers abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:14:58.630079 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d69d33529e1140e5bc3a0a561012dfa1" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:14:58.630079 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d69d33529e1140e5bc3a0a561012dfa1" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:14:58.630252 14090 raft_consensus.cc:3037] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:58.630257 14219 raft_consensus.cc:3037] T d69d33529e1140e5bc3a0a561012dfa1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:58.632928 14219 raft_consensus.cc:2454] T d69d33529e1140e5bc3a0a561012dfa1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 1.
I20240503 08:14:58.632927 14090 raft_consensus.cc:2454] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 1.
I20240503 08:14:58.633283 13895 leader_election.cc:302] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5761f1165461459f9aaa8cd5d8e9b0b9; no voters:
I20240503 08:14:58.633563 14887 raft_consensus.cc:2781] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:58.633688 14887 raft_consensus.cc:686] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:14:58.633972 14887 consensus_queue.cc:237] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:14:58.636224 13683 catalog_manager.cc:5484] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 1 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } }
I20240503 08:14:58.723896 14826 raft_consensus.cc:482] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:58.724092 14826 raft_consensus.cc:504] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:14:58.724838 14826 leader_election.cc:288] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:14:58.725364 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32e141d0103848c1a2d473429cffcca9" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" is_pre_election: true
I20240503 08:14:58.725625 14090 raft_consensus.cc:2454] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 0.
I20240503 08:14:58.726054 13766 leader_election.cc:302] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:14:58.726320 14826 raft_consensus.cc:2781] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:58.726559 14826 raft_consensus.cc:482] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:58.726667 14826 raft_consensus.cc:3037] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:58.728699 14826 raft_consensus.cc:504] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:14:58.729391 14826 leader_election.cc:288] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 1 election: Requested vote from peers abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:14:58.729815 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32e141d0103848c1a2d473429cffcca9" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:14:58.730032 14090 raft_consensus.cc:3037] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:58.731781 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32e141d0103848c1a2d473429cffcca9" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" is_pre_election: true
I20240503 08:14:58.732002 13960 raft_consensus.cc:2454] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 0.
I20240503 08:14:58.731950 13959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32e141d0103848c1a2d473429cffcca9" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:14:58.732228 14090 raft_consensus.cc:2454] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 1.
I20240503 08:14:58.732259 13959 raft_consensus.cc:3037] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:58.732576 13766 leader_election.cc:302] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:14:58.732797 14826 raft_consensus.cc:2781] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:58.732934 14826 raft_consensus.cc:686] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:14:58.733168 14826 consensus_queue.cc:237] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:14:58.734381 13959 raft_consensus.cc:2454] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 1.
I20240503 08:14:58.735635 13683 catalog_manager.cc:5484] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 1 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } }
I20240503 08:14:58.998895 14875 consensus_queue.cc:1035] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:59.007714 14924 consensus_queue.cc:1035] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:14:59.095996 14929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:59.096347 14929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:59.106755 14929 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:14:59.124982 14887 consensus_queue.cc:1035] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:59.129050 14887 consensus_queue.cc:1035] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
I20240503 08:14:59.231853 14826 consensus_queue.cc:1035] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:59.278530 14976 consensus_queue.cc:1035] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:59.318758 14973 mvcc.cc:205] Tried to move back new op lower bound from 7023509910711267328 to 7023509908412907520. Current Snapshot: MvccSnapshot[applied={T|T < 7023509910711267328}]
INSERT report
rows total: 2000
time total: 196.399 ms
time per row: 0.0981997 ms
I20240503 08:14:59.439004 13686 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44232:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:14:59.448078 13809 tablet_service.cc:1467] Processing CreateTablet for tablet a478c28e5d4646fdb956f5b65e5c6c94 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1715d113c3ea43718e34016461353726]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:59.448413 13810 tablet_service.cc:1467] Processing CreateTablet for tablet de418f9b6bb04763b88ccc3bbe6cd68f (DEFAULT_TABLE table=rebalance_test_table_2 [id=1715d113c3ea43718e34016461353726]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:59.448565 13809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a478c28e5d4646fdb956f5b65e5c6c94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:59.448853 13810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de418f9b6bb04763b88ccc3bbe6cd68f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:59.448805 14200 tablet_service.cc:1467] Processing CreateTablet for tablet a478c28e5d4646fdb956f5b65e5c6c94 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1715d113c3ea43718e34016461353726]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:59.449218 14200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a478c28e5d4646fdb956f5b65e5c6c94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:59.451223 13939 tablet_service.cc:1467] Processing CreateTablet for tablet a478c28e5d4646fdb956f5b65e5c6c94 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1715d113c3ea43718e34016461353726]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:59.451639 13939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a478c28e5d4646fdb956f5b65e5c6c94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:59.454222 14070 tablet_service.cc:1467] Processing CreateTablet for tablet de418f9b6bb04763b88ccc3bbe6cd68f (DEFAULT_TABLE table=rebalance_test_table_2 [id=1715d113c3ea43718e34016461353726]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:59.454650 14070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de418f9b6bb04763b88ccc3bbe6cd68f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:59.458142 14992 tablet_bootstrap.cc:492] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f: Bootstrap starting.
I20240503 08:14:59.458614 14199 tablet_service.cc:1467] Processing CreateTablet for tablet de418f9b6bb04763b88ccc3bbe6cd68f (DEFAULT_TABLE table=rebalance_test_table_2 [id=1715d113c3ea43718e34016461353726]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:59.458634 14198 tablet_service.cc:1467] Processing CreateTablet for tablet f1e94dd86fc74c549979d30f0f705da7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1715d113c3ea43718e34016461353726]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:59.459093 14199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de418f9b6bb04763b88ccc3bbe6cd68f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:59.459300 14198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1e94dd86fc74c549979d30f0f705da7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:59.460216 14992 tablet_bootstrap.cc:654] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:59.462464 13938 tablet_service.cc:1467] Processing CreateTablet for tablet f1e94dd86fc74c549979d30f0f705da7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1715d113c3ea43718e34016461353726]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:59.462893 13938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1e94dd86fc74c549979d30f0f705da7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:59.463064 13809 tablet_service.cc:1467] Processing CreateTablet for tablet f1e94dd86fc74c549979d30f0f705da7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1715d113c3ea43718e34016461353726]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:59.463464 13809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1e94dd86fc74c549979d30f0f705da7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:59.474560 14993 tablet_bootstrap.cc:492] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0: Bootstrap starting.
I20240503 08:14:59.476253 14993 tablet_bootstrap.cc:654] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:59.476294 14994 tablet_bootstrap.cc:492] T a478c28e5d4646fdb956f5b65e5c6c94 P 5761f1165461459f9aaa8cd5d8e9b0b9: Bootstrap starting.
I20240503 08:14:59.476838 14991 tablet_bootstrap.cc:492] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc: Bootstrap starting.
I20240503 08:14:59.478801 14991 tablet_bootstrap.cc:654] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:59.481220 14993 tablet_bootstrap.cc:492] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0: No bootstrap required, opened a new log
I20240503 08:14:59.481351 14993 ts_tablet_manager.cc:1367] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20240503 08:14:59.482092 14993 raft_consensus.cc:348] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:14:59.482414 14993 raft_consensus.cc:374] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:59.482527 14993 raft_consensus.cc:729] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Initialized, Role: FOLLOWER
I20240503 08:14:59.482792 14993 consensus_queue.cc:260] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:14:59.483927 14994 tablet_bootstrap.cc:654] T a478c28e5d4646fdb956f5b65e5c6c94 P 5761f1165461459f9aaa8cd5d8e9b0b9: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:59.484270 14993 ts_tablet_manager.cc:1398] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:14:59.486002 14991 tablet_bootstrap.cc:492] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc: No bootstrap required, opened a new log
I20240503 08:14:59.486160 14991 ts_tablet_manager.cc:1367] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20240503 08:14:59.486865 14994 tablet_bootstrap.cc:492] T a478c28e5d4646fdb956f5b65e5c6c94 P 5761f1165461459f9aaa8cd5d8e9b0b9: No bootstrap required, opened a new log
I20240503 08:14:59.487067 14991 raft_consensus.cc:348] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:14:59.487318 14991 raft_consensus.cc:374] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:59.487344 14994 ts_tablet_manager.cc:1367] T a478c28e5d4646fdb956f5b65e5c6c94 P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240503 08:14:59.487430 14991 raft_consensus.cc:729] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Initialized, Role: FOLLOWER
I20240503 08:14:59.487668 14991 consensus_queue.cc:260] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:14:59.488555 14991 ts_tablet_manager.cc:1398] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:14:59.488720 14992 tablet_bootstrap.cc:492] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f: No bootstrap required, opened a new log
I20240503 08:14:59.488054 14994 raft_consensus.cc:348] T a478c28e5d4646fdb956f5b65e5c6c94 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:14:59.488806 14994 raft_consensus.cc:374] T a478c28e5d4646fdb956f5b65e5c6c94 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:59.488909 14994 raft_consensus.cc:729] T a478c28e5d4646fdb956f5b65e5c6c94 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Initialized, Role: FOLLOWER
I20240503 08:14:59.489001 14991 tablet_bootstrap.cc:492] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc: Bootstrap starting.
I20240503 08:14:59.489140 14994 consensus_queue.cc:260] T a478c28e5d4646fdb956f5b65e5c6c94 P 5761f1165461459f9aaa8cd5d8e9b0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:14:59.489632 14994 ts_tablet_manager.cc:1398] T a478c28e5d4646fdb956f5b65e5c6c94 P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:14:59.489920 14994 tablet_bootstrap.cc:492] T de418f9b6bb04763b88ccc3bbe6cd68f P 5761f1165461459f9aaa8cd5d8e9b0b9: Bootstrap starting.
I20240503 08:14:59.490356 14992 ts_tablet_manager.cc:1367] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.015s
I20240503 08:14:59.491055 14991 tablet_bootstrap.cc:654] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:59.491855 14994 tablet_bootstrap.cc:654] T de418f9b6bb04763b88ccc3bbe6cd68f P 5761f1165461459f9aaa8cd5d8e9b0b9: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:59.494338 14992 raft_consensus.cc:348] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:14:59.494592 14992 raft_consensus.cc:374] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:59.494701 14992 raft_consensus.cc:729] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Initialized, Role: FOLLOWER
I20240503 08:14:59.494961 14992 consensus_queue.cc:260] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:14:59.495250 14994 tablet_bootstrap.cc:492] T de418f9b6bb04763b88ccc3bbe6cd68f P 5761f1165461459f9aaa8cd5d8e9b0b9: No bootstrap required, opened a new log
I20240503 08:14:59.495409 14994 ts_tablet_manager.cc:1367] T de418f9b6bb04763b88ccc3bbe6cd68f P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 08:14:59.495800 14992 ts_tablet_manager.cc:1398] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 08:14:59.496088 14992 tablet_bootstrap.cc:492] T f1e94dd86fc74c549979d30f0f705da7 P 296f3a209cf5447aaff2b8ea96a3bc4f: Bootstrap starting.
I20240503 08:14:59.496099 14994 raft_consensus.cc:348] T de418f9b6bb04763b88ccc3bbe6cd68f P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:14:59.496311 14994 raft_consensus.cc:374] T de418f9b6bb04763b88ccc3bbe6cd68f P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:59.496372 14994 raft_consensus.cc:729] T de418f9b6bb04763b88ccc3bbe6cd68f P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Initialized, Role: FOLLOWER
I20240503 08:14:59.496598 14994 consensus_queue.cc:260] T de418f9b6bb04763b88ccc3bbe6cd68f P 5761f1165461459f9aaa8cd5d8e9b0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:14:59.497175 14994 ts_tablet_manager.cc:1398] T de418f9b6bb04763b88ccc3bbe6cd68f P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:59.497442 14994 tablet_bootstrap.cc:492] T f1e94dd86fc74c549979d30f0f705da7 P 5761f1165461459f9aaa8cd5d8e9b0b9: Bootstrap starting.
I20240503 08:14:59.498901 14994 tablet_bootstrap.cc:654] T f1e94dd86fc74c549979d30f0f705da7 P 5761f1165461459f9aaa8cd5d8e9b0b9: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:59.500670 14991 tablet_bootstrap.cc:492] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc: No bootstrap required, opened a new log
I20240503 08:14:59.500833 14991 ts_tablet_manager.cc:1367] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.002s
I20240503 08:14:59.501545 14991 raft_consensus.cc:348] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:14:59.501830 14991 raft_consensus.cc:374] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:59.502030 14991 raft_consensus.cc:729] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Initialized, Role: FOLLOWER
I20240503 08:14:59.502272 14994 tablet_bootstrap.cc:492] T f1e94dd86fc74c549979d30f0f705da7 P 5761f1165461459f9aaa8cd5d8e9b0b9: No bootstrap required, opened a new log
I20240503 08:14:59.502398 14994 ts_tablet_manager.cc:1367] T f1e94dd86fc74c549979d30f0f705da7 P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 08:14:59.502321 14991 consensus_queue.cc:260] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:14:59.502435 14992 tablet_bootstrap.cc:654] T f1e94dd86fc74c549979d30f0f705da7 P 296f3a209cf5447aaff2b8ea96a3bc4f: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:59.502959 14991 ts_tablet_manager.cc:1398] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:59.503247 14991 tablet_bootstrap.cc:492] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc: Bootstrap starting.
I20240503 08:14:59.503504 14994 raft_consensus.cc:348] T f1e94dd86fc74c549979d30f0f705da7 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:14:59.503671 14994 raft_consensus.cc:374] T f1e94dd86fc74c549979d30f0f705da7 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:59.503748 14994 raft_consensus.cc:729] T f1e94dd86fc74c549979d30f0f705da7 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Initialized, Role: FOLLOWER
I20240503 08:14:59.503969 14994 consensus_queue.cc:260] T f1e94dd86fc74c549979d30f0f705da7 P 5761f1165461459f9aaa8cd5d8e9b0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:14:59.505067 14994 ts_tablet_manager.cc:1398] T f1e94dd86fc74c549979d30f0f705da7 P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:14:59.505192 14991 tablet_bootstrap.cc:654] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:59.507283 14991 tablet_bootstrap.cc:492] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc: No bootstrap required, opened a new log
I20240503 08:14:59.507421 14991 ts_tablet_manager.cc:1367] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240503 08:14:59.508132 14991 raft_consensus.cc:348] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:14:59.508304 14991 raft_consensus.cc:374] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:59.509066 14991 raft_consensus.cc:729] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Initialized, Role: FOLLOWER
I20240503 08:14:59.509382 14991 consensus_queue.cc:260] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:14:59.509918 14991 ts_tablet_manager.cc:1398] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:14:59.510850 14992 tablet_bootstrap.cc:492] T f1e94dd86fc74c549979d30f0f705da7 P 296f3a209cf5447aaff2b8ea96a3bc4f: No bootstrap required, opened a new log
I20240503 08:14:59.510987 14992 ts_tablet_manager.cc:1367] T f1e94dd86fc74c549979d30f0f705da7 P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.001s
I20240503 08:14:59.511759 14992 raft_consensus.cc:348] T f1e94dd86fc74c549979d30f0f705da7 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:14:59.511958 14992 raft_consensus.cc:374] T f1e94dd86fc74c549979d30f0f705da7 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:59.512042 14992 raft_consensus.cc:729] T f1e94dd86fc74c549979d30f0f705da7 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Initialized, Role: FOLLOWER
I20240503 08:14:59.512257 14992 consensus_queue.cc:260] T f1e94dd86fc74c549979d30f0f705da7 P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:14:59.512663 14992 ts_tablet_manager.cc:1398] T f1e94dd86fc74c549979d30f0f705da7 P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:14:59.525460 14976 raft_consensus.cc:482] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:59.525594 14976 raft_consensus.cc:504] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:14:59.526257 14976 leader_election.cc:288] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:14:59.526626 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a478c28e5d4646fdb956f5b65e5c6c94" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" is_pre_election: true
I20240503 08:14:59.526693 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a478c28e5d4646fdb956f5b65e5c6c94" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" is_pre_election: true
I20240503 08:14:59.526826 14219 raft_consensus.cc:2454] T a478c28e5d4646fdb956f5b65e5c6c94 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 0.
I20240503 08:14:59.526894 13960 raft_consensus.cc:2454] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 0.
I20240503 08:14:59.527390 13765 leader_election.cc:302] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:14:59.527653 14976 raft_consensus.cc:2781] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:59.527731 14976 raft_consensus.cc:482] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:59.527804 14976 raft_consensus.cc:3037] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:59.529961 14976 raft_consensus.cc:504] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:14:59.530439 14976 leader_election.cc:288] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 1 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:14:59.530772 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a478c28e5d4646fdb956f5b65e5c6c94" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:14:59.530966 14219 raft_consensus.cc:3037] T a478c28e5d4646fdb956f5b65e5c6c94 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:59.530998 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a478c28e5d4646fdb956f5b65e5c6c94" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:14:59.531172 13960 raft_consensus.cc:3037] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:59.533124 14219 raft_consensus.cc:2454] T a478c28e5d4646fdb956f5b65e5c6c94 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 1.
I20240503 08:14:59.533228 13960 raft_consensus.cc:2454] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 1.
I20240503 08:14:59.533473 13765 leader_election.cc:302] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:14:59.533675 14976 raft_consensus.cc:2781] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:59.533851 14976 raft_consensus.cc:686] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:14:59.534099 14976 consensus_queue.cc:237] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:14:59.536360 13684 catalog_manager.cc:5484] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 1 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } }
I20240503 08:14:59.556998 14976 raft_consensus.cc:482] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:59.557123 14976 raft_consensus.cc:504] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:14:59.557636 14976 leader_election.cc:288] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:14:59.558005 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e94dd86fc74c549979d30f0f705da7" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" is_pre_election: true
I20240503 08:14:59.558058 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e94dd86fc74c549979d30f0f705da7" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" is_pre_election: true
I20240503 08:14:59.558167 13960 raft_consensus.cc:2454] T f1e94dd86fc74c549979d30f0f705da7 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 0.
I20240503 08:14:59.558270 14219 raft_consensus.cc:2454] T f1e94dd86fc74c549979d30f0f705da7 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 0.
I20240503 08:14:59.558481 13763 leader_election.cc:302] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:14:59.558720 14976 raft_consensus.cc:2781] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:59.558815 14976 raft_consensus.cc:482] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:59.558895 14976 raft_consensus.cc:3037] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:59.560662 14976 raft_consensus.cc:504] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:14:59.561149 14976 leader_election.cc:288] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 1 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:14:59.561519 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e94dd86fc74c549979d30f0f705da7" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:14:59.561553 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e94dd86fc74c549979d30f0f705da7" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:14:59.561667 13960 raft_consensus.cc:3037] T f1e94dd86fc74c549979d30f0f705da7 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:59.561678 14219 raft_consensus.cc:3037] T f1e94dd86fc74c549979d30f0f705da7 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:59.563345 13960 raft_consensus.cc:2454] T f1e94dd86fc74c549979d30f0f705da7 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 1.
I20240503 08:14:59.563380 14219 raft_consensus.cc:2454] T f1e94dd86fc74c549979d30f0f705da7 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 1.
I20240503 08:14:59.563665 13763 leader_election.cc:302] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:14:59.563938 14976 raft_consensus.cc:2781] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:59.564061 14976 raft_consensus.cc:686] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:14:59.564322 14976 consensus_queue.cc:237] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:14:59.566682 13684 catalog_manager.cc:5484] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 1 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } }
I20240503 08:14:59.635136 14976 raft_consensus.cc:482] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:59.635329 14976 raft_consensus.cc:504] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:14:59.636014 14976 leader_election.cc:288] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:14:59.636482 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de418f9b6bb04763b88ccc3bbe6cd68f" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" is_pre_election: true
I20240503 08:14:59.636540 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de418f9b6bb04763b88ccc3bbe6cd68f" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" is_pre_election: true
I20240503 08:14:59.636734 14219 raft_consensus.cc:2454] T de418f9b6bb04763b88ccc3bbe6cd68f P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 0.
I20240503 08:14:59.636740 14090 raft_consensus.cc:2454] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 0.
I20240503 08:14:59.637142 13765 leader_election.cc:302] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, 5a198ad76e4f462c8be0859184aeb2fc, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:14:59.637754 14976 raft_consensus.cc:2781] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:59.637867 14976 raft_consensus.cc:482] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:59.637950 14976 raft_consensus.cc:3037] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:59.639825 14976 raft_consensus.cc:504] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:14:59.640367 14976 leader_election.cc:288] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 1 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:14:59.640630 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de418f9b6bb04763b88ccc3bbe6cd68f" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:14:59.640799 14219 raft_consensus.cc:3037] T de418f9b6bb04763b88ccc3bbe6cd68f P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:59.640792 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de418f9b6bb04763b88ccc3bbe6cd68f" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:14:59.640973 14090 raft_consensus.cc:3037] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:59.642637 14219 raft_consensus.cc:2454] T de418f9b6bb04763b88ccc3bbe6cd68f P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 1.
I20240503 08:14:59.642735 14090 raft_consensus.cc:2454] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 1.
I20240503 08:14:59.642976 13765 leader_election.cc:302] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:14:59.643195 14976 raft_consensus.cc:2781] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:59.643362 14976 raft_consensus.cc:686] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [term 1 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:14:59.643636 14976 consensus_queue.cc:237] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:14:59.646024 13684 catalog_manager.cc:5484] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 1 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:14:59.930553 14976 consensus_queue.cc:1035] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:59.937865 14977 consensus_queue.cc:1035] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:59.960639 14977 consensus_queue.cc:1035] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:59.965159 14977 consensus_queue.cc:1035] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:14:59.980847 15007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:59.981221 15007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:59.989033 15007 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 08:15:00.041038 14977 consensus_queue.cc:1035] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:15:00.045786 15039 consensus_queue.cc:1035] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 208.78 ms
time per row: 0.10439 ms
I20240503 08:15:00.335767 13682 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44232:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:15:00.344841 13810 tablet_service.cc:1467] Processing CreateTablet for tablet d2ba977460694815a38d415a14e1cd5f (DEFAULT_TABLE table=rebalance_test_table_3 [id=01644c009611457a84b8e0af126ed83d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:00.345036 14070 tablet_service.cc:1467] Processing CreateTablet for tablet d2ba977460694815a38d415a14e1cd5f (DEFAULT_TABLE table=rebalance_test_table_3 [id=01644c009611457a84b8e0af126ed83d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:00.345373 13810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2ba977460694815a38d415a14e1cd5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:00.345489 14070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2ba977460694815a38d415a14e1cd5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:00.345528 14200 tablet_service.cc:1467] Processing CreateTablet for tablet 06e7243793e9467daeed79f3b1f5be5a (DEFAULT_TABLE table=rebalance_test_table_3 [id=01644c009611457a84b8e0af126ed83d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:00.345968 14200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06e7243793e9467daeed79f3b1f5be5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:00.349679 13939 tablet_service.cc:1467] Processing CreateTablet for tablet d2ba977460694815a38d415a14e1cd5f (DEFAULT_TABLE table=rebalance_test_table_3 [id=01644c009611457a84b8e0af126ed83d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:00.350136 13939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2ba977460694815a38d415a14e1cd5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:00.350288 13938 tablet_service.cc:1467] Processing CreateTablet for tablet 150f0d2d17bc4a59a8d82982790b4594 (DEFAULT_TABLE table=rebalance_test_table_3 [id=01644c009611457a84b8e0af126ed83d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:00.350656 13938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 150f0d2d17bc4a59a8d82982790b4594. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:00.350785 13809 tablet_service.cc:1467] Processing CreateTablet for tablet 06e7243793e9467daeed79f3b1f5be5a (DEFAULT_TABLE table=rebalance_test_table_3 [id=01644c009611457a84b8e0af126ed83d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:00.351212 13809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06e7243793e9467daeed79f3b1f5be5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:00.354454 14068 tablet_service.cc:1467] Processing CreateTablet for tablet 06e7243793e9467daeed79f3b1f5be5a (DEFAULT_TABLE table=rebalance_test_table_3 [id=01644c009611457a84b8e0af126ed83d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:00.354560 14069 tablet_service.cc:1467] Processing CreateTablet for tablet 150f0d2d17bc4a59a8d82982790b4594 (DEFAULT_TABLE table=rebalance_test_table_3 [id=01644c009611457a84b8e0af126ed83d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:00.354852 14068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06e7243793e9467daeed79f3b1f5be5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:00.354895 14199 tablet_service.cc:1467] Processing CreateTablet for tablet 150f0d2d17bc4a59a8d82982790b4594 (DEFAULT_TABLE table=rebalance_test_table_3 [id=01644c009611457a84b8e0af126ed83d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:00.355037 14069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 150f0d2d17bc4a59a8d82982790b4594. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:00.355270 14199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 150f0d2d17bc4a59a8d82982790b4594. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:00.364858 15071 tablet_bootstrap.cc:492] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc: Bootstrap starting.
I20240503 08:15:00.366780 15071 tablet_bootstrap.cc:654] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:00.369446 15073 tablet_bootstrap.cc:492] T 150f0d2d17bc4a59a8d82982790b4594 P 296f3a209cf5447aaff2b8ea96a3bc4f: Bootstrap starting.
I20240503 08:15:00.371217 15073 tablet_bootstrap.cc:654] T 150f0d2d17bc4a59a8d82982790b4594 P 296f3a209cf5447aaff2b8ea96a3bc4f: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:00.371946 15070 tablet_bootstrap.cc:492] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0: Bootstrap starting.
I20240503 08:15:00.373060 15072 tablet_bootstrap.cc:492] T 06e7243793e9467daeed79f3b1f5be5a P 5761f1165461459f9aaa8cd5d8e9b0b9: Bootstrap starting.
I20240503 08:15:00.374576 15070 tablet_bootstrap.cc:654] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:00.374780 15072 tablet_bootstrap.cc:654] T 06e7243793e9467daeed79f3b1f5be5a P 5761f1165461459f9aaa8cd5d8e9b0b9: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:00.375171 15071 tablet_bootstrap.cc:492] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc: No bootstrap required, opened a new log
I20240503 08:15:00.375339 15071 ts_tablet_manager.cc:1367] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20240503 08:15:00.376119 15071 raft_consensus.cc:348] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:00.376354 15071 raft_consensus.cc:374] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:00.376372 15070 tablet_bootstrap.cc:492] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0: No bootstrap required, opened a new log
I20240503 08:15:00.376451 15071 raft_consensus.cc:729] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Initialized, Role: FOLLOWER
I20240503 08:15:00.376494 15070 ts_tablet_manager.cc:1367] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 08:15:00.376541 15073 tablet_bootstrap.cc:492] T 150f0d2d17bc4a59a8d82982790b4594 P 296f3a209cf5447aaff2b8ea96a3bc4f: No bootstrap required, opened a new log
I20240503 08:15:00.376670 15073 ts_tablet_manager.cc:1367] T 150f0d2d17bc4a59a8d82982790b4594 P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 08:15:00.376703 15071 consensus_queue.cc:260] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:00.377166 15070 raft_consensus.cc:348] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:00.377369 15070 raft_consensus.cc:374] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:00.377415 15071 ts_tablet_manager.cc:1398] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240503 08:15:00.377452 15070 raft_consensus.cc:729] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Initialized, Role: FOLLOWER
I20240503 08:15:00.377370 15073 raft_consensus.cc:348] T 150f0d2d17bc4a59a8d82982790b4594 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:00.377562 15073 raft_consensus.cc:374] T 150f0d2d17bc4a59a8d82982790b4594 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:00.377676 15071 tablet_bootstrap.cc:492] T d2ba977460694815a38d415a14e1cd5f P 5a198ad76e4f462c8be0859184aeb2fc: Bootstrap starting.
I20240503 08:15:00.377694 15070 consensus_queue.cc:260] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:00.378300 15070 ts_tablet_manager.cc:1398] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:15:00.378561 15070 tablet_bootstrap.cc:492] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0: Bootstrap starting.
I20240503 08:15:00.379143 15071 tablet_bootstrap.cc:654] T d2ba977460694815a38d415a14e1cd5f P 5a198ad76e4f462c8be0859184aeb2fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:00.377642 15073 raft_consensus.cc:729] T 150f0d2d17bc4a59a8d82982790b4594 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Initialized, Role: FOLLOWER
I20240503 08:15:00.380089 15073 consensus_queue.cc:260] T 150f0d2d17bc4a59a8d82982790b4594 P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:00.380334 15070 tablet_bootstrap.cc:654] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:00.385002 15073 ts_tablet_manager.cc:1398] T 150f0d2d17bc4a59a8d82982790b4594 P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 08:15:00.385358 15073 tablet_bootstrap.cc:492] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f: Bootstrap starting.
I20240503 08:15:00.387073 15071 tablet_bootstrap.cc:492] T d2ba977460694815a38d415a14e1cd5f P 5a198ad76e4f462c8be0859184aeb2fc: No bootstrap required, opened a new log
I20240503 08:15:00.387238 15071 ts_tablet_manager.cc:1367] T d2ba977460694815a38d415a14e1cd5f P 5a198ad76e4f462c8be0859184aeb2fc: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20240503 08:15:00.387995 15071 raft_consensus.cc:348] T d2ba977460694815a38d415a14e1cd5f P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:00.388208 15073 tablet_bootstrap.cc:654] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:00.388237 15071 raft_consensus.cc:374] T d2ba977460694815a38d415a14e1cd5f P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:00.388310 15071 raft_consensus.cc:729] T d2ba977460694815a38d415a14e1cd5f P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Initialized, Role: FOLLOWER
I20240503 08:15:00.388552 15071 consensus_queue.cc:260] T d2ba977460694815a38d415a14e1cd5f P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:00.389071 15071 ts_tablet_manager.cc:1398] T d2ba977460694815a38d415a14e1cd5f P 5a198ad76e4f462c8be0859184aeb2fc: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:15:00.390046 15070 tablet_bootstrap.cc:492] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0: No bootstrap required, opened a new log
I20240503 08:15:00.390244 15070 ts_tablet_manager.cc:1367] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20240503 08:15:00.390620 15073 tablet_bootstrap.cc:492] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f: No bootstrap required, opened a new log
I20240503 08:15:00.390774 15073 ts_tablet_manager.cc:1367] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 08:15:00.391230 15072 tablet_bootstrap.cc:492] T 06e7243793e9467daeed79f3b1f5be5a P 5761f1165461459f9aaa8cd5d8e9b0b9: No bootstrap required, opened a new log
I20240503 08:15:00.391388 15072 ts_tablet_manager.cc:1367] T 06e7243793e9467daeed79f3b1f5be5a P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20240503 08:15:00.391451 15073 raft_consensus.cc:348] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:00.391429 15070 raft_consensus.cc:348] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:00.391650 15073 raft_consensus.cc:374] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:00.391654 15070 raft_consensus.cc:374] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:00.391714 15073 raft_consensus.cc:729] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Initialized, Role: FOLLOWER
I20240503 08:15:00.391726 15070 raft_consensus.cc:729] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Initialized, Role: FOLLOWER
I20240503 08:15:00.391955 15070 consensus_queue.cc:260] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:00.391978 15073 consensus_queue.cc:260] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:00.392549 15073 ts_tablet_manager.cc:1398] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:15:00.393352 15070 ts_tablet_manager.cc:1398] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:15:00.393330 15072 raft_consensus.cc:348] T 06e7243793e9467daeed79f3b1f5be5a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:00.393529 15072 raft_consensus.cc:374] T 06e7243793e9467daeed79f3b1f5be5a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:00.393623 15072 raft_consensus.cc:729] T 06e7243793e9467daeed79f3b1f5be5a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Initialized, Role: FOLLOWER
I20240503 08:15:00.393647 15070 tablet_bootstrap.cc:492] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0: Bootstrap starting.
I20240503 08:15:00.393894 15072 consensus_queue.cc:260] T 06e7243793e9467daeed79f3b1f5be5a P 5761f1165461459f9aaa8cd5d8e9b0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:00.394578 15072 ts_tablet_manager.cc:1398] T 06e7243793e9467daeed79f3b1f5be5a P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:15:00.394855 15072 tablet_bootstrap.cc:492] T 150f0d2d17bc4a59a8d82982790b4594 P 5761f1165461459f9aaa8cd5d8e9b0b9: Bootstrap starting.
I20240503 08:15:00.396400 15072 tablet_bootstrap.cc:654] T 150f0d2d17bc4a59a8d82982790b4594 P 5761f1165461459f9aaa8cd5d8e9b0b9: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:00.397825 15070 tablet_bootstrap.cc:654] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:00.399489 15072 tablet_bootstrap.cc:492] T 150f0d2d17bc4a59a8d82982790b4594 P 5761f1165461459f9aaa8cd5d8e9b0b9: No bootstrap required, opened a new log
I20240503 08:15:00.399636 15072 ts_tablet_manager.cc:1367] T 150f0d2d17bc4a59a8d82982790b4594 P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 08:15:00.399734 15070 tablet_bootstrap.cc:492] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0: No bootstrap required, opened a new log
I20240503 08:15:00.399859 15070 ts_tablet_manager.cc:1367] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20240503 08:15:00.400238 15072 raft_consensus.cc:348] T 150f0d2d17bc4a59a8d82982790b4594 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:00.400417 15072 raft_consensus.cc:374] T 150f0d2d17bc4a59a8d82982790b4594 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:00.400467 15072 raft_consensus.cc:729] T 150f0d2d17bc4a59a8d82982790b4594 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Initialized, Role: FOLLOWER
I20240503 08:15:00.400449 15070 raft_consensus.cc:348] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:00.400641 15070 raft_consensus.cc:374] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:00.400663 15072 consensus_queue.cc:260] T 150f0d2d17bc4a59a8d82982790b4594 P 5761f1165461459f9aaa8cd5d8e9b0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:00.400732 15070 raft_consensus.cc:729] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Initialized, Role: FOLLOWER
I20240503 08:15:00.400931 15070 consensus_queue.cc:260] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:00.401074 15072 ts_tablet_manager.cc:1398] T 150f0d2d17bc4a59a8d82982790b4594 P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:15:00.401546 15070 ts_tablet_manager.cc:1398] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:15:00.510500 15037 raft_consensus.cc:482] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:15:00.510694 15037 raft_consensus.cc:504] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:00.512162 15037 leader_election.cc:288] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:00.531733 14220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150f0d2d17bc4a59a8d82982790b4594" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" is_pre_election: true
I20240503 08:15:00.531940 14220 raft_consensus.cc:2454] T 150f0d2d17bc4a59a8d82982790b4594 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 0.
I20240503 08:15:00.532044 13959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150f0d2d17bc4a59a8d82982790b4594" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" is_pre_election: true
I20240503 08:15:00.532269 13959 raft_consensus.cc:2454] T 150f0d2d17bc4a59a8d82982790b4594 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 0.
I20240503 08:15:00.532506 14025 leader_election.cc:302] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:00.532912 15037 raft_consensus.cc:2781] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:15:00.533031 15037 raft_consensus.cc:482] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:15:00.533124 15037 raft_consensus.cc:3037] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:00.535179 15037 raft_consensus.cc:504] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:00.535753 15037 leader_election.cc:288] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:00.536111 13959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150f0d2d17bc4a59a8d82982790b4594" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:00.536206 14220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150f0d2d17bc4a59a8d82982790b4594" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:00.536315 13959 raft_consensus.cc:3037] T 150f0d2d17bc4a59a8d82982790b4594 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:00.536389 14220 raft_consensus.cc:3037] T 150f0d2d17bc4a59a8d82982790b4594 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:00.538973 13959 raft_consensus.cc:2454] T 150f0d2d17bc4a59a8d82982790b4594 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 1.
I20240503 08:15:00.539000 14220 raft_consensus.cc:2454] T 150f0d2d17bc4a59a8d82982790b4594 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 1.
I20240503 08:15:00.539340 14023 leader_election.cc:302] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:00.539588 15037 raft_consensus.cc:2781] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:15:00.539945 15037 raft_consensus.cc:686] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Becoming Leader. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:00.540380 15037 consensus_queue.cc:237] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:00.543152 13687 catalog_manager.cc:5484] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67). New cstate: current_term: 1 leader_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } }
I20240503 08:15:00.614261 15037 raft_consensus.cc:482] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:15:00.614444 15037 raft_consensus.cc:504] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:00.615147 15037 leader_election.cc:288] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:15:00.615700 13959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ba977460694815a38d415a14e1cd5f" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" is_pre_election: true
I20240503 08:15:00.615940 13959 raft_consensus.cc:2454] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 0.
I20240503 08:15:00.616298 14023 leader_election.cc:302] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:00.616626 15037 raft_consensus.cc:2781] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:15:00.616744 15037 raft_consensus.cc:482] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:15:00.616860 15037 raft_consensus.cc:3037] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:00.619202 15037 raft_consensus.cc:504] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:00.619760 15037 leader_election.cc:288] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:15:00.620342 13959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ba977460694815a38d415a14e1cd5f" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:00.620532 13959 raft_consensus.cc:3037] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:00.621209 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ba977460694815a38d415a14e1cd5f" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc" is_pre_election: true
I20240503 08:15:00.621361 13829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ba977460694815a38d415a14e1cd5f" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:00.621448 13830 raft_consensus.cc:2454] T d2ba977460694815a38d415a14e1cd5f P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 0.
I20240503 08:15:00.622793 13959 raft_consensus.cc:2454] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 1.
I20240503 08:15:00.623107 14023 leader_election.cc:302] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, abb3df711d3a46809274b5cfbd4ce4e0; no voters: 5a198ad76e4f462c8be0859184aeb2fc
I20240503 08:15:00.623374 15037 raft_consensus.cc:2781] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:15:00.623538 15037 raft_consensus.cc:686] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Becoming Leader. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:00.623780 15037 consensus_queue.cc:237] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:00.626216 13687 catalog_manager.cc:5484] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67). New cstate: current_term: 1 leader_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } }
I20240503 08:15:00.651001 15039 raft_consensus.cc:482] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:15:00.651167 15039 raft_consensus.cc:504] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:00.651757 15039 leader_election.cc:288] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:00.652168 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e7243793e9467daeed79f3b1f5be5a" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" is_pre_election: true
I20240503 08:15:00.652215 14220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e7243793e9467daeed79f3b1f5be5a" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" is_pre_election: true
I20240503 08:15:00.652351 14090 raft_consensus.cc:2454] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 0.
I20240503 08:15:00.652468 14220 raft_consensus.cc:2454] T 06e7243793e9467daeed79f3b1f5be5a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 0.
I20240503 08:15:00.652740 13766 leader_election.cc:302] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:00.653034 15039 raft_consensus.cc:2781] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:15:00.653159 15039 raft_consensus.cc:482] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:15:00.653261 15039 raft_consensus.cc:3037] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:00.655325 15039 raft_consensus.cc:504] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:00.655830 15039 leader_election.cc:288] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 1 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:00.656131 14220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e7243793e9467daeed79f3b1f5be5a" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:00.656219 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e7243793e9467daeed79f3b1f5be5a" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:00.656333 14220 raft_consensus.cc:3037] T 06e7243793e9467daeed79f3b1f5be5a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:00.656390 14090 raft_consensus.cc:3037] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:00.659016 14220 raft_consensus.cc:2454] T 06e7243793e9467daeed79f3b1f5be5a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 1.
I20240503 08:15:00.659016 14090 raft_consensus.cc:2454] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 1.
I20240503 08:15:00.659390 13766 leader_election.cc:302] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:00.659611 15039 raft_consensus.cc:2781] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:15:00.659766 15039 raft_consensus.cc:686] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [term 1 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:00.659989 15039 consensus_queue.cc:237] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:00.663378 13687 catalog_manager.cc:5484] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 1 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
W20240503 08:15:01.037087 15091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:15:01.037358 15091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:15:01.044312 15091 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 08:15:01.127207 13829 raft_consensus.cc:3037] T d2ba977460694815a38d415a14e1cd5f P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:01.128129 15115 consensus_queue.cc:1035] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:15:01.130937 15114 consensus_queue.cc:1035] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:15:01.151201 15121 consensus_queue.cc:1035] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:15:01.150408 15114 consensus_queue.cc:1035] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 08:15:01.180913 14089 raft_consensus.cc:1261] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Refusing update from remote peer 5a198ad76e4f462c8be0859184aeb2fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:15:01.182471 14220 raft_consensus.cc:1261] T 06e7243793e9467daeed79f3b1f5be5a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Refusing update from remote peer 5a198ad76e4f462c8be0859184aeb2fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:15:01.183523 15039 consensus_queue.cc:1035] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:15:01.188956 15134 consensus_queue.cc:1035] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:15:01.210990 15140 mvcc.cc:205] Tried to move back new op lower bound from 7023509918429175808 to 7023509916305203200. Current Snapshot: MvccSnapshot[applied={T|T < 7023509918429175808}]
INSERT report
rows total: 2000
time total: 203.786 ms
time per row: 0.101893 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 08:15:02.250383 13684 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44232:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:15:02.259104 13938 tablet_service.cc:1467] Processing CreateTablet for tablet 666be8071a904e6cbff984fa293d00b5 (DEFAULT_TABLE table=rebalance_test_table_4 [id=5f1b249fd55b4bd09733d8b97cecbcc2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:02.259600 13938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 666be8071a904e6cbff984fa293d00b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:02.259938 14070 tablet_service.cc:1467] Processing CreateTablet for tablet 666be8071a904e6cbff984fa293d00b5 (DEFAULT_TABLE table=rebalance_test_table_4 [id=5f1b249fd55b4bd09733d8b97cecbcc2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:02.260303 14070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 666be8071a904e6cbff984fa293d00b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:02.260398 13809 tablet_service.cc:1467] Processing CreateTablet for tablet 84204cbcda32446ab708717e9633af69 (DEFAULT_TABLE table=rebalance_test_table_4 [id=5f1b249fd55b4bd09733d8b97cecbcc2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:02.260854 13809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84204cbcda32446ab708717e9633af69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:02.260807 13808 tablet_service.cc:1467] Processing CreateTablet for tablet 9cae16af5a034cc29b8bb410a758c332 (DEFAULT_TABLE table=rebalance_test_table_4 [id=5f1b249fd55b4bd09733d8b97cecbcc2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:02.261143 13808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cae16af5a034cc29b8bb410a758c332. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:02.261507 14200 tablet_service.cc:1467] Processing CreateTablet for tablet 84204cbcda32446ab708717e9633af69 (DEFAULT_TABLE table=rebalance_test_table_4 [id=5f1b249fd55b4bd09733d8b97cecbcc2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:02.261906 14200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84204cbcda32446ab708717e9633af69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:02.265522 13810 tablet_service.cc:1467] Processing CreateTablet for tablet 666be8071a904e6cbff984fa293d00b5 (DEFAULT_TABLE table=rebalance_test_table_4 [id=5f1b249fd55b4bd09733d8b97cecbcc2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:02.265882 13810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 666be8071a904e6cbff984fa293d00b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:02.266551 13939 tablet_service.cc:1467] Processing CreateTablet for tablet 84204cbcda32446ab708717e9633af69 (DEFAULT_TABLE table=rebalance_test_table_4 [id=5f1b249fd55b4bd09733d8b97cecbcc2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:02.266878 13939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84204cbcda32446ab708717e9633af69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:02.271099 14199 tablet_service.cc:1467] Processing CreateTablet for tablet 9cae16af5a034cc29b8bb410a758c332 (DEFAULT_TABLE table=rebalance_test_table_4 [id=5f1b249fd55b4bd09733d8b97cecbcc2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:02.271463 14199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cae16af5a034cc29b8bb410a758c332. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:02.274515 14069 tablet_service.cc:1467] Processing CreateTablet for tablet 9cae16af5a034cc29b8bb410a758c332 (DEFAULT_TABLE table=rebalance_test_table_4 [id=5f1b249fd55b4bd09733d8b97cecbcc2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:02.274868 14069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cae16af5a034cc29b8bb410a758c332. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:02.279754 15162 tablet_bootstrap.cc:492] T 9cae16af5a034cc29b8bb410a758c332 P 5a198ad76e4f462c8be0859184aeb2fc: Bootstrap starting.
I20240503 08:15:02.279995 15163 tablet_bootstrap.cc:492] T 84204cbcda32446ab708717e9633af69 P 5761f1165461459f9aaa8cd5d8e9b0b9: Bootstrap starting.
I20240503 08:15:02.281334 15162 tablet_bootstrap.cc:654] T 9cae16af5a034cc29b8bb410a758c332 P 5a198ad76e4f462c8be0859184aeb2fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:02.284024 15163 tablet_bootstrap.cc:654] T 84204cbcda32446ab708717e9633af69 P 5761f1165461459f9aaa8cd5d8e9b0b9: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:02.292774 15163 tablet_bootstrap.cc:492] T 84204cbcda32446ab708717e9633af69 P 5761f1165461459f9aaa8cd5d8e9b0b9: No bootstrap required, opened a new log
I20240503 08:15:02.293116 15163 ts_tablet_manager.cc:1367] T 84204cbcda32446ab708717e9633af69 P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20240503 08:15:02.295617 15166 tablet_bootstrap.cc:492] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0: Bootstrap starting.
I20240503 08:15:02.300374 15166 tablet_bootstrap.cc:654] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:02.302023 15161 tablet_bootstrap.cc:492] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f: Bootstrap starting.
I20240503 08:15:02.302919 15162 tablet_bootstrap.cc:492] T 9cae16af5a034cc29b8bb410a758c332 P 5a198ad76e4f462c8be0859184aeb2fc: No bootstrap required, opened a new log
I20240503 08:15:02.303184 15162 ts_tablet_manager.cc:1367] T 9cae16af5a034cc29b8bb410a758c332 P 5a198ad76e4f462c8be0859184aeb2fc: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.019s
I20240503 08:15:02.303189 15166 tablet_bootstrap.cc:492] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0: No bootstrap required, opened a new log
I20240503 08:15:02.303332 15166 ts_tablet_manager.cc:1367] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240503 08:15:02.303591 15161 tablet_bootstrap.cc:654] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:02.303948 15162 raft_consensus.cc:348] T 9cae16af5a034cc29b8bb410a758c332 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:02.304178 15162 raft_consensus.cc:374] T 9cae16af5a034cc29b8bb410a758c332 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:02.304128 15166 raft_consensus.cc:348] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:02.304282 15162 raft_consensus.cc:729] T 9cae16af5a034cc29b8bb410a758c332 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Initialized, Role: FOLLOWER
I20240503 08:15:02.304369 15166 raft_consensus.cc:374] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:02.304490 15166 raft_consensus.cc:729] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Initialized, Role: FOLLOWER
I20240503 08:15:02.304528 15162 consensus_queue.cc:260] T 9cae16af5a034cc29b8bb410a758c332 P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:02.304739 15166 consensus_queue.cc:260] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:02.305088 15162 ts_tablet_manager.cc:1398] T 9cae16af5a034cc29b8bb410a758c332 P 5a198ad76e4f462c8be0859184aeb2fc: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:15:02.305006 15163 raft_consensus.cc:348] T 84204cbcda32446ab708717e9633af69 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:02.305449 15162 tablet_bootstrap.cc:492] T 666be8071a904e6cbff984fa293d00b5 P 5a198ad76e4f462c8be0859184aeb2fc: Bootstrap starting.
I20240503 08:15:02.305876 15166 ts_tablet_manager.cc:1398] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:15:02.306170 15166 tablet_bootstrap.cc:492] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0: Bootstrap starting.
I20240503 08:15:02.307020 15162 tablet_bootstrap.cc:654] T 666be8071a904e6cbff984fa293d00b5 P 5a198ad76e4f462c8be0859184aeb2fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:02.310761 15166 tablet_bootstrap.cc:654] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:02.313851 15163 raft_consensus.cc:374] T 84204cbcda32446ab708717e9633af69 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:02.314033 15163 raft_consensus.cc:729] T 84204cbcda32446ab708717e9633af69 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Initialized, Role: FOLLOWER
I20240503 08:15:02.314360 15163 consensus_queue.cc:260] T 84204cbcda32446ab708717e9633af69 P 5761f1165461459f9aaa8cd5d8e9b0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:02.317677 15163 ts_tablet_manager.cc:1398] T 84204cbcda32446ab708717e9633af69 P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20240503 08:15:02.320919 15166 tablet_bootstrap.cc:492] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0: No bootstrap required, opened a new log
I20240503 08:15:02.321148 15166 ts_tablet_manager.cc:1367] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.007s
I20240503 08:15:02.321955 15166 raft_consensus.cc:348] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:02.322261 15166 raft_consensus.cc:374] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:02.322381 15166 raft_consensus.cc:729] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Initialized, Role: FOLLOWER
I20240503 08:15:02.322669 15166 consensus_queue.cc:260] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:02.325080 15163 tablet_bootstrap.cc:492] T 9cae16af5a034cc29b8bb410a758c332 P 5761f1165461459f9aaa8cd5d8e9b0b9: Bootstrap starting.
I20240503 08:15:02.325440 15166 ts_tablet_manager.cc:1398] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:15:02.328894 15163 tablet_bootstrap.cc:654] T 9cae16af5a034cc29b8bb410a758c332 P 5761f1165461459f9aaa8cd5d8e9b0b9: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:02.332406 15161 tablet_bootstrap.cc:492] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f: No bootstrap required, opened a new log
I20240503 08:15:02.332473 15162 tablet_bootstrap.cc:492] T 666be8071a904e6cbff984fa293d00b5 P 5a198ad76e4f462c8be0859184aeb2fc: No bootstrap required, opened a new log
I20240503 08:15:02.332551 15161 ts_tablet_manager.cc:1367] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.022s
I20240503 08:15:02.332577 15163 tablet_bootstrap.cc:492] T 9cae16af5a034cc29b8bb410a758c332 P 5761f1165461459f9aaa8cd5d8e9b0b9: No bootstrap required, opened a new log
I20240503 08:15:02.332648 15162 ts_tablet_manager.cc:1367] T 666be8071a904e6cbff984fa293d00b5 P 5a198ad76e4f462c8be0859184aeb2fc: Time spent bootstrapping tablet: real 0.027s user 0.016s sys 0.011s
I20240503 08:15:02.332710 15163 ts_tablet_manager.cc:1367] T 9cae16af5a034cc29b8bb410a758c332 P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20240503 08:15:02.333173 15161 raft_consensus.cc:348] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:02.333335 15161 raft_consensus.cc:374] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:02.333338 15162 raft_consensus.cc:348] T 666be8071a904e6cbff984fa293d00b5 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:02.333454 15161 raft_consensus.cc:729] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Initialized, Role: FOLLOWER
I20240503 08:15:02.333395 15163 raft_consensus.cc:348] T 9cae16af5a034cc29b8bb410a758c332 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:02.333570 15163 raft_consensus.cc:374] T 9cae16af5a034cc29b8bb410a758c332 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:02.333568 15162 raft_consensus.cc:374] T 666be8071a904e6cbff984fa293d00b5 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:02.333642 15162 raft_consensus.cc:729] T 666be8071a904e6cbff984fa293d00b5 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Initialized, Role: FOLLOWER
I20240503 08:15:02.333683 15163 raft_consensus.cc:729] T 9cae16af5a034cc29b8bb410a758c332 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Initialized, Role: FOLLOWER
I20240503 08:15:02.333669 15161 consensus_queue.cc:260] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:02.333895 15162 consensus_queue.cc:260] T 666be8071a904e6cbff984fa293d00b5 P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:02.334024 15163 consensus_queue.cc:260] T 9cae16af5a034cc29b8bb410a758c332 P 5761f1165461459f9aaa8cd5d8e9b0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:02.334220 15161 ts_tablet_manager.cc:1398] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:15:02.334543 15163 ts_tablet_manager.cc:1398] T 9cae16af5a034cc29b8bb410a758c332 P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:15:02.334580 15161 tablet_bootstrap.cc:492] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f: Bootstrap starting.
I20240503 08:15:02.335083 15162 ts_tablet_manager.cc:1398] T 666be8071a904e6cbff984fa293d00b5 P 5a198ad76e4f462c8be0859184aeb2fc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:15:02.335337 15162 tablet_bootstrap.cc:492] T 84204cbcda32446ab708717e9633af69 P 5a198ad76e4f462c8be0859184aeb2fc: Bootstrap starting.
I20240503 08:15:02.336105 15161 tablet_bootstrap.cc:654] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:02.336766 15162 tablet_bootstrap.cc:654] T 84204cbcda32446ab708717e9633af69 P 5a198ad76e4f462c8be0859184aeb2fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:02.339450 15161 tablet_bootstrap.cc:492] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f: No bootstrap required, opened a new log
I20240503 08:15:02.339592 15161 ts_tablet_manager.cc:1367] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240503 08:15:02.340236 15161 raft_consensus.cc:348] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:02.340440 15161 raft_consensus.cc:374] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:02.340530 15161 raft_consensus.cc:729] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Initialized, Role: FOLLOWER
I20240503 08:15:02.340556 15162 tablet_bootstrap.cc:492] T 84204cbcda32446ab708717e9633af69 P 5a198ad76e4f462c8be0859184aeb2fc: No bootstrap required, opened a new log
I20240503 08:15:02.340670 15162 ts_tablet_manager.cc:1367] T 84204cbcda32446ab708717e9633af69 P 5a198ad76e4f462c8be0859184aeb2fc: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20240503 08:15:02.340754 15161 consensus_queue.cc:260] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:02.341259 15161 ts_tablet_manager.cc:1398] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:15:02.341471 15162 raft_consensus.cc:348] T 84204cbcda32446ab708717e9633af69 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:02.341637 15162 raft_consensus.cc:374] T 84204cbcda32446ab708717e9633af69 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:02.341724 15162 raft_consensus.cc:729] T 84204cbcda32446ab708717e9633af69 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Initialized, Role: FOLLOWER
I20240503 08:15:02.341943 15162 consensus_queue.cc:260] T 84204cbcda32446ab708717e9633af69 P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:02.342422 15162 ts_tablet_manager.cc:1398] T 84204cbcda32446ab708717e9633af69 P 5a198ad76e4f462c8be0859184aeb2fc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:15:02.389075 15156 raft_consensus.cc:482] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:15:02.389247 15156 raft_consensus.cc:504] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:02.389838 15156 leader_election.cc:288] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:02.390233 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84204cbcda32446ab708717e9633af69" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc" is_pre_election: true
I20240503 08:15:02.390292 14220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84204cbcda32446ab708717e9633af69" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" is_pre_election: true
I20240503 08:15:02.390456 13830 raft_consensus.cc:2454] T 84204cbcda32446ab708717e9633af69 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 0.
I20240503 08:15:02.390518 14220 raft_consensus.cc:2454] T 84204cbcda32446ab708717e9633af69 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 0.
I20240503 08:15:02.390872 13894 leader_election.cc:302] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:02.391129 15156 raft_consensus.cc:2781] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:15:02.391238 15156 raft_consensus.cc:482] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:15:02.391304 15156 raft_consensus.cc:3037] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:02.393102 15156 raft_consensus.cc:504] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:02.393582 15156 leader_election.cc:288] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 1 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:02.393970 14220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84204cbcda32446ab708717e9633af69" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:02.394052 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84204cbcda32446ab708717e9633af69" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:02.394174 14220 raft_consensus.cc:3037] T 84204cbcda32446ab708717e9633af69 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:02.394229 13830 raft_consensus.cc:3037] T 84204cbcda32446ab708717e9633af69 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:02.396819 13830 raft_consensus.cc:2454] T 84204cbcda32446ab708717e9633af69 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 1.
I20240503 08:15:02.396836 14220 raft_consensus.cc:2454] T 84204cbcda32446ab708717e9633af69 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 1.
I20240503 08:15:02.397192 13894 leader_election.cc:302] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:02.397486 15156 raft_consensus.cc:2781] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:15:02.397871 15156 raft_consensus.cc:686] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:02.398193 15156 consensus_queue.cc:237] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:02.400401 13687 catalog_manager.cc:5484] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 1 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } }
I20240503 08:15:02.517652 15114 raft_consensus.cc:482] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:15:02.517827 15114 raft_consensus.cc:504] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:02.518441 15114 leader_election.cc:288] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:02.518859 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "666be8071a904e6cbff984fa293d00b5" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" is_pre_election: true
I20240503 08:15:02.518906 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "666be8071a904e6cbff984fa293d00b5" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc" is_pre_election: true
I20240503 08:15:02.519073 13960 raft_consensus.cc:2454] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 0.
I20240503 08:15:02.519110 13830 raft_consensus.cc:2454] T 666be8071a904e6cbff984fa293d00b5 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 0.
I20240503 08:15:02.519495 14024 leader_election.cc:302] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:02.519775 15114 raft_consensus.cc:2781] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:15:02.519888 15114 raft_consensus.cc:482] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:15:02.519991 15114 raft_consensus.cc:3037] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:02.522235 15114 raft_consensus.cc:504] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:02.522732 15114 leader_election.cc:288] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:02.523084 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "666be8071a904e6cbff984fa293d00b5" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:02.523190 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "666be8071a904e6cbff984fa293d00b5" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:02.523276 13960 raft_consensus.cc:3037] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:02.523382 13830 raft_consensus.cc:3037] T 666be8071a904e6cbff984fa293d00b5 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:02.525969 13830 raft_consensus.cc:2454] T 666be8071a904e6cbff984fa293d00b5 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 1.
I20240503 08:15:02.525983 13960 raft_consensus.cc:2454] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 1.
I20240503 08:15:02.526350 14024 leader_election.cc:302] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5a198ad76e4f462c8be0859184aeb2fc, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:02.526629 15114 raft_consensus.cc:2781] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:15:02.526933 15114 raft_consensus.cc:686] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Becoming Leader. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:02.527230 15114 consensus_queue.cc:237] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:02.529989 13687 catalog_manager.cc:5484] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67). New cstate: current_term: 1 leader_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:02.530735 15114 raft_consensus.cc:482] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:15:02.530875 15114 raft_consensus.cc:504] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:02.531576 15114 leader_election.cc:288] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:02.531971 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cae16af5a034cc29b8bb410a758c332" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc" is_pre_election: true
I20240503 08:15:02.532166 13830 raft_consensus.cc:2454] T 9cae16af5a034cc29b8bb410a758c332 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 0.
I20240503 08:15:02.532199 14220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cae16af5a034cc29b8bb410a758c332" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" is_pre_election: true
I20240503 08:15:02.532416 14220 raft_consensus.cc:2454] T 9cae16af5a034cc29b8bb410a758c332 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 0.
I20240503 08:15:02.532737 14024 leader_election.cc:302] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:02.532979 15114 raft_consensus.cc:2781] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:15:02.533056 15114 raft_consensus.cc:482] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:15:02.533139 15114 raft_consensus.cc:3037] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:02.535059 15114 raft_consensus.cc:504] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:02.535643 15114 leader_election.cc:288] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:02.535954 14220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cae16af5a034cc29b8bb410a758c332" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:02.535979 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cae16af5a034cc29b8bb410a758c332" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:02.536101 14220 raft_consensus.cc:3037] T 9cae16af5a034cc29b8bb410a758c332 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:02.536131 13830 raft_consensus.cc:3037] T 9cae16af5a034cc29b8bb410a758c332 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:02.538110 14220 raft_consensus.cc:2454] T 9cae16af5a034cc29b8bb410a758c332 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 1.
I20240503 08:15:02.538281 13830 raft_consensus.cc:2454] T 9cae16af5a034cc29b8bb410a758c332 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 1.
I20240503 08:15:02.538462 14025 leader_election.cc:302] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:02.538698 15114 raft_consensus.cc:2781] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:15:02.538836 15114 raft_consensus.cc:686] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Becoming Leader. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:02.539039 15114 consensus_queue.cc:237] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:02.541150 13687 catalog_manager.cc:5484] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67). New cstate: current_term: 1 leader_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } }
W20240503 08:15:02.883915 15179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:15:02.884202 15179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:15:02.891705 15179 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 08:15:02.936980 15156 consensus_queue.cc:1035] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:15:02.954511 15156 consensus_queue.cc:1035] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:15:02.957267 15114 consensus_queue.cc:1035] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:15:03.016372 15114 consensus_queue.cc:1035] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:15:03.064559 15114 consensus_queue.cc:1035] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:15:03.092195 15210 consensus_queue.cc:1035] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 302.658 ms
time per row: 0.151329 ms
I20240503 08:15:03.331899 13681 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44232:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:15:03.340831 13938 tablet_service.cc:1467] Processing CreateTablet for tablet c310dd5906a24299b6f62201b30b86db (DEFAULT_TABLE table=rebalance_test_table_5 [id=05121a38ed284454b8c39a32cf7bbd18]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:03.341259 13938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c310dd5906a24299b6f62201b30b86db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:03.341150 14070 tablet_service.cc:1467] Processing CreateTablet for tablet c310dd5906a24299b6f62201b30b86db (DEFAULT_TABLE table=rebalance_test_table_5 [id=05121a38ed284454b8c39a32cf7bbd18]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:03.341578 14070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c310dd5906a24299b6f62201b30b86db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:03.342069 13808 tablet_service.cc:1467] Processing CreateTablet for tablet 94cfd3b764b148bb8b42e5a5cb7883b3 (DEFAULT_TABLE table=rebalance_test_table_5 [id=05121a38ed284454b8c39a32cf7bbd18]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:03.342444 13808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94cfd3b764b148bb8b42e5a5cb7883b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:03.342732 14199 tablet_service.cc:1467] Processing CreateTablet for tablet c310dd5906a24299b6f62201b30b86db (DEFAULT_TABLE table=rebalance_test_table_5 [id=05121a38ed284454b8c39a32cf7bbd18]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:03.343114 14199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c310dd5906a24299b6f62201b30b86db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:03.346184 14200 tablet_service.cc:1467] Processing CreateTablet for tablet f27cc9b78d084ac791de35b29e08a09a (DEFAULT_TABLE table=rebalance_test_table_5 [id=05121a38ed284454b8c39a32cf7bbd18]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:03.346534 14200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f27cc9b78d084ac791de35b29e08a09a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:03.350904 13940 tablet_service.cc:1467] Processing CreateTablet for tablet 94cfd3b764b148bb8b42e5a5cb7883b3 (DEFAULT_TABLE table=rebalance_test_table_5 [id=05121a38ed284454b8c39a32cf7bbd18]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:03.350975 13939 tablet_service.cc:1467] Processing CreateTablet for tablet f27cc9b78d084ac791de35b29e08a09a (DEFAULT_TABLE table=rebalance_test_table_5 [id=05121a38ed284454b8c39a32cf7bbd18]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:03.351531 14069 tablet_service.cc:1467] Processing CreateTablet for tablet f27cc9b78d084ac791de35b29e08a09a (DEFAULT_TABLE table=rebalance_test_table_5 [id=05121a38ed284454b8c39a32cf7bbd18]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:03.351949 14069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f27cc9b78d084ac791de35b29e08a09a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:03.354143 14068 tablet_service.cc:1467] Processing CreateTablet for tablet 94cfd3b764b148bb8b42e5a5cb7883b3 (DEFAULT_TABLE table=rebalance_test_table_5 [id=05121a38ed284454b8c39a32cf7bbd18]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:03.354350 15246 tablet_bootstrap.cc:492] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc: Bootstrap starting.
I20240503 08:15:03.354511 14068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94cfd3b764b148bb8b42e5a5cb7883b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:03.355939 15246 tablet_bootstrap.cc:654] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:03.359869 13940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94cfd3b764b148bb8b42e5a5cb7883b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:03.360517 15245 tablet_bootstrap.cc:492] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0: Bootstrap starting.
I20240503 08:15:03.362306 15245 tablet_bootstrap.cc:654] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:03.363951 15247 tablet_bootstrap.cc:492] T c310dd5906a24299b6f62201b30b86db P 5761f1165461459f9aaa8cd5d8e9b0b9: Bootstrap starting.
I20240503 08:15:03.364264 15246 tablet_bootstrap.cc:492] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc: No bootstrap required, opened a new log
I20240503 08:15:03.364392 15246 ts_tablet_manager.cc:1367] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240503 08:15:03.365151 15246 raft_consensus.cc:348] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:03.365342 15246 raft_consensus.cc:374] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:03.365428 15246 raft_consensus.cc:729] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Initialized, Role: FOLLOWER
I20240503 08:15:03.365556 15247 tablet_bootstrap.cc:654] T c310dd5906a24299b6f62201b30b86db P 5761f1165461459f9aaa8cd5d8e9b0b9: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:03.365667 15246 consensus_queue.cc:260] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:03.366215 15246 ts_tablet_manager.cc:1398] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:15:03.366355 13939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f27cc9b78d084ac791de35b29e08a09a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:03.367937 15244 tablet_bootstrap.cc:492] T c310dd5906a24299b6f62201b30b86db P 296f3a209cf5447aaff2b8ea96a3bc4f: Bootstrap starting.
I20240503 08:15:03.367982 15245 tablet_bootstrap.cc:492] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0: No bootstrap required, opened a new log
I20240503 08:15:03.368099 15245 ts_tablet_manager.cc:1367] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240503 08:15:03.368853 15245 raft_consensus.cc:348] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:03.369036 15245 raft_consensus.cc:374] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:03.369112 15245 raft_consensus.cc:729] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Initialized, Role: FOLLOWER
I20240503 08:15:03.369413 15245 consensus_queue.cc:260] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:03.370759 15245 ts_tablet_manager.cc:1398] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:15:03.370803 15244 tablet_bootstrap.cc:654] T c310dd5906a24299b6f62201b30b86db P 296f3a209cf5447aaff2b8ea96a3bc4f: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:03.372995 15244 tablet_bootstrap.cc:492] T c310dd5906a24299b6f62201b30b86db P 296f3a209cf5447aaff2b8ea96a3bc4f: No bootstrap required, opened a new log
I20240503 08:15:03.373133 15244 ts_tablet_manager.cc:1367] T c310dd5906a24299b6f62201b30b86db P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240503 08:15:03.373922 15244 raft_consensus.cc:348] T c310dd5906a24299b6f62201b30b86db P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:03.374106 15244 raft_consensus.cc:374] T c310dd5906a24299b6f62201b30b86db P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:03.374184 15244 raft_consensus.cc:729] T c310dd5906a24299b6f62201b30b86db P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Initialized, Role: FOLLOWER
I20240503 08:15:03.374402 15244 consensus_queue.cc:260] T c310dd5906a24299b6f62201b30b86db P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:03.374958 15244 ts_tablet_manager.cc:1398] T c310dd5906a24299b6f62201b30b86db P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:15:03.379678 15244 tablet_bootstrap.cc:492] T f27cc9b78d084ac791de35b29e08a09a P 296f3a209cf5447aaff2b8ea96a3bc4f: Bootstrap starting.
I20240503 08:15:03.381335 15244 tablet_bootstrap.cc:654] T f27cc9b78d084ac791de35b29e08a09a P 296f3a209cf5447aaff2b8ea96a3bc4f: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:03.381742 15245 tablet_bootstrap.cc:492] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0: Bootstrap starting.
I20240503 08:15:03.383229 15245 tablet_bootstrap.cc:654] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:03.385430 15244 tablet_bootstrap.cc:492] T f27cc9b78d084ac791de35b29e08a09a P 296f3a209cf5447aaff2b8ea96a3bc4f: No bootstrap required, opened a new log
I20240503 08:15:03.385458 15247 tablet_bootstrap.cc:492] T c310dd5906a24299b6f62201b30b86db P 5761f1165461459f9aaa8cd5d8e9b0b9: No bootstrap required, opened a new log
I20240503 08:15:03.385674 15244 ts_tablet_manager.cc:1367] T f27cc9b78d084ac791de35b29e08a09a P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20240503 08:15:03.385710 15247 ts_tablet_manager.cc:1367] T c310dd5906a24299b6f62201b30b86db P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.008s
I20240503 08:15:03.385917 15245 tablet_bootstrap.cc:492] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0: No bootstrap required, opened a new log
I20240503 08:15:03.386057 15245 ts_tablet_manager.cc:1367] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 08:15:03.386443 15244 raft_consensus.cc:348] T f27cc9b78d084ac791de35b29e08a09a P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:03.386471 15247 raft_consensus.cc:348] T c310dd5906a24299b6f62201b30b86db P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:03.386651 15247 raft_consensus.cc:374] T c310dd5906a24299b6f62201b30b86db P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:03.386737 15247 raft_consensus.cc:729] T c310dd5906a24299b6f62201b30b86db P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Initialized, Role: FOLLOWER
I20240503 08:15:03.386741 15244 raft_consensus.cc:374] T f27cc9b78d084ac791de35b29e08a09a P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:03.386807 15244 raft_consensus.cc:729] T f27cc9b78d084ac791de35b29e08a09a P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Initialized, Role: FOLLOWER
I20240503 08:15:03.386965 15247 consensus_queue.cc:260] T c310dd5906a24299b6f62201b30b86db P 5761f1165461459f9aaa8cd5d8e9b0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:03.387050 15244 consensus_queue.cc:260] T f27cc9b78d084ac791de35b29e08a09a P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:03.386968 15245 raft_consensus.cc:348] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:03.387296 15245 raft_consensus.cc:374] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:03.387387 15245 raft_consensus.cc:729] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Initialized, Role: FOLLOWER
I20240503 08:15:03.387483 15247 ts_tablet_manager.cc:1398] T c310dd5906a24299b6f62201b30b86db P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:15:03.387637 15244 ts_tablet_manager.cc:1398] T f27cc9b78d084ac791de35b29e08a09a P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:15:03.387643 15245 consensus_queue.cc:260] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:03.387913 15244 tablet_bootstrap.cc:492] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f: Bootstrap starting.
I20240503 08:15:03.388223 15245 ts_tablet_manager.cc:1398] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:15:03.388492 15245 tablet_bootstrap.cc:492] T f27cc9b78d084ac791de35b29e08a09a P abb3df711d3a46809274b5cfbd4ce4e0: Bootstrap starting.
I20240503 08:15:03.389329 15247 tablet_bootstrap.cc:492] T f27cc9b78d084ac791de35b29e08a09a P 5761f1165461459f9aaa8cd5d8e9b0b9: Bootstrap starting.
I20240503 08:15:03.389837 15244 tablet_bootstrap.cc:654] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:03.391287 15245 tablet_bootstrap.cc:654] T f27cc9b78d084ac791de35b29e08a09a P abb3df711d3a46809274b5cfbd4ce4e0: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:03.391718 15247 tablet_bootstrap.cc:654] T f27cc9b78d084ac791de35b29e08a09a P 5761f1165461459f9aaa8cd5d8e9b0b9: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:03.397085 15245 tablet_bootstrap.cc:492] T f27cc9b78d084ac791de35b29e08a09a P abb3df711d3a46809274b5cfbd4ce4e0: No bootstrap required, opened a new log
I20240503 08:15:03.397248 15244 tablet_bootstrap.cc:492] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f: No bootstrap required, opened a new log
I20240503 08:15:03.397255 15245 ts_tablet_manager.cc:1367] T f27cc9b78d084ac791de35b29e08a09a P abb3df711d3a46809274b5cfbd4ce4e0: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20240503 08:15:03.397368 15244 ts_tablet_manager.cc:1367] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20240503 08:15:03.397696 15247 tablet_bootstrap.cc:492] T f27cc9b78d084ac791de35b29e08a09a P 5761f1165461459f9aaa8cd5d8e9b0b9: No bootstrap required, opened a new log
I20240503 08:15:03.397881 15247 ts_tablet_manager.cc:1367] T f27cc9b78d084ac791de35b29e08a09a P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20240503 08:15:03.397971 15244 raft_consensus.cc:348] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:03.397987 15245 raft_consensus.cc:348] T f27cc9b78d084ac791de35b29e08a09a P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:03.398136 15244 raft_consensus.cc:374] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:03.398162 15245 raft_consensus.cc:374] T f27cc9b78d084ac791de35b29e08a09a P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:03.398232 15244 raft_consensus.cc:729] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Initialized, Role: FOLLOWER
I20240503 08:15:03.398239 15245 raft_consensus.cc:729] T f27cc9b78d084ac791de35b29e08a09a P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Initialized, Role: FOLLOWER
I20240503 08:15:03.398454 15244 consensus_queue.cc:260] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:03.398450 15245 consensus_queue.cc:260] T f27cc9b78d084ac791de35b29e08a09a P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:03.398586 15247 raft_consensus.cc:348] T f27cc9b78d084ac791de35b29e08a09a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:03.398804 15247 raft_consensus.cc:374] T f27cc9b78d084ac791de35b29e08a09a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:03.398897 15244 ts_tablet_manager.cc:1398] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 08:15:03.398953 15247 raft_consensus.cc:729] T f27cc9b78d084ac791de35b29e08a09a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Initialized, Role: FOLLOWER
I20240503 08:15:03.399112 15245 ts_tablet_manager.cc:1398] T f27cc9b78d084ac791de35b29e08a09a P abb3df711d3a46809274b5cfbd4ce4e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:15:03.399220 15247 consensus_queue.cc:260] T f27cc9b78d084ac791de35b29e08a09a P 5761f1165461459f9aaa8cd5d8e9b0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:03.400830 15247 ts_tablet_manager.cc:1398] T f27cc9b78d084ac791de35b29e08a09a P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:15:03.509164 15039 raft_consensus.cc:482] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:15:03.509373 15039 raft_consensus.cc:504] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:03.509969 15039 leader_election.cc:288] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:03.510445 14089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94cfd3b764b148bb8b42e5a5cb7883b3" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" is_pre_election: true
I20240503 08:15:03.510473 13959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94cfd3b764b148bb8b42e5a5cb7883b3" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" is_pre_election: true
I20240503 08:15:03.510680 14089 raft_consensus.cc:2454] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 0.
I20240503 08:15:03.510689 13959 raft_consensus.cc:2454] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 0.
I20240503 08:15:03.511094 13763 leader_election.cc:302] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:03.511356 15039 raft_consensus.cc:2781] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:15:03.511437 15039 raft_consensus.cc:482] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:15:03.511528 15039 raft_consensus.cc:3037] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:03.513315 15039 raft_consensus.cc:504] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:03.513854 15039 leader_election.cc:288] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 1 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:03.514201 13959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94cfd3b764b148bb8b42e5a5cb7883b3" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:03.514200 14089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94cfd3b764b148bb8b42e5a5cb7883b3" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:03.514408 13959 raft_consensus.cc:3037] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:03.514410 14089 raft_consensus.cc:3037] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:03.517112 14089 raft_consensus.cc:2454] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 1.
I20240503 08:15:03.517112 13959 raft_consensus.cc:2454] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 1.
I20240503 08:15:03.517473 13763 leader_election.cc:302] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:03.517721 15039 raft_consensus.cc:2781] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:15:03.518149 15039 raft_consensus.cc:686] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:03.518384 15039 consensus_queue.cc:237] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:03.520979 13681 catalog_manager.cc:5484] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 1 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:03.565531 15114 raft_consensus.cc:482] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:15:03.565711 15114 raft_consensus.cc:504] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:03.566381 15114 leader_election.cc:288] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:03.566794 13959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c310dd5906a24299b6f62201b30b86db" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" is_pre_election: true
I20240503 08:15:03.566859 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c310dd5906a24299b6f62201b30b86db" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" is_pre_election: true
I20240503 08:15:03.567016 13959 raft_consensus.cc:2454] T c310dd5906a24299b6f62201b30b86db P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 0.
I20240503 08:15:03.567047 14219 raft_consensus.cc:2454] T c310dd5906a24299b6f62201b30b86db P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 0.
I20240503 08:15:03.567431 14025 leader_election.cc:302] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:03.567740 15114 raft_consensus.cc:2781] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:15:03.567821 15114 raft_consensus.cc:482] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:15:03.567910 15114 raft_consensus.cc:3037] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:03.569667 15114 raft_consensus.cc:504] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:03.570219 15114 leader_election.cc:288] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:03.570559 13959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c310dd5906a24299b6f62201b30b86db" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:03.570631 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c310dd5906a24299b6f62201b30b86db" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:03.570717 13959 raft_consensus.cc:3037] T c310dd5906a24299b6f62201b30b86db P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:03.570788 14219 raft_consensus.cc:3037] T c310dd5906a24299b6f62201b30b86db P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:03.572432 13959 raft_consensus.cc:2454] T c310dd5906a24299b6f62201b30b86db P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 1.
I20240503 08:15:03.572602 14219 raft_consensus.cc:2454] T c310dd5906a24299b6f62201b30b86db P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 1.
I20240503 08:15:03.572773 14023 leader_election.cc:302] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:03.573040 15114 raft_consensus.cc:2781] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:15:03.573181 15114 raft_consensus.cc:686] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Becoming Leader. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:03.573410 15114 consensus_queue.cc:237] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:03.575989 13681 catalog_manager.cc:5484] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67). New cstate: current_term: 1 leader_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } }
I20240503 08:15:03.584321 15114 raft_consensus.cc:482] T f27cc9b78d084ac791de35b29e08a09a P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:15:03.584456 15114 raft_consensus.cc:504] T f27cc9b78d084ac791de35b29e08a09a P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:03.584949 15114 leader_election.cc:288] T f27cc9b78d084ac791de35b29e08a09a P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:03.585312 13959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f27cc9b78d084ac791de35b29e08a09a" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" is_pre_election: true
I20240503 08:15:03.585404 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f27cc9b78d084ac791de35b29e08a09a" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" is_pre_election: true
I20240503 08:15:03.585542 13959 raft_consensus.cc:2454] T f27cc9b78d084ac791de35b29e08a09a P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 0.
I20240503 08:15:03.585623 14219 raft_consensus.cc:2454] T f27cc9b78d084ac791de35b29e08a09a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 0.
I20240503 08:15:03.585937 14023 leader_election.cc:302] T f27cc9b78d084ac791de35b29e08a09a P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:03.586162 15114 raft_consensus.cc:2781] T f27cc9b78d084ac791de35b29e08a09a P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:15:03.586257 15114 raft_consensus.cc:482] T f27cc9b78d084ac791de35b29e08a09a P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:15:03.586310 15114 raft_consensus.cc:3037] T f27cc9b78d084ac791de35b29e08a09a P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:03.627436 15114 raft_consensus.cc:504] T f27cc9b78d084ac791de35b29e08a09a P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:03.628005 15114 leader_election.cc:288] T f27cc9b78d084ac791de35b29e08a09a P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:03.628429 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f27cc9b78d084ac791de35b29e08a09a" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:03.628479 13959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f27cc9b78d084ac791de35b29e08a09a" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:03.628648 14219 raft_consensus.cc:3037] T f27cc9b78d084ac791de35b29e08a09a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:03.628687 13959 raft_consensus.cc:3037] T f27cc9b78d084ac791de35b29e08a09a P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:03.630728 13959 raft_consensus.cc:2454] T f27cc9b78d084ac791de35b29e08a09a P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 1.
I20240503 08:15:03.630929 14219 raft_consensus.cc:2454] T f27cc9b78d084ac791de35b29e08a09a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 1.
I20240503 08:15:03.631134 14023 leader_election.cc:302] T f27cc9b78d084ac791de35b29e08a09a P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:03.631423 15114 raft_consensus.cc:2781] T f27cc9b78d084ac791de35b29e08a09a P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:15:03.631557 15114 raft_consensus.cc:686] T f27cc9b78d084ac791de35b29e08a09a P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Becoming Leader. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:03.631913 15114 consensus_queue.cc:237] T f27cc9b78d084ac791de35b29e08a09a P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:03.634574 13681 catalog_manager.cc:5484] T f27cc9b78d084ac791de35b29e08a09a P abb3df711d3a46809274b5cfbd4ce4e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67). New cstate: current_term: 1 leader_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } }
I20240503 08:15:03.907807 15039 consensus_queue.cc:1035] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:15:03.913177 15039 consensus_queue.cc:1035] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:15:03.967167 15114 consensus_queue.cc:1035] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:15:03.973537 15210 consensus_queue.cc:1035] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:15:04.001926 15261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:15:04.002228 15261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:15:04.009691 15261 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:15:04.010596 15210 consensus_queue.cc:1035] T f27cc9b78d084ac791de35b29e08a09a P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:15:04.015475 15210 consensus_queue.cc:1035] T f27cc9b78d084ac791de35b29e08a09a P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_5'
INSERT report
rows total: 2000
time total: 217.666 ms
time per row: 0.108833 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 08:15:05.242910 13680 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44232:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:15:05.251797 14199 tablet_service.cc:1467] Processing CreateTablet for tablet d7ebe785e08640cb866350eae5797e9f (DEFAULT_TABLE table=rebalance_test_table_6 [id=7b4f8ed5d46f4c988636db037108b354]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:05.252264 14199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7ebe785e08640cb866350eae5797e9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:05.252346 13940 tablet_service.cc:1467] Processing CreateTablet for tablet d7ebe785e08640cb866350eae5797e9f (DEFAULT_TABLE table=rebalance_test_table_6 [id=7b4f8ed5d46f4c988636db037108b354]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:05.252745 13940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7ebe785e08640cb866350eae5797e9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:05.252843 14068 tablet_service.cc:1467] Processing CreateTablet for tablet e82cb19fece24181b7607d9447ba1eb2 (DEFAULT_TABLE table=rebalance_test_table_6 [id=7b4f8ed5d46f4c988636db037108b354]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:05.253229 14068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e82cb19fece24181b7607d9447ba1eb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:05.256311 14200 tablet_service.cc:1467] Processing CreateTablet for tablet e82cb19fece24181b7607d9447ba1eb2 (DEFAULT_TABLE table=rebalance_test_table_6 [id=7b4f8ed5d46f4c988636db037108b354]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:05.256338 14198 tablet_service.cc:1467] Processing CreateTablet for tablet 377cd840a48f47238bdaa5ad2257e1f4 (DEFAULT_TABLE table=rebalance_test_table_6 [id=7b4f8ed5d46f4c988636db037108b354]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:05.256629 14200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e82cb19fece24181b7607d9447ba1eb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:05.257915 14198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 377cd840a48f47238bdaa5ad2257e1f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:05.258514 13808 tablet_service.cc:1467] Processing CreateTablet for tablet d7ebe785e08640cb866350eae5797e9f (DEFAULT_TABLE table=rebalance_test_table_6 [id=7b4f8ed5d46f4c988636db037108b354]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:05.258677 13939 tablet_service.cc:1467] Processing CreateTablet for tablet 377cd840a48f47238bdaa5ad2257e1f4 (DEFAULT_TABLE table=rebalance_test_table_6 [id=7b4f8ed5d46f4c988636db037108b354]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:05.258778 13809 tablet_service.cc:1467] Processing CreateTablet for tablet e82cb19fece24181b7607d9447ba1eb2 (DEFAULT_TABLE table=rebalance_test_table_6 [id=7b4f8ed5d46f4c988636db037108b354]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:05.258924 13808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7ebe785e08640cb866350eae5797e9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:05.258997 13939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 377cd840a48f47238bdaa5ad2257e1f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:05.259124 13809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e82cb19fece24181b7607d9447ba1eb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:05.263576 13810 tablet_service.cc:1467] Processing CreateTablet for tablet 377cd840a48f47238bdaa5ad2257e1f4 (DEFAULT_TABLE table=rebalance_test_table_6 [id=7b4f8ed5d46f4c988636db037108b354]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:05.263983 13810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 377cd840a48f47238bdaa5ad2257e1f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:05.269568 15333 tablet_bootstrap.cc:492] T d7ebe785e08640cb866350eae5797e9f P 5761f1165461459f9aaa8cd5d8e9b0b9: Bootstrap starting.
I20240503 08:15:05.271572 15333 tablet_bootstrap.cc:654] T d7ebe785e08640cb866350eae5797e9f P 5761f1165461459f9aaa8cd5d8e9b0b9: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:05.272221 15332 tablet_bootstrap.cc:492] T d7ebe785e08640cb866350eae5797e9f P 5a198ad76e4f462c8be0859184aeb2fc: Bootstrap starting.
I20240503 08:15:05.274718 15332 tablet_bootstrap.cc:654] T d7ebe785e08640cb866350eae5797e9f P 5a198ad76e4f462c8be0859184aeb2fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:05.276778 15334 tablet_bootstrap.cc:492] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f: Bootstrap starting.
I20240503 08:15:05.279178 15334 tablet_bootstrap.cc:654] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:05.280429 15331 tablet_bootstrap.cc:492] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0: Bootstrap starting.
I20240503 08:15:05.281422 15332 tablet_bootstrap.cc:492] T d7ebe785e08640cb866350eae5797e9f P 5a198ad76e4f462c8be0859184aeb2fc: No bootstrap required, opened a new log
I20240503 08:15:05.281554 15332 ts_tablet_manager.cc:1367] T d7ebe785e08640cb866350eae5797e9f P 5a198ad76e4f462c8be0859184aeb2fc: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20240503 08:15:05.282080 15331 tablet_bootstrap.cc:654] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:05.282501 15333 tablet_bootstrap.cc:492] T d7ebe785e08640cb866350eae5797e9f P 5761f1165461459f9aaa8cd5d8e9b0b9: No bootstrap required, opened a new log
I20240503 08:15:05.282719 15333 ts_tablet_manager.cc:1367] T d7ebe785e08640cb866350eae5797e9f P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.008s
I20240503 08:15:05.283612 15333 raft_consensus.cc:348] T d7ebe785e08640cb866350eae5797e9f P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:05.283937 15333 raft_consensus.cc:374] T d7ebe785e08640cb866350eae5797e9f P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:05.284413 15333 raft_consensus.cc:729] T d7ebe785e08640cb866350eae5797e9f P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Initialized, Role: FOLLOWER
I20240503 08:15:05.284785 15333 consensus_queue.cc:260] T d7ebe785e08640cb866350eae5797e9f P 5761f1165461459f9aaa8cd5d8e9b0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:05.287990 15333 ts_tablet_manager.cc:1398] T d7ebe785e08640cb866350eae5797e9f P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 08:15:05.289178 15334 tablet_bootstrap.cc:492] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f: No bootstrap required, opened a new log
I20240503 08:15:05.289328 15334 ts_tablet_manager.cc:1367] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20240503 08:15:05.289709 15331 tablet_bootstrap.cc:492] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0: No bootstrap required, opened a new log
I20240503 08:15:05.289871 15331 ts_tablet_manager.cc:1367] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240503 08:15:05.290048 15334 raft_consensus.cc:348] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:05.290237 15334 raft_consensus.cc:374] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:05.290329 15334 raft_consensus.cc:729] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Initialized, Role: FOLLOWER
I20240503 08:15:05.290555 15334 consensus_queue.cc:260] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:05.290601 15331 raft_consensus.cc:348] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:05.290825 15331 raft_consensus.cc:374] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:05.290972 15331 raft_consensus.cc:729] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Initialized, Role: FOLLOWER
I20240503 08:15:05.291165 15334 ts_tablet_manager.cc:1398] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:15:05.291280 15331 consensus_queue.cc:260] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:05.291563 15334 tablet_bootstrap.cc:492] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f: Bootstrap starting.
I20240503 08:15:05.292078 15331 ts_tablet_manager.cc:1398] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:15:05.293346 15334 tablet_bootstrap.cc:654] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:05.296869 15333 tablet_bootstrap.cc:492] T 377cd840a48f47238bdaa5ad2257e1f4 P 5761f1165461459f9aaa8cd5d8e9b0b9: Bootstrap starting.
I20240503 08:15:05.298590 15333 tablet_bootstrap.cc:654] T 377cd840a48f47238bdaa5ad2257e1f4 P 5761f1165461459f9aaa8cd5d8e9b0b9: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:05.303700 15332 raft_consensus.cc:348] T d7ebe785e08640cb866350eae5797e9f P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:05.304013 15332 raft_consensus.cc:374] T d7ebe785e08640cb866350eae5797e9f P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:05.304142 15332 raft_consensus.cc:729] T d7ebe785e08640cb866350eae5797e9f P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Initialized, Role: FOLLOWER
I20240503 08:15:05.304419 15332 consensus_queue.cc:260] T d7ebe785e08640cb866350eae5797e9f P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:05.305027 15332 ts_tablet_manager.cc:1398] T d7ebe785e08640cb866350eae5797e9f P 5a198ad76e4f462c8be0859184aeb2fc: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20240503 08:15:05.305358 15332 tablet_bootstrap.cc:492] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc: Bootstrap starting.
I20240503 08:15:05.307121 15332 tablet_bootstrap.cc:654] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:05.308779 15333 tablet_bootstrap.cc:492] T 377cd840a48f47238bdaa5ad2257e1f4 P 5761f1165461459f9aaa8cd5d8e9b0b9: No bootstrap required, opened a new log
I20240503 08:15:05.309005 15333 ts_tablet_manager.cc:1367] T 377cd840a48f47238bdaa5ad2257e1f4 P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20240503 08:15:05.309724 15333 raft_consensus.cc:348] T 377cd840a48f47238bdaa5ad2257e1f4 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:05.310000 15333 raft_consensus.cc:374] T 377cd840a48f47238bdaa5ad2257e1f4 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:05.310097 15333 raft_consensus.cc:729] T 377cd840a48f47238bdaa5ad2257e1f4 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Initialized, Role: FOLLOWER
I20240503 08:15:05.310359 15333 consensus_queue.cc:260] T 377cd840a48f47238bdaa5ad2257e1f4 P 5761f1165461459f9aaa8cd5d8e9b0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:05.311568 15334 tablet_bootstrap.cc:492] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f: No bootstrap required, opened a new log
I20240503 08:15:05.311676 15333 ts_tablet_manager.cc:1398] T 377cd840a48f47238bdaa5ad2257e1f4 P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:15:05.311787 15334 ts_tablet_manager.cc:1367] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.009s
I20240503 08:15:05.312006 15333 tablet_bootstrap.cc:492] T e82cb19fece24181b7607d9447ba1eb2 P 5761f1165461459f9aaa8cd5d8e9b0b9: Bootstrap starting.
I20240503 08:15:05.312589 15334 raft_consensus.cc:348] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:05.312830 15334 raft_consensus.cc:374] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:05.312918 15334 raft_consensus.cc:729] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Initialized, Role: FOLLOWER
I20240503 08:15:05.313201 15334 consensus_queue.cc:260] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:05.313625 15332 tablet_bootstrap.cc:492] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc: No bootstrap required, opened a new log
I20240503 08:15:05.313838 15332 ts_tablet_manager.cc:1367] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240503 08:15:05.313908 15334 ts_tablet_manager.cc:1398] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:15:05.314483 15332 raft_consensus.cc:348] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:05.314707 15332 raft_consensus.cc:374] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:05.314790 15332 raft_consensus.cc:729] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Initialized, Role: FOLLOWER
I20240503 08:15:05.314992 15332 consensus_queue.cc:260] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:05.315212 15333 tablet_bootstrap.cc:654] T e82cb19fece24181b7607d9447ba1eb2 P 5761f1165461459f9aaa8cd5d8e9b0b9: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:05.315404 15332 ts_tablet_manager.cc:1398] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:15:05.315723 15332 tablet_bootstrap.cc:492] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc: Bootstrap starting.
I20240503 08:15:05.317346 15332 tablet_bootstrap.cc:654] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:05.322971 15333 tablet_bootstrap.cc:492] T e82cb19fece24181b7607d9447ba1eb2 P 5761f1165461459f9aaa8cd5d8e9b0b9: No bootstrap required, opened a new log
I20240503 08:15:05.323105 15333 ts_tablet_manager.cc:1367] T e82cb19fece24181b7607d9447ba1eb2 P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.005s
I20240503 08:15:05.323207 15332 tablet_bootstrap.cc:492] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc: No bootstrap required, opened a new log
I20240503 08:15:05.323335 15332 ts_tablet_manager.cc:1367] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240503 08:15:05.323756 15333 raft_consensus.cc:348] T e82cb19fece24181b7607d9447ba1eb2 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:05.323843 15332 raft_consensus.cc:348] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:05.323940 15333 raft_consensus.cc:374] T e82cb19fece24181b7607d9447ba1eb2 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:05.323980 15332 raft_consensus.cc:374] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:05.324059 15333 raft_consensus.cc:729] T e82cb19fece24181b7607d9447ba1eb2 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Initialized, Role: FOLLOWER
I20240503 08:15:05.324093 15332 raft_consensus.cc:729] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Initialized, Role: FOLLOWER
I20240503 08:15:05.324339 15332 consensus_queue.cc:260] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:05.324303 15333 consensus_queue.cc:260] T e82cb19fece24181b7607d9447ba1eb2 P 5761f1165461459f9aaa8cd5d8e9b0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:05.324887 15332 ts_tablet_manager.cc:1398] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 08:15:05.325376 15333 ts_tablet_manager.cc:1398] T e82cb19fece24181b7607d9447ba1eb2 P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:15:05.376019 15321 raft_consensus.cc:482] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:15:05.376174 15321 raft_consensus.cc:504] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:05.376776 15321 leader_election.cc:288] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:05.377116 14220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7ebe785e08640cb866350eae5797e9f" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" is_pre_election: true
I20240503 08:15:05.377200 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7ebe785e08640cb866350eae5797e9f" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc" is_pre_election: true
I20240503 08:15:05.377333 14220 raft_consensus.cc:2454] T d7ebe785e08640cb866350eae5797e9f P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 0.
I20240503 08:15:05.377422 13830 raft_consensus.cc:2454] T d7ebe785e08640cb866350eae5797e9f P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 0.
I20240503 08:15:05.377781 13895 leader_election.cc:302] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5761f1165461459f9aaa8cd5d8e9b0b9; no voters:
I20240503 08:15:05.378032 15321 raft_consensus.cc:2781] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:15:05.378140 15321 raft_consensus.cc:482] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:15:05.378211 15321 raft_consensus.cc:3037] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:05.380020 15321 raft_consensus.cc:504] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:05.380581 15321 leader_election.cc:288] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 1 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:05.380882 14220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7ebe785e08640cb866350eae5797e9f" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:05.381086 14220 raft_consensus.cc:3037] T d7ebe785e08640cb866350eae5797e9f P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:05.381084 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7ebe785e08640cb866350eae5797e9f" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:05.381258 13830 raft_consensus.cc:3037] T d7ebe785e08640cb866350eae5797e9f P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:05.382894 14220 raft_consensus.cc:2454] T d7ebe785e08640cb866350eae5797e9f P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 1.
I20240503 08:15:05.383041 13830 raft_consensus.cc:2454] T d7ebe785e08640cb866350eae5797e9f P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 1.
I20240503 08:15:05.383230 13895 leader_election.cc:302] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5761f1165461459f9aaa8cd5d8e9b0b9; no voters:
I20240503 08:15:05.383483 15321 raft_consensus.cc:2781] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:15:05.383638 15321 raft_consensus.cc:686] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:05.383913 15321 consensus_queue.cc:237] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:05.386413 13681 catalog_manager.cc:5484] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 1 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } }
I20240503 08:15:05.504772 15039 raft_consensus.cc:482] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:15:05.504928 15039 raft_consensus.cc:504] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:05.505474 15039 leader_election.cc:288] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:05.505913 14220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82cb19fece24181b7607d9447ba1eb2" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" is_pre_election: true
I20240503 08:15:05.505947 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82cb19fece24181b7607d9447ba1eb2" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" is_pre_election: true
I20240503 08:15:05.506104 14220 raft_consensus.cc:2454] T e82cb19fece24181b7607d9447ba1eb2 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 0.
I20240503 08:15:05.506122 14090 raft_consensus.cc:2454] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 0.
I20240503 08:15:05.506462 13765 leader_election.cc:302] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:05.506737 15039 raft_consensus.cc:2781] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:15:05.506841 15039 raft_consensus.cc:482] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:15:05.506927 15039 raft_consensus.cc:3037] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:05.508919 15039 raft_consensus.cc:504] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:05.509393 15039 leader_election.cc:288] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 1 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:05.509835 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82cb19fece24181b7607d9447ba1eb2" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:05.509855 14220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82cb19fece24181b7607d9447ba1eb2" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:05.509991 14090 raft_consensus.cc:3037] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:05.510012 14220 raft_consensus.cc:3037] T e82cb19fece24181b7607d9447ba1eb2 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:05.511724 14090 raft_consensus.cc:2454] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 1.
I20240503 08:15:05.511855 14220 raft_consensus.cc:2454] T e82cb19fece24181b7607d9447ba1eb2 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 1.
I20240503 08:15:05.512106 13766 leader_election.cc:302] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:05.512387 15039 raft_consensus.cc:2781] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:15:05.512727 15039 raft_consensus.cc:686] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:05.513021 15039 consensus_queue.cc:237] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:05.515383 13681 catalog_manager.cc:5484] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 1 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:05.642519 15039 raft_consensus.cc:482] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:15:05.642750 15039 raft_consensus.cc:504] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:05.643358 15039 leader_election.cc:288] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:05.643703 13958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377cd840a48f47238bdaa5ad2257e1f4" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" is_pre_election: true
I20240503 08:15:05.643862 14220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377cd840a48f47238bdaa5ad2257e1f4" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" is_pre_election: true
I20240503 08:15:05.643936 13958 raft_consensus.cc:2454] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 0.
I20240503 08:15:05.644088 14220 raft_consensus.cc:2454] T 377cd840a48f47238bdaa5ad2257e1f4 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 0.
I20240503 08:15:05.644354 13763 leader_election.cc:302] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:05.644645 15039 raft_consensus.cc:2781] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:15:05.644738 15039 raft_consensus.cc:482] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:15:05.644837 15039 raft_consensus.cc:3037] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:05.646737 15039 raft_consensus.cc:504] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:05.647322 15039 leader_election.cc:288] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 1 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:05.647634 13958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377cd840a48f47238bdaa5ad2257e1f4" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:05.647792 13958 raft_consensus.cc:3037] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:05.647747 14220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377cd840a48f47238bdaa5ad2257e1f4" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:05.647958 14220 raft_consensus.cc:3037] T 377cd840a48f47238bdaa5ad2257e1f4 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:05.649966 14220 raft_consensus.cc:2454] T 377cd840a48f47238bdaa5ad2257e1f4 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 1.
I20240503 08:15:05.649967 13958 raft_consensus.cc:2454] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 1.
I20240503 08:15:05.650326 13765 leader_election.cc:302] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:05.650636 15346 raft_consensus.cc:2781] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:15:05.650771 15346 raft_consensus.cc:686] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:05.651046 15346 consensus_queue.cc:237] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:05.653880 13681 catalog_manager.cc:5484] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 1 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } }
I20240503 08:15:05.803447 15321 consensus_queue.cc:1035] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:15:05.809454 15344 consensus_queue.cc:1035] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:15:05.992398 15039 consensus_queue.cc:1035] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:15:05.996479 15039 consensus_queue.cc:1035] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:15:06.044029 15350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:15:06.044334 15350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:15:06.052402 15350 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240503 08:15:06.196856 14220 raft_consensus.cc:1261] T 377cd840a48f47238bdaa5ad2257e1f4 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Refusing update from remote peer 5a198ad76e4f462c8be0859184aeb2fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:15:06.197660 15039 consensus_queue.cc:1035] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:15:06.260020 13960 raft_consensus.cc:1261] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Refusing update from remote peer 5a198ad76e4f462c8be0859184aeb2fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:15:06.261976 15398 consensus_queue.cc:1035] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:15:06.355577 15404 mvcc.cc:205] Tried to move back new op lower bound from 7023509938971500544 to 7023509936748634112. Current Snapshot: MvccSnapshot[applied={T|T < 7023509938971500544}]
INSERT report
rows total: 2000
time total: 283.857 ms
time per row: 0.141928 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 08:15:07.329512 13687 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44232:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:15:07.338289 13810 tablet_service.cc:1467] Processing CreateTablet for tablet dd3d6353358d4b02af928cd9f9410aca (DEFAULT_TABLE table=rebalance_test_table_7 [id=3facf8544202407f89043c19bfcf3813]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:07.338488 13939 tablet_service.cc:1467] Processing CreateTablet for tablet dd3d6353358d4b02af928cd9f9410aca (DEFAULT_TABLE table=rebalance_test_table_7 [id=3facf8544202407f89043c19bfcf3813]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:07.338786 13810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd3d6353358d4b02af928cd9f9410aca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:07.338874 13939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd3d6353358d4b02af928cd9f9410aca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:07.339883 14200 tablet_service.cc:1467] Processing CreateTablet for tablet 70e8bfaa82464d3ab3338a779457dc02 (DEFAULT_TABLE table=rebalance_test_table_7 [id=3facf8544202407f89043c19bfcf3813]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:07.340319 14200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70e8bfaa82464d3ab3338a779457dc02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:07.340437 14198 tablet_service.cc:1467] Processing CreateTablet for tablet 79014e600505478883fc90159784ce6c (DEFAULT_TABLE table=rebalance_test_table_7 [id=3facf8544202407f89043c19bfcf3813]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:07.340727 14198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79014e600505478883fc90159784ce6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:07.342594 14068 tablet_service.cc:1467] Processing CreateTablet for tablet dd3d6353358d4b02af928cd9f9410aca (DEFAULT_TABLE table=rebalance_test_table_7 [id=3facf8544202407f89043c19bfcf3813]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:07.342706 13808 tablet_service.cc:1467] Processing CreateTablet for tablet 79014e600505478883fc90159784ce6c (DEFAULT_TABLE table=rebalance_test_table_7 [id=3facf8544202407f89043c19bfcf3813]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:07.342970 14068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd3d6353358d4b02af928cd9f9410aca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:07.342991 13808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79014e600505478883fc90159784ce6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:07.345960 14070 tablet_service.cc:1467] Processing CreateTablet for tablet 79014e600505478883fc90159784ce6c (DEFAULT_TABLE table=rebalance_test_table_7 [id=3facf8544202407f89043c19bfcf3813]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:07.346293 14070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79014e600505478883fc90159784ce6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:07.348428 14069 tablet_service.cc:1467] Processing CreateTablet for tablet 70e8bfaa82464d3ab3338a779457dc02 (DEFAULT_TABLE table=rebalance_test_table_7 [id=3facf8544202407f89043c19bfcf3813]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:07.348803 14069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70e8bfaa82464d3ab3338a779457dc02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:07.355737 13940 tablet_service.cc:1467] Processing CreateTablet for tablet 70e8bfaa82464d3ab3338a779457dc02 (DEFAULT_TABLE table=rebalance_test_table_7 [id=3facf8544202407f89043c19bfcf3813]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:07.356137 13940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70e8bfaa82464d3ab3338a779457dc02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:07.358248 15424 tablet_bootstrap.cc:492] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0: Bootstrap starting.
I20240503 08:15:07.358781 15422 tablet_bootstrap.cc:492] T 70e8bfaa82464d3ab3338a779457dc02 P 5761f1165461459f9aaa8cd5d8e9b0b9: Bootstrap starting.
I20240503 08:15:07.360504 15424 tablet_bootstrap.cc:654] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:07.362120 15422 tablet_bootstrap.cc:654] T 70e8bfaa82464d3ab3338a779457dc02 P 5761f1165461459f9aaa8cd5d8e9b0b9: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:07.365327 15423 tablet_bootstrap.cc:492] T 79014e600505478883fc90159784ce6c P 5a198ad76e4f462c8be0859184aeb2fc: Bootstrap starting.
I20240503 08:15:07.366652 15422 tablet_bootstrap.cc:492] T 70e8bfaa82464d3ab3338a779457dc02 P 5761f1165461459f9aaa8cd5d8e9b0b9: No bootstrap required, opened a new log
I20240503 08:15:07.367089 15423 tablet_bootstrap.cc:654] T 79014e600505478883fc90159784ce6c P 5a198ad76e4f462c8be0859184aeb2fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:07.367092 15422 ts_tablet_manager.cc:1367] T 70e8bfaa82464d3ab3338a779457dc02 P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20240503 08:15:07.367962 15422 raft_consensus.cc:348] T 70e8bfaa82464d3ab3338a779457dc02 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:07.368245 15426 tablet_bootstrap.cc:492] T 70e8bfaa82464d3ab3338a779457dc02 P 296f3a209cf5447aaff2b8ea96a3bc4f: Bootstrap starting.
I20240503 08:15:07.368270 15422 raft_consensus.cc:374] T 70e8bfaa82464d3ab3338a779457dc02 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:07.368420 15422 raft_consensus.cc:729] T 70e8bfaa82464d3ab3338a779457dc02 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Initialized, Role: FOLLOWER
I20240503 08:15:07.368680 15422 consensus_queue.cc:260] T 70e8bfaa82464d3ab3338a779457dc02 P 5761f1165461459f9aaa8cd5d8e9b0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:07.369304 15422 ts_tablet_manager.cc:1398] T 70e8bfaa82464d3ab3338a779457dc02 P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:15:07.369781 15426 tablet_bootstrap.cc:654] T 70e8bfaa82464d3ab3338a779457dc02 P 296f3a209cf5447aaff2b8ea96a3bc4f: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:07.370393 15424 tablet_bootstrap.cc:492] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0: No bootstrap required, opened a new log
I20240503 08:15:07.370535 15424 ts_tablet_manager.cc:1367] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240503 08:15:07.371265 15424 raft_consensus.cc:348] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:07.371451 15424 raft_consensus.cc:374] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:07.371544 15424 raft_consensus.cc:729] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Initialized, Role: FOLLOWER
I20240503 08:15:07.371799 15424 consensus_queue.cc:260] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:07.372121 15423 tablet_bootstrap.cc:492] T 79014e600505478883fc90159784ce6c P 5a198ad76e4f462c8be0859184aeb2fc: No bootstrap required, opened a new log
I20240503 08:15:07.372253 15423 ts_tablet_manager.cc:1367] T 79014e600505478883fc90159784ce6c P 5a198ad76e4f462c8be0859184aeb2fc: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 08:15:07.372309 15424 ts_tablet_manager.cc:1398] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:15:07.372534 15424 tablet_bootstrap.cc:492] T 70e8bfaa82464d3ab3338a779457dc02 P abb3df711d3a46809274b5cfbd4ce4e0: Bootstrap starting.
I20240503 08:15:07.372931 15423 raft_consensus.cc:348] T 79014e600505478883fc90159784ce6c P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:07.373149 15423 raft_consensus.cc:374] T 79014e600505478883fc90159784ce6c P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:07.373227 15423 raft_consensus.cc:729] T 79014e600505478883fc90159784ce6c P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Initialized, Role: FOLLOWER
I20240503 08:15:07.373454 15423 consensus_queue.cc:260] T 79014e600505478883fc90159784ce6c P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:07.374146 15422 tablet_bootstrap.cc:492] T 79014e600505478883fc90159784ce6c P 5761f1165461459f9aaa8cd5d8e9b0b9: Bootstrap starting.
I20240503 08:15:07.375439 15424 tablet_bootstrap.cc:654] T 70e8bfaa82464d3ab3338a779457dc02 P abb3df711d3a46809274b5cfbd4ce4e0: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:07.376479 15423 ts_tablet_manager.cc:1398] T 79014e600505478883fc90159784ce6c P 5a198ad76e4f462c8be0859184aeb2fc: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:15:07.376730 15422 tablet_bootstrap.cc:654] T 79014e600505478883fc90159784ce6c P 5761f1165461459f9aaa8cd5d8e9b0b9: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:07.377630 15423 tablet_bootstrap.cc:492] T dd3d6353358d4b02af928cd9f9410aca P 5a198ad76e4f462c8be0859184aeb2fc: Bootstrap starting.
I20240503 08:15:07.379076 15423 tablet_bootstrap.cc:654] T dd3d6353358d4b02af928cd9f9410aca P 5a198ad76e4f462c8be0859184aeb2fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:07.380304 15417 raft_consensus.cc:482] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:15:07.380439 15417 raft_consensus.cc:504] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:07.380960 15426 tablet_bootstrap.cc:492] T 70e8bfaa82464d3ab3338a779457dc02 P 296f3a209cf5447aaff2b8ea96a3bc4f: No bootstrap required, opened a new log
I20240503 08:15:07.381089 15426 ts_tablet_manager.cc:1367] T 70e8bfaa82464d3ab3338a779457dc02 P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20240503 08:15:07.381130 15417 leader_election.cc:288] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:07.381856 15426 raft_consensus.cc:348] T 70e8bfaa82464d3ab3338a779457dc02 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:07.382036 15426 raft_consensus.cc:374] T 70e8bfaa82464d3ab3338a779457dc02 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:07.382158 15426 raft_consensus.cc:729] T 70e8bfaa82464d3ab3338a779457dc02 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Initialized, Role: FOLLOWER
I20240503 08:15:07.382373 15426 consensus_queue.cc:260] T 70e8bfaa82464d3ab3338a779457dc02 P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:07.382483 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79014e600505478883fc90159784ce6c" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc" is_pre_election: true
I20240503 08:15:07.382699 13830 raft_consensus.cc:2454] T 79014e600505478883fc90159784ce6c P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 0.
I20240503 08:15:07.383065 15426 ts_tablet_manager.cc:1398] T 70e8bfaa82464d3ab3338a779457dc02 P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:15:07.383106 14024 leader_election.cc:302] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:07.383332 15426 tablet_bootstrap.cc:492] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f: Bootstrap starting.
I20240503 08:15:07.384805 15426 tablet_bootstrap.cc:654] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:07.384945 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79014e600505478883fc90159784ce6c" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" is_pre_election: true
W20240503 08:15:07.385710 14025 leader_election.cc:341] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:15:07.386575 15417 raft_consensus.cc:2781] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:15:07.386690 15417 raft_consensus.cc:482] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:15:07.386775 15417 raft_consensus.cc:3037] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:07.388855 15424 tablet_bootstrap.cc:492] T 70e8bfaa82464d3ab3338a779457dc02 P abb3df711d3a46809274b5cfbd4ce4e0: No bootstrap required, opened a new log
I20240503 08:15:07.389017 15424 ts_tablet_manager.cc:1367] T 70e8bfaa82464d3ab3338a779457dc02 P abb3df711d3a46809274b5cfbd4ce4e0: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.005s
I20240503 08:15:07.389389 15417 raft_consensus.cc:504] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:07.389740 15424 raft_consensus.cc:348] T 70e8bfaa82464d3ab3338a779457dc02 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:07.389976 15424 raft_consensus.cc:374] T 70e8bfaa82464d3ab3338a779457dc02 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:07.390064 15417 leader_election.cc:288] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:07.390056 15424 raft_consensus.cc:729] T 70e8bfaa82464d3ab3338a779457dc02 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Initialized, Role: FOLLOWER
I20240503 08:15:07.390514 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79014e600505478883fc90159784ce6c" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:07.390626 15424 consensus_queue.cc:260] T 70e8bfaa82464d3ab3338a779457dc02 P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
W20240503 08:15:07.391114 14025 leader_election.cc:341] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:15:07.391192 15424 ts_tablet_manager.cc:1398] T 70e8bfaa82464d3ab3338a779457dc02 P abb3df711d3a46809274b5cfbd4ce4e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:15:07.391440 15424 tablet_bootstrap.cc:492] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0: Bootstrap starting.
I20240503 08:15:07.392151 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79014e600505478883fc90159784ce6c" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:07.392342 13830 raft_consensus.cc:3037] T 79014e600505478883fc90159784ce6c P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:07.392881 15426 tablet_bootstrap.cc:492] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f: No bootstrap required, opened a new log
I20240503 08:15:07.392881 15424 tablet_bootstrap.cc:654] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:07.393424 15426 ts_tablet_manager.cc:1367] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.004s
I20240503 08:15:07.394431 15422 tablet_bootstrap.cc:492] T 79014e600505478883fc90159784ce6c P 5761f1165461459f9aaa8cd5d8e9b0b9: No bootstrap required, opened a new log
I20240503 08:15:07.394333 15426 raft_consensus.cc:348] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:07.394534 15426 raft_consensus.cc:374] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:07.394627 15422 ts_tablet_manager.cc:1367] T 79014e600505478883fc90159784ce6c P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.004s
I20240503 08:15:07.395399 13830 raft_consensus.cc:2454] T 79014e600505478883fc90159784ce6c P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 1.
I20240503 08:15:07.395349 15422 raft_consensus.cc:348] T 79014e600505478883fc90159784ce6c P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:07.395562 15422 raft_consensus.cc:374] T 79014e600505478883fc90159784ce6c P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:07.395557 15426 raft_consensus.cc:729] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Initialized, Role: FOLLOWER
I20240503 08:15:07.395630 15422 raft_consensus.cc:729] T 79014e600505478883fc90159784ce6c P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Initialized, Role: FOLLOWER
I20240503 08:15:07.395792 15426 consensus_queue.cc:260] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:07.395830 15422 consensus_queue.cc:260] T 79014e600505478883fc90159784ce6c P 5761f1165461459f9aaa8cd5d8e9b0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:07.396270 15426 ts_tablet_manager.cc:1398] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 08:15:07.396680 14024 leader_election.cc:302] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc, abb3df711d3a46809274b5cfbd4ce4e0; no voters: 5761f1165461459f9aaa8cd5d8e9b0b9
I20240503 08:15:07.396950 15417 raft_consensus.cc:2781] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:15:07.397339 15417 raft_consensus.cc:686] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Becoming Leader. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:07.397661 15417 consensus_queue.cc:237] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:07.399972 15422 ts_tablet_manager.cc:1398] T 79014e600505478883fc90159784ce6c P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 08:15:07.400969 15423 tablet_bootstrap.cc:492] T dd3d6353358d4b02af928cd9f9410aca P 5a198ad76e4f462c8be0859184aeb2fc: No bootstrap required, opened a new log
I20240503 08:15:07.401149 15423 ts_tablet_manager.cc:1367] T dd3d6353358d4b02af928cd9f9410aca P 5a198ad76e4f462c8be0859184aeb2fc: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.009s
I20240503 08:15:07.401405 15424 tablet_bootstrap.cc:492] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0: No bootstrap required, opened a new log
I20240503 08:15:07.401554 15424 ts_tablet_manager.cc:1367] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.003s
I20240503 08:15:07.401898 15423 raft_consensus.cc:348] T dd3d6353358d4b02af928cd9f9410aca P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:07.402129 15423 raft_consensus.cc:374] T dd3d6353358d4b02af928cd9f9410aca P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:07.402267 15423 raft_consensus.cc:729] T dd3d6353358d4b02af928cd9f9410aca P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Initialized, Role: FOLLOWER
I20240503 08:15:07.402258 15424 raft_consensus.cc:348] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:07.402451 15424 raft_consensus.cc:374] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:07.402536 15424 raft_consensus.cc:729] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Initialized, Role: FOLLOWER
I20240503 08:15:07.402263 13688 catalog_manager.cc:5484] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67). New cstate: current_term: 1 leader_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } }
I20240503 08:15:07.402508 15423 consensus_queue.cc:260] T dd3d6353358d4b02af928cd9f9410aca P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:07.402756 15424 consensus_queue.cc:260] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:07.403048 15423 ts_tablet_manager.cc:1398] T dd3d6353358d4b02af928cd9f9410aca P 5a198ad76e4f462c8be0859184aeb2fc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:15:07.403373 15424 ts_tablet_manager.cc:1398] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:15:07.441913 15344 raft_consensus.cc:482] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:15:07.442106 15344 raft_consensus.cc:504] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:07.442696 15344 leader_election.cc:288] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:07.443174 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd3d6353358d4b02af928cd9f9410aca" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc" is_pre_election: true
I20240503 08:15:07.443226 14089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd3d6353358d4b02af928cd9f9410aca" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" is_pre_election: true
I20240503 08:15:07.443418 14089 raft_consensus.cc:2454] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 0.
I20240503 08:15:07.443418 13830 raft_consensus.cc:2454] T dd3d6353358d4b02af928cd9f9410aca P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 0.
I20240503 08:15:07.443822 13894 leader_election.cc:302] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:07.444116 15344 raft_consensus.cc:2781] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:15:07.444211 15344 raft_consensus.cc:482] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:15:07.444314 15344 raft_consensus.cc:3037] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:07.446642 15344 raft_consensus.cc:504] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:07.447178 15344 leader_election.cc:288] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 1 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:07.447567 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd3d6353358d4b02af928cd9f9410aca" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:07.447678 14089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd3d6353358d4b02af928cd9f9410aca" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:07.447758 13830 raft_consensus.cc:3037] T dd3d6353358d4b02af928cd9f9410aca P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:07.447877 14089 raft_consensus.cc:3037] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:07.450425 13830 raft_consensus.cc:2454] T dd3d6353358d4b02af928cd9f9410aca P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 1.
I20240503 08:15:07.450467 14089 raft_consensus.cc:2454] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 1.
I20240503 08:15:07.450862 13896 leader_election.cc:302] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:07.451128 15344 raft_consensus.cc:2781] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:15:07.451481 15344 raft_consensus.cc:686] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:07.451766 15344 consensus_queue.cc:237] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:07.454403 13681 catalog_manager.cc:5484] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 1 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:07.454886 15417 raft_consensus.cc:482] T 70e8bfaa82464d3ab3338a779457dc02 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:15:07.455031 15417 raft_consensus.cc:504] T 70e8bfaa82464d3ab3338a779457dc02 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:07.455638 15417 leader_election.cc:288] T 70e8bfaa82464d3ab3338a779457dc02 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:07.455929 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70e8bfaa82464d3ab3338a779457dc02" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" is_pre_election: true
I20240503 08:15:07.456137 13960 raft_consensus.cc:2454] T 70e8bfaa82464d3ab3338a779457dc02 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 0.
I20240503 08:15:07.456262 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70e8bfaa82464d3ab3338a779457dc02" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" is_pre_election: true
I20240503 08:15:07.456465 14219 raft_consensus.cc:2454] T 70e8bfaa82464d3ab3338a779457dc02 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 0.
I20240503 08:15:07.456864 14025 leader_election.cc:302] T 70e8bfaa82464d3ab3338a779457dc02 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:07.457146 15417 raft_consensus.cc:2781] T 70e8bfaa82464d3ab3338a779457dc02 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:15:07.457242 15417 raft_consensus.cc:482] T 70e8bfaa82464d3ab3338a779457dc02 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:15:07.457343 15417 raft_consensus.cc:3037] T 70e8bfaa82464d3ab3338a779457dc02 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:07.459031 15417 raft_consensus.cc:504] T 70e8bfaa82464d3ab3338a779457dc02 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:07.459494 15417 leader_election.cc:288] T 70e8bfaa82464d3ab3338a779457dc02 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:07.459829 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70e8bfaa82464d3ab3338a779457dc02" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:07.459879 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70e8bfaa82464d3ab3338a779457dc02" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:07.460000 13960 raft_consensus.cc:3037] T 70e8bfaa82464d3ab3338a779457dc02 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:07.460135 14219 raft_consensus.cc:3037] T 70e8bfaa82464d3ab3338a779457dc02 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:07.461928 13960 raft_consensus.cc:2454] T 70e8bfaa82464d3ab3338a779457dc02 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 1.
I20240503 08:15:07.462303 14219 raft_consensus.cc:2454] T 70e8bfaa82464d3ab3338a779457dc02 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 1.
I20240503 08:15:07.462338 14023 leader_election.cc:302] T 70e8bfaa82464d3ab3338a779457dc02 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:07.462698 15417 raft_consensus.cc:2781] T 70e8bfaa82464d3ab3338a779457dc02 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:15:07.462852 15417 raft_consensus.cc:686] T 70e8bfaa82464d3ab3338a779457dc02 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Becoming Leader. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:07.463085 15417 consensus_queue.cc:237] T 70e8bfaa82464d3ab3338a779457dc02 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:07.465548 13681 catalog_manager.cc:5484] T 70e8bfaa82464d3ab3338a779457dc02 P abb3df711d3a46809274b5cfbd4ce4e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67). New cstate: current_term: 1 leader_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } }
I20240503 08:15:07.506479 14875 raft_consensus.cc:482] T 79014e600505478883fc90159784ce6c P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:15:07.506664 14875 raft_consensus.cc:504] T 79014e600505478883fc90159784ce6c P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:07.507396 14875 leader_election.cc:288] T 79014e600505478883fc90159784ce6c P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:07.508002 14089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79014e600505478883fc90159784ce6c" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" is_pre_election: true
I20240503 08:15:07.513065 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79014e600505478883fc90159784ce6c" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc" is_pre_election: true
I20240503 08:15:07.513370 13830 raft_consensus.cc:2379] T 79014e600505478883fc90159784ce6c P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5761f1165461459f9aaa8cd5d8e9b0b9 in current term 1: Already voted for candidate abb3df711d3a46809274b5cfbd4ce4e0 in this term.
I20240503 08:15:07.513897 14154 leader_election.cc:302] T 79014e600505478883fc90159784ce6c P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9; no voters: 5a198ad76e4f462c8be0859184aeb2fc, abb3df711d3a46809274b5cfbd4ce4e0
I20240503 08:15:07.514201 14875 raft_consensus.cc:3037] T 79014e600505478883fc90159784ce6c P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:07.516149 14875 raft_consensus.cc:2726] T 79014e600505478883fc90159784ce6c P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 08:15:07.794099 15440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:15:07.794423 15440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:15:07.802378 15440 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240503 08:15:07.858004 15416 consensus_queue.cc:1035] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:15:07.858850 15344 consensus_queue.cc:1035] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:15:07.870602 15437 consensus_queue.cc:1035] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:15:07.875797 15417 consensus_queue.cc:1035] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:15:07.883278 15417 consensus_queue.cc:1035] T 70e8bfaa82464d3ab3338a779457dc02 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:15:07.910619 15416 consensus_queue.cc:1035] T 70e8bfaa82464d3ab3338a779457dc02 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:15:08.025879 14023 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.230.66:44781, user_credentials={real_user=slave}} blocked reactor thread for 58669.4us
INSERT report
rows total: 2000
time total: 247.405 ms
time per row: 0.123703 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 08:15:09.054724 13689 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44232:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:15:09.063333 13808 tablet_service.cc:1467] Processing CreateTablet for tablet 8bb1779c7ac64d599cde0665505b3dba (DEFAULT_TABLE table=rebalance_test_table_8 [id=8c84df1667e844b78ac9f88a945c1f30]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:09.063787 13808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bb1779c7ac64d599cde0665505b3dba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:09.064522 14068 tablet_service.cc:1467] Processing CreateTablet for tablet 667abbd8fb6c49489127c12977d7a323 (DEFAULT_TABLE table=rebalance_test_table_8 [id=8c84df1667e844b78ac9f88a945c1f30]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:09.064837 14068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 667abbd8fb6c49489127c12977d7a323. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:09.068513 15513 tablet_bootstrap.cc:492] T 8bb1779c7ac64d599cde0665505b3dba P 5a198ad76e4f462c8be0859184aeb2fc: Bootstrap starting.
I20240503 08:15:09.068590 13810 tablet_service.cc:1467] Processing CreateTablet for tablet 51b32125655f4ef78dc49ba7174775da (DEFAULT_TABLE table=rebalance_test_table_8 [id=8c84df1667e844b78ac9f88a945c1f30]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:09.068938 13810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51b32125655f4ef78dc49ba7174775da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:09.069540 15514 tablet_bootstrap.cc:492] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0: Bootstrap starting.
I20240503 08:15:09.070019 14069 tablet_service.cc:1467] Processing CreateTablet for tablet 8bb1779c7ac64d599cde0665505b3dba (DEFAULT_TABLE table=rebalance_test_table_8 [id=8c84df1667e844b78ac9f88a945c1f30]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:09.070366 14069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bb1779c7ac64d599cde0665505b3dba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:09.070418 15513 tablet_bootstrap.cc:654] T 8bb1779c7ac64d599cde0665505b3dba P 5a198ad76e4f462c8be0859184aeb2fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:09.071506 14200 tablet_service.cc:1467] Processing CreateTablet for tablet 51b32125655f4ef78dc49ba7174775da (DEFAULT_TABLE table=rebalance_test_table_8 [id=8c84df1667e844b78ac9f88a945c1f30]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:09.071875 14200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51b32125655f4ef78dc49ba7174775da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:09.071915 15514 tablet_bootstrap.cc:654] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:09.074183 13940 tablet_service.cc:1467] Processing CreateTablet for tablet 8bb1779c7ac64d599cde0665505b3dba (DEFAULT_TABLE table=rebalance_test_table_8 [id=8c84df1667e844b78ac9f88a945c1f30]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:09.074594 13940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bb1779c7ac64d599cde0665505b3dba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:09.076488 15513 tablet_bootstrap.cc:492] T 8bb1779c7ac64d599cde0665505b3dba P 5a198ad76e4f462c8be0859184aeb2fc: No bootstrap required, opened a new log
I20240503 08:15:09.076491 13938 tablet_service.cc:1467] Processing CreateTablet for tablet 667abbd8fb6c49489127c12977d7a323 (DEFAULT_TABLE table=rebalance_test_table_8 [id=8c84df1667e844b78ac9f88a945c1f30]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:09.076660 15513 ts_tablet_manager.cc:1367] T 8bb1779c7ac64d599cde0665505b3dba P 5a198ad76e4f462c8be0859184aeb2fc: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240503 08:15:09.076849 13938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 667abbd8fb6c49489127c12977d7a323. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:09.077364 15513 raft_consensus.cc:348] T 8bb1779c7ac64d599cde0665505b3dba P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:09.077554 15513 raft_consensus.cc:374] T 8bb1779c7ac64d599cde0665505b3dba P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:09.077632 15513 raft_consensus.cc:729] T 8bb1779c7ac64d599cde0665505b3dba P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Initialized, Role: FOLLOWER
I20240503 08:15:09.077956 15513 consensus_queue.cc:260] T 8bb1779c7ac64d599cde0665505b3dba P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:09.078449 15513 ts_tablet_manager.cc:1398] T 8bb1779c7ac64d599cde0665505b3dba P 5a198ad76e4f462c8be0859184aeb2fc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:15:09.078704 15513 tablet_bootstrap.cc:492] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc: Bootstrap starting.
I20240503 08:15:09.079059 13939 tablet_service.cc:1467] Processing CreateTablet for tablet 51b32125655f4ef78dc49ba7174775da (DEFAULT_TABLE table=rebalance_test_table_8 [id=8c84df1667e844b78ac9f88a945c1f30]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:09.079399 13939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51b32125655f4ef78dc49ba7174775da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:09.080098 15513 tablet_bootstrap.cc:654] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:09.084378 15518 tablet_bootstrap.cc:492] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f: Bootstrap starting.
I20240503 08:15:09.085613 15513 tablet_bootstrap.cc:492] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc: No bootstrap required, opened a new log
I20240503 08:15:09.085786 15513 ts_tablet_manager.cc:1367] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20240503 08:15:09.086581 15513 raft_consensus.cc:348] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:09.086871 15513 raft_consensus.cc:374] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:09.086982 15513 raft_consensus.cc:729] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Initialized, Role: FOLLOWER
I20240503 08:15:09.087225 15513 consensus_queue.cc:260] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:09.087795 15513 ts_tablet_manager.cc:1398] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:15:09.087970 15518 tablet_bootstrap.cc:654] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:09.092883 15518 tablet_bootstrap.cc:492] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f: No bootstrap required, opened a new log
I20240503 08:15:09.093446 15518 ts_tablet_manager.cc:1367] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240503 08:15:09.093590 15520 tablet_bootstrap.cc:492] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9: Bootstrap starting.
I20240503 08:15:09.094064 15518 raft_consensus.cc:348] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:09.094218 15518 raft_consensus.cc:374] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:09.095242 15518 raft_consensus.cc:729] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Initialized, Role: FOLLOWER
I20240503 08:15:09.095561 15520 tablet_bootstrap.cc:654] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:09.095974 14198 tablet_service.cc:1467] Processing CreateTablet for tablet 667abbd8fb6c49489127c12977d7a323 (DEFAULT_TABLE table=rebalance_test_table_8 [id=8c84df1667e844b78ac9f88a945c1f30]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:09.096303 14198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 667abbd8fb6c49489127c12977d7a323. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:09.097393 15520 tablet_bootstrap.cc:492] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9: No bootstrap required, opened a new log
I20240503 08:15:09.097596 15520 ts_tablet_manager.cc:1367] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:15:09.098601 15520 raft_consensus.cc:348] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:09.098850 15520 raft_consensus.cc:374] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:09.099054 15520 raft_consensus.cc:729] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Initialized, Role: FOLLOWER
I20240503 08:15:09.099342 15520 consensus_queue.cc:260] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:09.099591 15518 consensus_queue.cc:260] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:09.100242 15514 tablet_bootstrap.cc:492] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0: No bootstrap required, opened a new log
I20240503 08:15:09.100428 15514 ts_tablet_manager.cc:1367] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0: Time spent bootstrapping tablet: real 0.031s user 0.011s sys 0.007s
I20240503 08:15:09.100891 15518 ts_tablet_manager.cc:1398] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240503 08:15:09.101172 15518 tablet_bootstrap.cc:492] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f: Bootstrap starting.
I20240503 08:15:09.102170 15514 raft_consensus.cc:348] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:09.102641 15514 raft_consensus.cc:374] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:09.102722 15518 tablet_bootstrap.cc:654] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:09.102782 15514 raft_consensus.cc:729] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Initialized, Role: FOLLOWER
I20240503 08:15:09.103078 15514 consensus_queue.cc:260] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:09.103245 15520 ts_tablet_manager.cc:1398] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20240503 08:15:09.103693 15514 ts_tablet_manager.cc:1398] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:15:09.104017 15514 tablet_bootstrap.cc:492] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0: Bootstrap starting.
I20240503 08:15:09.105347 15520 tablet_bootstrap.cc:492] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9: Bootstrap starting.
I20240503 08:15:09.105976 15514 tablet_bootstrap.cc:654] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:09.106801 15520 tablet_bootstrap.cc:654] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:09.113220 15518 tablet_bootstrap.cc:492] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f: No bootstrap required, opened a new log
I20240503 08:15:09.113301 15514 tablet_bootstrap.cc:492] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0: No bootstrap required, opened a new log
I20240503 08:15:09.113380 15518 ts_tablet_manager.cc:1367] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
I20240503 08:15:09.113430 15520 tablet_bootstrap.cc:492] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9: No bootstrap required, opened a new log
I20240503 08:15:09.113502 15514 ts_tablet_manager.cc:1367] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20240503 08:15:09.114213 15518 raft_consensus.cc:348] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:09.114225 15514 raft_consensus.cc:348] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:09.114429 15518 raft_consensus.cc:374] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:09.114437 15514 raft_consensus.cc:374] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:09.114504 15518 raft_consensus.cc:729] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Initialized, Role: FOLLOWER
I20240503 08:15:09.114513 15514 raft_consensus.cc:729] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Initialized, Role: FOLLOWER
I20240503 08:15:09.113595 15520 ts_tablet_manager.cc:1367] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20240503 08:15:09.114730 15514 consensus_queue.cc:260] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:09.114792 15518 consensus_queue.cc:260] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:09.115247 15514 ts_tablet_manager.cc:1398] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:15:09.115274 15518 ts_tablet_manager.cc:1398] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:15:09.115674 15518 tablet_bootstrap.cc:492] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f: Bootstrap starting.
I20240503 08:15:09.116425 14875 raft_consensus.cc:482] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:15:09.116580 14875 raft_consensus.cc:504] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:09.117229 14875 leader_election.cc:288] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:09.117758 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51b32125655f4ef78dc49ba7174775da" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc" is_pre_election: true
I20240503 08:15:09.118011 13830 raft_consensus.cc:2454] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5761f1165461459f9aaa8cd5d8e9b0b9 in term 0.
I20240503 08:15:09.118468 13958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51b32125655f4ef78dc49ba7174775da" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" is_pre_election: true
I20240503 08:15:09.119529 15520 raft_consensus.cc:348] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:09.119740 15520 raft_consensus.cc:374] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:09.119813 15520 raft_consensus.cc:729] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Initialized, Role: FOLLOWER
I20240503 08:15:09.120025 15520 consensus_queue.cc:260] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
W20240503 08:15:09.120455 14153 leader_election.cc:341] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:15:09.124557 15520 ts_tablet_manager.cc:1398] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240503 08:15:09.126034 14154 leader_election.cc:302] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, 5a198ad76e4f462c8be0859184aeb2fc; no voters: 296f3a209cf5447aaff2b8ea96a3bc4f
I20240503 08:15:09.126827 14875 raft_consensus.cc:2781] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:15:09.126912 14875 raft_consensus.cc:482] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:15:09.127009 14875 raft_consensus.cc:3037] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:09.129639 14875 raft_consensus.cc:504] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:09.130414 15518 tablet_bootstrap.cc:654] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:09.130810 13958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51b32125655f4ef78dc49ba7174775da" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
W20240503 08:15:09.131361 14153 leader_election.cc:341] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:15:09.131456 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51b32125655f4ef78dc49ba7174775da" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:09.131660 13830 raft_consensus.cc:3037] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:09.133925 13830 raft_consensus.cc:2454] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5761f1165461459f9aaa8cd5d8e9b0b9 in term 1.
I20240503 08:15:09.134312 14154 leader_election.cc:302] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, 5a198ad76e4f462c8be0859184aeb2fc; no voters: 296f3a209cf5447aaff2b8ea96a3bc4f
I20240503 08:15:09.134876 15518 tablet_bootstrap.cc:492] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f: No bootstrap required, opened a new log
I20240503 08:15:09.134933 15526 raft_consensus.cc:2781] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:15:09.135008 15518 ts_tablet_manager.cc:1367] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
I20240503 08:15:09.135064 14875 leader_election.cc:288] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 1 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:09.135352 15526 raft_consensus.cc:686] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 LEADER]: Becoming Leader. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Running, Role: LEADER
I20240503 08:15:09.135715 15518 raft_consensus.cc:348] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:09.135896 15518 raft_consensus.cc:374] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:09.135982 15518 raft_consensus.cc:729] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Initialized, Role: FOLLOWER
I20240503 08:15:09.135890 15526 consensus_queue.cc:237] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:09.136188 15518 consensus_queue.cc:260] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:09.136727 15518 ts_tablet_manager.cc:1398] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:15:09.141312 13687 catalog_manager.cc:5484] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68). New cstate: current_term: 1 leader_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } }
I20240503 08:15:09.231659 15529 raft_consensus.cc:482] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:15:09.231817 15529 raft_consensus.cc:504] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:09.232398 15529 leader_election.cc:288] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:15:09.232753 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb1779c7ac64d599cde0665505b3dba" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc" is_pre_election: true
I20240503 08:15:09.232810 13958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb1779c7ac64d599cde0665505b3dba" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" is_pre_election: true
I20240503 08:15:09.232998 13830 raft_consensus.cc:2454] T 8bb1779c7ac64d599cde0665505b3dba P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 0.
I20240503 08:15:09.233024 13958 raft_consensus.cc:2454] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 0.
I20240503 08:15:09.233354 14024 leader_election.cc:302] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:09.233598 15529 raft_consensus.cc:2781] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:15:09.233705 15529 raft_consensus.cc:482] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:15:09.233791 15529 raft_consensus.cc:3037] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:09.235663 15529 raft_consensus.cc:504] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:09.236110 15529 leader_election.cc:288] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:15:09.236523 13958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb1779c7ac64d599cde0665505b3dba" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:09.236485 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb1779c7ac64d599cde0665505b3dba" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:09.236687 13830 raft_consensus.cc:3037] T 8bb1779c7ac64d599cde0665505b3dba P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:09.236701 13958 raft_consensus.cc:3037] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:09.238588 13958 raft_consensus.cc:2454] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 1.
I20240503 08:15:09.238721 13830 raft_consensus.cc:2454] T 8bb1779c7ac64d599cde0665505b3dba P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 1.
I20240503 08:15:09.238936 14023 leader_election.cc:302] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:09.239187 15529 raft_consensus.cc:2781] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:15:09.239320 15529 raft_consensus.cc:686] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Becoming Leader. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:09.239588 15529 consensus_queue.cc:237] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:09.242269 13687 catalog_manager.cc:5484] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67). New cstate: current_term: 1 leader_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:09.281208 15526 raft_consensus.cc:482] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:15:09.281400 15526 raft_consensus.cc:504] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:09.281968 15526 leader_election.cc:288] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:15:09.282513 13958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "667abbd8fb6c49489127c12977d7a323" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" is_pre_election: true
I20240503 08:15:09.282503 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "667abbd8fb6c49489127c12977d7a323" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" is_pre_election: true
I20240503 08:15:09.282733 13958 raft_consensus.cc:2454] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5761f1165461459f9aaa8cd5d8e9b0b9 in term 0.
I20240503 08:15:09.282753 14090 raft_consensus.cc:2454] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5761f1165461459f9aaa8cd5d8e9b0b9 in term 0.
I20240503 08:15:09.283155 14153 leader_election.cc:302] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5761f1165461459f9aaa8cd5d8e9b0b9; no voters:
I20240503 08:15:09.283505 15526 raft_consensus.cc:2781] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:15:09.283586 15526 raft_consensus.cc:482] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:15:09.283675 15526 raft_consensus.cc:3037] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:09.285486 15526 raft_consensus.cc:504] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:09.285992 15526 leader_election.cc:288] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 1 election: Requested vote from peers abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:15:09.286412 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "667abbd8fb6c49489127c12977d7a323" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:09.286496 13958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "667abbd8fb6c49489127c12977d7a323" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:09.286626 14090 raft_consensus.cc:3037] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:09.286664 13958 raft_consensus.cc:3037] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:09.288664 13958 raft_consensus.cc:2454] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5761f1165461459f9aaa8cd5d8e9b0b9 in term 1.
I20240503 08:15:09.288736 14090 raft_consensus.cc:2454] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5761f1165461459f9aaa8cd5d8e9b0b9 in term 1.
I20240503 08:15:09.289011 14153 leader_election.cc:302] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5761f1165461459f9aaa8cd5d8e9b0b9; no voters:
I20240503 08:15:09.289249 15526 raft_consensus.cc:2781] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:15:09.289399 15526 raft_consensus.cc:686] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 LEADER]: Becoming Leader. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Running, Role: LEADER
I20240503 08:15:09.289654 15526 consensus_queue.cc:237] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:09.292809 13687 catalog_manager.cc:5484] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68). New cstate: current_term: 1 leader_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } }
I20240503 08:15:09.346168 15437 raft_consensus.cc:482] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:15:09.346340 15437 raft_consensus.cc:504] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:09.346952 15437 leader_election.cc:288] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:09.347391 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51b32125655f4ef78dc49ba7174775da" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc" is_pre_election: true
I20240503 08:15:09.347400 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51b32125655f4ef78dc49ba7174775da" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" is_pre_election: true
I20240503 08:15:09.347652 13830 raft_consensus.cc:2379] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 296f3a209cf5447aaff2b8ea96a3bc4f in current term 1: Already voted for candidate 5761f1165461459f9aaa8cd5d8e9b0b9 in this term.
I20240503 08:15:09.348111 13894 leader_election.cc:302] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f; no voters: 5761f1165461459f9aaa8cd5d8e9b0b9, 5a198ad76e4f462c8be0859184aeb2fc
I20240503 08:15:09.348420 15437 raft_consensus.cc:3037] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:09.350394 15437 raft_consensus.cc:2726] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:15:09.639290 15493 consensus_queue.cc:1035] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:15:09.639357 15532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:15:09.639685 15532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 08:15:09.643808 15493 consensus_queue.cc:1035] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:15:09.650959 15532 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240503 08:15:09.702483 15526 consensus_queue.cc:1035] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:15:09.723229 14875 consensus_queue.cc:1035] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:15:09.731491 14875 consensus_queue.cc:1035] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:15:09.760699 14875 consensus_queue.cc:1035] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 247.38 ms
time per row: 0.12369 ms
I20240503 08:15:10.046067 13687 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44232:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:15:10.054293 13808 tablet_service.cc:1467] Processing CreateTablet for tablet 045fa6c60d334659aab79f7ba4999661 (DEFAULT_TABLE table=rebalance_test_table_9 [id=b493cf9bcb0448099ac284f7fed461ba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:10.054491 14198 tablet_service.cc:1467] Processing CreateTablet for tablet 045fa6c60d334659aab79f7ba4999661 (DEFAULT_TABLE table=rebalance_test_table_9 [id=b493cf9bcb0448099ac284f7fed461ba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:10.054760 13808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 045fa6c60d334659aab79f7ba4999661. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:10.054718 14068 tablet_service.cc:1467] Processing CreateTablet for tablet 045fa6c60d334659aab79f7ba4999661 (DEFAULT_TABLE table=rebalance_test_table_9 [id=b493cf9bcb0448099ac284f7fed461ba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:10.054922 14198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 045fa6c60d334659aab79f7ba4999661. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:10.055074 14068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 045fa6c60d334659aab79f7ba4999661. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:10.058491 13939 tablet_service.cc:1467] Processing CreateTablet for tablet c244226a1c40488f949735420bfacd0b (DEFAULT_TABLE table=rebalance_test_table_9 [id=b493cf9bcb0448099ac284f7fed461ba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:10.058614 14200 tablet_service.cc:1467] Processing CreateTablet for tablet c7a958fc685e41fb86108dfcb9c6dc8b (DEFAULT_TABLE table=rebalance_test_table_9 [id=b493cf9bcb0448099ac284f7fed461ba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:10.058918 13939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c244226a1c40488f949735420bfacd0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:10.058918 14200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7a958fc685e41fb86108dfcb9c6dc8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:10.058818 13810 tablet_service.cc:1467] Processing CreateTablet for tablet c7a958fc685e41fb86108dfcb9c6dc8b (DEFAULT_TABLE table=rebalance_test_table_9 [id=b493cf9bcb0448099ac284f7fed461ba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:10.058840 13809 tablet_service.cc:1467] Processing CreateTablet for tablet c244226a1c40488f949735420bfacd0b (DEFAULT_TABLE table=rebalance_test_table_9 [id=b493cf9bcb0448099ac284f7fed461ba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:10.059213 13810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7a958fc685e41fb86108dfcb9c6dc8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:10.059381 13809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c244226a1c40488f949735420bfacd0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:10.060695 14069 tablet_service.cc:1467] Processing CreateTablet for tablet c7a958fc685e41fb86108dfcb9c6dc8b (DEFAULT_TABLE table=rebalance_test_table_9 [id=b493cf9bcb0448099ac284f7fed461ba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:10.061048 14069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7a958fc685e41fb86108dfcb9c6dc8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:10.066094 14070 tablet_service.cc:1467] Processing CreateTablet for tablet c244226a1c40488f949735420bfacd0b (DEFAULT_TABLE table=rebalance_test_table_9 [id=b493cf9bcb0448099ac284f7fed461ba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:10.066457 14070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c244226a1c40488f949735420bfacd0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:10.071365 15600 tablet_bootstrap.cc:492] T c7a958fc685e41fb86108dfcb9c6dc8b P 5a198ad76e4f462c8be0859184aeb2fc: Bootstrap starting.
I20240503 08:15:10.073930 15600 tablet_bootstrap.cc:654] T c7a958fc685e41fb86108dfcb9c6dc8b P 5a198ad76e4f462c8be0859184aeb2fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:10.079560 15599 tablet_bootstrap.cc:492] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f: Bootstrap starting.
I20240503 08:15:10.080404 15600 tablet_bootstrap.cc:492] T c7a958fc685e41fb86108dfcb9c6dc8b P 5a198ad76e4f462c8be0859184aeb2fc: No bootstrap required, opened a new log
I20240503 08:15:10.080549 15600 ts_tablet_manager.cc:1367] T c7a958fc685e41fb86108dfcb9c6dc8b P 5a198ad76e4f462c8be0859184aeb2fc: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20240503 08:15:10.081310 15599 tablet_bootstrap.cc:654] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:10.081270 15600 raft_consensus.cc:348] T c7a958fc685e41fb86108dfcb9c6dc8b P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:10.081516 15600 raft_consensus.cc:374] T c7a958fc685e41fb86108dfcb9c6dc8b P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:10.081614 15600 raft_consensus.cc:729] T c7a958fc685e41fb86108dfcb9c6dc8b P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Initialized, Role: FOLLOWER
I20240503 08:15:10.081871 15600 consensus_queue.cc:260] T c7a958fc685e41fb86108dfcb9c6dc8b P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:10.083391 15600 ts_tablet_manager.cc:1398] T c7a958fc685e41fb86108dfcb9c6dc8b P 5a198ad76e4f462c8be0859184aeb2fc: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:15:10.083747 15600 tablet_bootstrap.cc:492] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc: Bootstrap starting.
I20240503 08:15:10.086809 15600 tablet_bootstrap.cc:654] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:10.089231 15598 tablet_bootstrap.cc:492] T 045fa6c60d334659aab79f7ba4999661 P 5761f1165461459f9aaa8cd5d8e9b0b9: Bootstrap starting.
I20240503 08:15:10.089418 15599 tablet_bootstrap.cc:492] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f: No bootstrap required, opened a new log
I20240503 08:15:10.089548 15599 ts_tablet_manager.cc:1367] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20240503 08:15:10.090234 15599 raft_consensus.cc:348] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:10.090484 15600 tablet_bootstrap.cc:492] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc: No bootstrap required, opened a new log
I20240503 08:15:10.090466 15599 raft_consensus.cc:374] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:10.090562 15599 raft_consensus.cc:729] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Initialized, Role: FOLLOWER
I20240503 08:15:10.090622 15600 ts_tablet_manager.cc:1367] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.002s
I20240503 08:15:10.090801 15599 consensus_queue.cc:260] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:10.091253 15600 raft_consensus.cc:348] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:10.091471 15600 raft_consensus.cc:374] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:10.091552 15600 raft_consensus.cc:729] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Initialized, Role: FOLLOWER
I20240503 08:15:10.091802 15600 consensus_queue.cc:260] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:10.092129 15599 ts_tablet_manager.cc:1398] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:15:10.092908 15600 ts_tablet_manager.cc:1398] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:15:10.093173 15600 tablet_bootstrap.cc:492] T c244226a1c40488f949735420bfacd0b P 5a198ad76e4f462c8be0859184aeb2fc: Bootstrap starting.
I20240503 08:15:10.094758 15602 tablet_bootstrap.cc:492] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0: Bootstrap starting.
I20240503 08:15:10.095582 15600 tablet_bootstrap.cc:654] T c244226a1c40488f949735420bfacd0b P 5a198ad76e4f462c8be0859184aeb2fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:10.096349 15598 tablet_bootstrap.cc:654] T 045fa6c60d334659aab79f7ba4999661 P 5761f1165461459f9aaa8cd5d8e9b0b9: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:10.097848 15602 tablet_bootstrap.cc:654] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:10.100376 15602 tablet_bootstrap.cc:492] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0: No bootstrap required, opened a new log
I20240503 08:15:10.100503 15602 ts_tablet_manager.cc:1367] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:15:10.101194 15602 raft_consensus.cc:348] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:10.101382 15602 raft_consensus.cc:374] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:10.101478 15602 raft_consensus.cc:729] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Initialized, Role: FOLLOWER
I20240503 08:15:10.101703 15602 consensus_queue.cc:260] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:10.102188 15602 ts_tablet_manager.cc:1398] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:15:10.102432 15602 tablet_bootstrap.cc:492] T c7a958fc685e41fb86108dfcb9c6dc8b P abb3df711d3a46809274b5cfbd4ce4e0: Bootstrap starting.
I20240503 08:15:10.104897 15602 tablet_bootstrap.cc:654] T c7a958fc685e41fb86108dfcb9c6dc8b P abb3df711d3a46809274b5cfbd4ce4e0: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:10.106899 15598 tablet_bootstrap.cc:492] T 045fa6c60d334659aab79f7ba4999661 P 5761f1165461459f9aaa8cd5d8e9b0b9: No bootstrap required, opened a new log
I20240503 08:15:10.107023 15598 ts_tablet_manager.cc:1367] T 045fa6c60d334659aab79f7ba4999661 P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240503 08:15:10.107026 15600 tablet_bootstrap.cc:492] T c244226a1c40488f949735420bfacd0b P 5a198ad76e4f462c8be0859184aeb2fc: No bootstrap required, opened a new log
I20240503 08:15:10.107158 15600 ts_tablet_manager.cc:1367] T c244226a1c40488f949735420bfacd0b P 5a198ad76e4f462c8be0859184aeb2fc: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.011s
I20240503 08:15:10.107669 15598 raft_consensus.cc:348] T 045fa6c60d334659aab79f7ba4999661 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:10.107702 15600 raft_consensus.cc:348] T c244226a1c40488f949735420bfacd0b P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:10.107842 15598 raft_consensus.cc:374] T 045fa6c60d334659aab79f7ba4999661 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:10.107887 15600 raft_consensus.cc:374] T c244226a1c40488f949735420bfacd0b P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:10.107930 15598 raft_consensus.cc:729] T 045fa6c60d334659aab79f7ba4999661 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Initialized, Role: FOLLOWER
I20240503 08:15:10.107975 15600 raft_consensus.cc:729] T c244226a1c40488f949735420bfacd0b P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Initialized, Role: FOLLOWER
I20240503 08:15:10.108160 15598 consensus_queue.cc:260] T 045fa6c60d334659aab79f7ba4999661 P 5761f1165461459f9aaa8cd5d8e9b0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:10.108181 15600 consensus_queue.cc:260] T c244226a1c40488f949735420bfacd0b P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:10.108588 15600 ts_tablet_manager.cc:1398] T c244226a1c40488f949735420bfacd0b P 5a198ad76e4f462c8be0859184aeb2fc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:15:10.108757 15598 ts_tablet_manager.cc:1398] T 045fa6c60d334659aab79f7ba4999661 P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:15:10.109028 15598 tablet_bootstrap.cc:492] T c7a958fc685e41fb86108dfcb9c6dc8b P 5761f1165461459f9aaa8cd5d8e9b0b9: Bootstrap starting.
I20240503 08:15:10.109388 15602 tablet_bootstrap.cc:492] T c7a958fc685e41fb86108dfcb9c6dc8b P abb3df711d3a46809274b5cfbd4ce4e0: No bootstrap required, opened a new log
I20240503 08:15:10.109524 15602 ts_tablet_manager.cc:1367] T c7a958fc685e41fb86108dfcb9c6dc8b P abb3df711d3a46809274b5cfbd4ce4e0: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:15:10.110177 15602 raft_consensus.cc:348] T c7a958fc685e41fb86108dfcb9c6dc8b P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:10.110420 15602 raft_consensus.cc:374] T c7a958fc685e41fb86108dfcb9c6dc8b P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:10.110477 15598 tablet_bootstrap.cc:654] T c7a958fc685e41fb86108dfcb9c6dc8b P 5761f1165461459f9aaa8cd5d8e9b0b9: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:10.110515 15602 raft_consensus.cc:729] T c7a958fc685e41fb86108dfcb9c6dc8b P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Initialized, Role: FOLLOWER
I20240503 08:15:10.110762 15602 consensus_queue.cc:260] T c7a958fc685e41fb86108dfcb9c6dc8b P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:10.111261 15602 ts_tablet_manager.cc:1398] T c7a958fc685e41fb86108dfcb9c6dc8b P abb3df711d3a46809274b5cfbd4ce4e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:15:10.111589 15602 tablet_bootstrap.cc:492] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0: Bootstrap starting.
I20240503 08:15:10.112929 15602 tablet_bootstrap.cc:654] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:10.114773 15602 tablet_bootstrap.cc:492] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0: No bootstrap required, opened a new log
I20240503 08:15:10.114887 15602 ts_tablet_manager.cc:1367] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:15:10.115480 15602 raft_consensus.cc:348] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:10.115618 15602 raft_consensus.cc:374] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:10.115717 15602 raft_consensus.cc:729] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Initialized, Role: FOLLOWER
I20240503 08:15:10.115902 15602 consensus_queue.cc:260] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:10.116426 15602 ts_tablet_manager.cc:1398] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:15:10.120759 15598 tablet_bootstrap.cc:492] T c7a958fc685e41fb86108dfcb9c6dc8b P 5761f1165461459f9aaa8cd5d8e9b0b9: No bootstrap required, opened a new log
I20240503 08:15:10.120880 15598 ts_tablet_manager.cc:1367] T c7a958fc685e41fb86108dfcb9c6dc8b P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.010s
I20240503 08:15:10.121387 15598 raft_consensus.cc:348] T c7a958fc685e41fb86108dfcb9c6dc8b P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:10.121533 15598 raft_consensus.cc:374] T c7a958fc685e41fb86108dfcb9c6dc8b P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:10.121587 15598 raft_consensus.cc:729] T c7a958fc685e41fb86108dfcb9c6dc8b P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Initialized, Role: FOLLOWER
I20240503 08:15:10.121794 15598 consensus_queue.cc:260] T c7a958fc685e41fb86108dfcb9c6dc8b P 5761f1165461459f9aaa8cd5d8e9b0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:10.122222 15598 ts_tablet_manager.cc:1398] T c7a958fc685e41fb86108dfcb9c6dc8b P 5761f1165461459f9aaa8cd5d8e9b0b9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:15:10.240350 15593 raft_consensus.cc:482] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:15:10.240504 15593 raft_consensus.cc:504] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:10.241165 15593 leader_election.cc:288] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:10.241477 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "045fa6c60d334659aab79f7ba4999661" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" is_pre_election: true
I20240503 08:15:10.241680 14219 raft_consensus.cc:2454] T 045fa6c60d334659aab79f7ba4999661 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 0.
I20240503 08:15:10.241658 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "045fa6c60d334659aab79f7ba4999661" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" is_pre_election: true
I20240503 08:15:10.241914 14090 raft_consensus.cc:2454] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 0.
I20240503 08:15:10.242091 13765 leader_election.cc:302] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:10.242344 15593 raft_consensus.cc:2781] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:15:10.242444 15593 raft_consensus.cc:482] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:15:10.242506 15593 raft_consensus.cc:3037] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:10.244313 15593 raft_consensus.cc:504] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:10.244822 15593 leader_election.cc:288] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 1 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:10.245191 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "045fa6c60d334659aab79f7ba4999661" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:10.245229 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "045fa6c60d334659aab79f7ba4999661" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:10.245354 14219 raft_consensus.cc:3037] T 045fa6c60d334659aab79f7ba4999661 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:10.245365 14090 raft_consensus.cc:3037] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:10.246361 15529 raft_consensus.cc:482] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:15:10.246531 15529 raft_consensus.cc:504] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:10.247120 15529 leader_election.cc:288] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:10.247673 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c244226a1c40488f949735420bfacd0b" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc" is_pre_election: true
I20240503 08:15:10.247777 14090 raft_consensus.cc:2454] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 1.
I20240503 08:15:10.247735 13957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c244226a1c40488f949735420bfacd0b" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" is_pre_election: true
I20240503 08:15:10.247893 13830 raft_consensus.cc:2454] T c244226a1c40488f949735420bfacd0b P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 0.
I20240503 08:15:10.247939 13957 raft_consensus.cc:2454] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 0.
I20240503 08:15:10.248353 13766 leader_election.cc:302] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:10.248368 14024 leader_election.cc:302] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:10.248576 15593 raft_consensus.cc:2781] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:15:10.248709 15593 raft_consensus.cc:686] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:10.248905 15529 raft_consensus.cc:2781] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:15:10.248934 14219 raft_consensus.cc:2454] T 045fa6c60d334659aab79f7ba4999661 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 1.
I20240503 08:15:10.248996 15529 raft_consensus.cc:482] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:15:10.248960 15593 consensus_queue.cc:237] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:10.249083 15529 raft_consensus.cc:3037] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:10.251670 15529 raft_consensus.cc:504] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:10.251963 13689 catalog_manager.cc:5484] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 1 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:10.252702 15529 leader_election.cc:288] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:10.252676 13957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c244226a1c40488f949735420bfacd0b" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:10.252772 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c244226a1c40488f949735420bfacd0b" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:10.252843 13957 raft_consensus.cc:3037] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:10.252938 13830 raft_consensus.cc:3037] T c244226a1c40488f949735420bfacd0b P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:10.255508 13830 raft_consensus.cc:2454] T c244226a1c40488f949735420bfacd0b P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 1.
I20240503 08:15:10.255522 13957 raft_consensus.cc:2454] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 1.
I20240503 08:15:10.255882 14024 leader_election.cc:302] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:10.255882 15529 raft_consensus.cc:482] T c7a958fc685e41fb86108dfcb9c6dc8b P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:15:10.256183 15493 raft_consensus.cc:2781] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:15:10.256114 15529 raft_consensus.cc:504] T c7a958fc685e41fb86108dfcb9c6dc8b P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:10.256335 15493 raft_consensus.cc:686] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Becoming Leader. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:10.256613 15493 consensus_queue.cc:237] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:10.257021 15529 leader_election.cc:288] T c7a958fc685e41fb86108dfcb9c6dc8b P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:10.257139 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a958fc685e41fb86108dfcb9c6dc8b" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" is_pre_election: true
I20240503 08:15:10.257289 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a958fc685e41fb86108dfcb9c6dc8b" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc" is_pre_election: true
I20240503 08:15:10.257351 14219 raft_consensus.cc:2454] T c7a958fc685e41fb86108dfcb9c6dc8b P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 0.
I20240503 08:15:10.257452 13830 raft_consensus.cc:2454] T c7a958fc685e41fb86108dfcb9c6dc8b P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 0.
I20240503 08:15:10.257700 14025 leader_election.cc:302] T c7a958fc685e41fb86108dfcb9c6dc8b P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:10.257988 15529 raft_consensus.cc:2781] T c7a958fc685e41fb86108dfcb9c6dc8b P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:15:10.258096 15529 raft_consensus.cc:482] T c7a958fc685e41fb86108dfcb9c6dc8b P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:15:10.258224 15529 raft_consensus.cc:3037] T c7a958fc685e41fb86108dfcb9c6dc8b P abb3df711d3a46809274b5cfbd4ce4e0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:10.259951 13689 catalog_manager.cc:5484] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67). New cstate: current_term: 1 leader_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:10.260473 15529 raft_consensus.cc:504] T c7a958fc685e41fb86108dfcb9c6dc8b P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:10.262084 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a958fc685e41fb86108dfcb9c6dc8b" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:10.262251 14219 raft_consensus.cc:3037] T c7a958fc685e41fb86108dfcb9c6dc8b P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:10.262305 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a958fc685e41fb86108dfcb9c6dc8b" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:10.262485 13830 raft_consensus.cc:3037] T c7a958fc685e41fb86108dfcb9c6dc8b P 5a198ad76e4f462c8be0859184aeb2fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:10.263425 15529 leader_election.cc:288] T c7a958fc685e41fb86108dfcb9c6dc8b P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:10.264114 14219 raft_consensus.cc:2454] T c7a958fc685e41fb86108dfcb9c6dc8b P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 1.
I20240503 08:15:10.264461 14025 leader_election.cc:302] T c7a958fc685e41fb86108dfcb9c6dc8b P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:10.264587 13830 raft_consensus.cc:2454] T c7a958fc685e41fb86108dfcb9c6dc8b P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 1.
I20240503 08:15:10.264747 15529 raft_consensus.cc:2781] T c7a958fc685e41fb86108dfcb9c6dc8b P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:15:10.264907 15529 raft_consensus.cc:686] T c7a958fc685e41fb86108dfcb9c6dc8b P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Becoming Leader. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:10.265159 15529 consensus_queue.cc:237] T c7a958fc685e41fb86108dfcb9c6dc8b P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:10.267966 13689 catalog_manager.cc:5484] T c7a958fc685e41fb86108dfcb9c6dc8b P abb3df711d3a46809274b5cfbd4ce4e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67). New cstate: current_term: 1 leader_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
W20240503 08:15:10.636039 15614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:15:10.636335 15614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:15:10.643468 15614 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:15:10.652889 15493 consensus_queue.cc:1035] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:15:10.663472 15529 consensus_queue.cc:1035] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_9'
I20240503 08:15:10.758509 14219 raft_consensus.cc:1261] T c7a958fc685e41fb86108dfcb9c6dc8b P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Refusing update from remote peer abb3df711d3a46809274b5cfbd4ce4e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:15:10.758540 13830 raft_consensus.cc:1261] T c7a958fc685e41fb86108dfcb9c6dc8b P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Refusing update from remote peer abb3df711d3a46809274b5cfbd4ce4e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:15:10.759208 15493 consensus_queue.cc:1035] T c7a958fc685e41fb86108dfcb9c6dc8b P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:15:10.759744 15493 consensus_queue.cc:1035] T c7a958fc685e41fb86108dfcb9c6dc8b P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:15:10.790093 15593 consensus_queue.cc:1035] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20240503 08:15:10.813871 14090 raft_consensus.cc:1261] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Refusing update from remote peer 5a198ad76e4f462c8be0859184aeb2fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:15:10.814381 15593 consensus_queue.cc:1035] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 192.678 ms
time per row: 0.0963392 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 08:15:11.859345 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.69:33543
--local_ip_for_outbound_sockets=127.20.230.69
--tserver_master_addrs=127.20.230.126:38135
--webserver_port=40385
--webserver_interface=127.20.230.69
--builtin_ntp_servers=127.20.230.84:40715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 08:15:12.183418 15675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:15:12.183681 15675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:15:12.183794 15675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:15:12.191259 15675 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:15:12.191370 15675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:15:12.191543 15675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.69
I20240503 08:15:12.200183 15675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:40715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.69:33543
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.230.69
--webserver_port=40385
--tserver_master_addrs=127.20.230.126:38135
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.69
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:15:12.201365 15675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:15:12.203207 15675 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:15:12.215186 15681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:15:12.215875 15683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:15:12.216418 15675 server_base.cc:1034] running on GCE node
W20240503 08:15:12.217114 15680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:15:12.217815 15675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:15:12.218640 15675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:15:12.219832 15675 hybrid_clock.cc:648] HybridClock initialized: now 1714724112219780 us; error 47 us; skew 500 ppm
I20240503 08:15:12.220131 15675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:15:12.222276 15675 webserver.cc:460] Webserver started at http://127.20.230.69:40385/ using document root <none> and password file <none>
I20240503 08:15:12.222925 15675 fs_manager.cc:359] Metadata directory not provided
I20240503 08:15:12.223047 15675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:15:12.227742 15675 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20240503 08:15:12.230242 15689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:12.231573 15675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240503 08:15:12.231730 15675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "027334ce0b324689b9697ad372f38d8c"
format_stamp: "Formatted at 2024-05-03 08:14:52 on dist-test-slave-6vp8"
I20240503 08:15:12.232591 15675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:15:12.272055 15675 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:15:12.272843 15675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:15:12.273131 15675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:15:12.274091 15675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:15:12.275516 15675 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:15:12.275588 15675 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:12.275689 15675 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:15:12.275746 15675 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:12.313916 15675 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.69:33543
I20240503 08:15:12.313952 15801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.69:33543 every 8 connection(s)
I20240503 08:15:12.315665 15675 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:15:12.320459 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 15675
I20240503 08:15:12.320996 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.70:45427
--local_ip_for_outbound_sockets=127.20.230.70
--tserver_master_addrs=127.20.230.126:38135
--webserver_port=42161
--webserver_interface=127.20.230.70
--builtin_ntp_servers=127.20.230.84:40715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:15:12.328483 15802 heartbeater.cc:347] Connected to a master server at 127.20.230.126:38135
I20240503 08:15:12.328752 15802 heartbeater.cc:464] Registering TS with master...
I20240503 08:15:12.329457 15802 heartbeater.cc:511] Master 127.20.230.126:38135 requested a full tablet report, sending...
I20240503 08:15:12.330835 13689 ts_manager.cc:194] Re-registered known tserver with Master: 027334ce0b324689b9697ad372f38d8c (127.20.230.69:33543)
I20240503 08:15:12.331730 13689 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.69:52713
W20240503 08:15:12.640380 15806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:15:12.640661 15806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:15:12.640784 15806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:15:12.647843 15806 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:15:12.647969 15806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:15:12.648114 15806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.70
I20240503 08:15:12.656767 15806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:40715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.70:45427
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.20.230.70
--webserver_port=42161
--tserver_master_addrs=127.20.230.126:38135
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.70
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:15:12.658048 15806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:15:12.659855 15806 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:15:12.671255 15814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:15:12.672049 15811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:15:12.672768 15812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:15:12.673206 15806 server_base.cc:1034] running on GCE node
I20240503 08:15:12.673699 15806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:15:12.674548 15806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:15:12.675752 15806 hybrid_clock.cc:648] HybridClock initialized: now 1714724112675703 us; error 51 us; skew 500 ppm
I20240503 08:15:12.676010 15806 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:15:12.678145 15806 webserver.cc:460] Webserver started at http://127.20.230.70:42161/ using document root <none> and password file <none>
I20240503 08:15:12.678846 15806 fs_manager.cc:359] Metadata directory not provided
I20240503 08:15:12.678964 15806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:15:12.683946 15806 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 08:15:12.686848 15820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:12.688298 15806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:15:12.688457 15806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "fed845d7e14a4c07a62eeb5a383532ee"
format_stamp: "Formatted at 2024-05-03 08:14:52 on dist-test-slave-6vp8"
I20240503 08:15:12.689261 15806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:15:12.708048 15806 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:15:12.708842 15806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:15:12.709093 15806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:15:12.710068 15806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:15:12.711539 15806 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:15:12.711606 15806 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:12.711720 15806 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:15:12.711766 15806 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:12.740468 15806 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.70:45427
I20240503 08:15:12.740518 15932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.70:45427 every 8 connection(s)
I20240503 08:15:12.742156 15806 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 08:15:12.743394 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 15806
I20240503 08:15:12.744096 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.71:44051
--local_ip_for_outbound_sockets=127.20.230.71
--tserver_master_addrs=127.20.230.126:38135
--webserver_port=42117
--webserver_interface=127.20.230.71
--builtin_ntp_servers=127.20.230.84:40715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:15:12.755435 15933 heartbeater.cc:347] Connected to a master server at 127.20.230.126:38135
I20240503 08:15:12.755728 15933 heartbeater.cc:464] Registering TS with master...
I20240503 08:15:12.756933 15933 heartbeater.cc:511] Master 127.20.230.126:38135 requested a full tablet report, sending...
I20240503 08:15:12.758219 13689 ts_manager.cc:194] Re-registered known tserver with Master: fed845d7e14a4c07a62eeb5a383532ee (127.20.230.70:45427)
I20240503 08:15:12.758968 13689 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.70:35931
W20240503 08:15:13.077162 15936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:15:13.077451 15936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:15:13.077579 15936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:15:13.084863 15936 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:15:13.084995 15936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:15:13.085144 15936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.71
I20240503 08:15:13.094265 15936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:40715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.71:44051
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.20.230.71
--webserver_port=42117
--tserver_master_addrs=127.20.230.126:38135
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.71
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:15:13.095554 15936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:15:13.097214 15936 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:15:13.108734 15943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:15:13.109220 15944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:15:13.110026 15946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:15:13.110664 15936 server_base.cc:1034] running on GCE node
I20240503 08:15:13.111191 15936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:15:13.112187 15936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:15:13.113395 15936 hybrid_clock.cc:648] HybridClock initialized: now 1714724113113346 us; error 41 us; skew 500 ppm
I20240503 08:15:13.113704 15936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:15:13.115830 15936 webserver.cc:460] Webserver started at http://127.20.230.71:42117/ using document root <none> and password file <none>
I20240503 08:15:13.116631 15936 fs_manager.cc:359] Metadata directory not provided
I20240503 08:15:13.116782 15936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:15:13.122140 15936 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 08:15:13.124576 15952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:13.125741 15936 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:15:13.125926 15936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "2a4c6b598a44453bb25cf87f51b0a159"
format_stamp: "Formatted at 2024-05-03 08:14:52 on dist-test-slave-6vp8"
I20240503 08:15:13.126667 15936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:15:13.141034 15936 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:15:13.141723 15936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:15:13.142000 15936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:15:13.142835 15936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:15:13.144222 15936 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:15:13.144307 15936 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:13.144418 15936 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:15:13.144481 15936 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:13.175366 15936 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.71:44051
I20240503 08:15:13.175422 16064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.71:44051 every 8 connection(s)
I20240503 08:15:13.176716 15936 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 08:15:13.186827 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 15936
I20240503 08:15:13.187484 21401 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8bBFGA/build/asan/bin/kudu
/tmp/dist-test-task8bBFGA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.230.72:43911
--local_ip_for_outbound_sockets=127.20.230.72
--tserver_master_addrs=127.20.230.126:38135
--webserver_port=42803
--webserver_interface=127.20.230.72
--builtin_ntp_servers=127.20.230.84:40715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:15:13.188526 16065 heartbeater.cc:347] Connected to a master server at 127.20.230.126:38135
I20240503 08:15:13.188834 16065 heartbeater.cc:464] Registering TS with master...
I20240503 08:15:13.189633 16065 heartbeater.cc:511] Master 127.20.230.126:38135 requested a full tablet report, sending...
I20240503 08:15:13.190968 13689 ts_manager.cc:194] Re-registered known tserver with Master: 2a4c6b598a44453bb25cf87f51b0a159 (127.20.230.71:44051)
I20240503 08:15:13.191892 13689 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.71:34481
I20240503 08:15:13.334156 15802 heartbeater.cc:503] Master 127.20.230.126:38135 was elected leader, sending a full tablet report...
W20240503 08:15:13.520995 16068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:15:13.521256 16068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:15:13.521351 16068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:15:13.528587 16068 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:15:13.528693 16068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:15:13.528833 16068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.230.72
I20240503 08:15:13.537961 16068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.230.84:40715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.230.72:43911
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.20.230.72
--webserver_port=42803
--tserver_master_addrs=127.20.230.126:38135
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.230.72
--log_dir=/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:15:13.539299 16068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:15:13.540948 16068 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:15:13.552280 16074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:15:13.553022 16075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:15:13.553472 16077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:15:13.554033 16068 server_base.cc:1034] running on GCE node
I20240503 08:15:13.554656 16068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:15:13.555617 16068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:15:13.556838 16068 hybrid_clock.cc:648] HybridClock initialized: now 1714724113556795 us; error 37 us; skew 500 ppm
I20240503 08:15:13.557129 16068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:15:13.559199 16068 webserver.cc:460] Webserver started at http://127.20.230.72:42803/ using document root <none> and password file <none>
I20240503 08:15:13.559866 16068 fs_manager.cc:359] Metadata directory not provided
I20240503 08:15:13.559988 16068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:15:13.565128 16068 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 08:15:13.567796 16083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:13.569023 16068 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240503 08:15:13.569155 16068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "cab9267f70cb419dbc54ba686883abcb"
format_stamp: "Formatted at 2024-05-03 08:14:53 on dist-test-slave-6vp8"
I20240503 08:15:13.569927 16068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:15:13.598465 16068 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:15:13.599247 16068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:15:13.599519 16068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:15:13.600438 16068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:15:13.601918 16068 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:15:13.602008 16068 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:13.602129 16068 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:15:13.602190 16068 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:13.633905 16068 rpc_server.cc:307] RPC server started. Bound to: 127.20.230.72:43911
I20240503 08:15:13.633962 16195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.230.72:43911 every 8 connection(s)
I20240503 08:15:13.635391 16068 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8bBFGA/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714723866460737-21401-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 08:15:13.638211 21401 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8bBFGA/build/asan/bin/kudu as pid 16068
I20240503 08:15:13.651131 16196 heartbeater.cc:347] Connected to a master server at 127.20.230.126:38135
I20240503 08:15:13.651424 16196 heartbeater.cc:464] Registering TS with master...
I20240503 08:15:13.652237 16196 heartbeater.cc:511] Master 127.20.230.126:38135 requested a full tablet report, sending...
I20240503 08:15:13.653949 13689 ts_manager.cc:194] Re-registered known tserver with Master: cab9267f70cb419dbc54ba686883abcb (127.20.230.72:43911)
I20240503 08:15:13.654956 13689 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.230.72:49479
I20240503 08:15:13.702879 14090 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "045fa6c60d334659aab79f7ba4999661"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:13.703145 14090 raft_consensus.cc:482] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:13.703282 14090 raft_consensus.cc:3037] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.706050 14090 raft_consensus.cc:504] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:13.706938 14090 leader_election.cc:288] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 2 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:13.707485 13829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "045fa6c60d334659aab79f7ba4999661" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:13.707700 13829 raft_consensus.cc:3032] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:15:13.707803 13829 raft_consensus.cc:729] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:13.708094 13829 consensus_queue.cc:260] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:13.708665 13829 raft_consensus.cc:3037] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.710712 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "045fa6c60d334659aab79f7ba4999661" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:13.710922 14219 raft_consensus.cc:3037] T 045fa6c60d334659aab79f7ba4999661 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.711375 13829 raft_consensus.cc:2454] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 2.
I20240503 08:15:13.711858 14024 leader_election.cc:302] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:13.712162 15937 raft_consensus.cc:2781] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:15:13.713735 14219 raft_consensus.cc:2454] T 045fa6c60d334659aab79f7ba4999661 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 2.
I20240503 08:15:13.715138 14090 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dd3d6353358d4b02af928cd9f9410aca"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:13.715420 14090 raft_consensus.cc:482] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:13.715577 14090 raft_consensus.cc:3037] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.718302 14090 raft_consensus.cc:504] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:13.719439 13829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd3d6353358d4b02af928cd9f9410aca" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:13.719517 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd3d6353358d4b02af928cd9f9410aca" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:13.719702 13829 raft_consensus.cc:3037] T dd3d6353358d4b02af928cd9f9410aca P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.719709 13960 raft_consensus.cc:3032] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:15:13.719794 13960 raft_consensus.cc:729] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:13.720057 13960 consensus_queue.cc:260] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:13.720542 13960 raft_consensus.cc:3037] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.722270 13829 raft_consensus.cc:2454] T dd3d6353358d4b02af928cd9f9410aca P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 2.
I20240503 08:15:13.722659 14024 leader_election.cc:302] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:13.723059 13960 raft_consensus.cc:2454] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 2.
I20240503 08:15:13.723747 14090 leader_election.cc:288] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 2 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:15:13.724695 14090 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c7a958fc685e41fb86108dfcb9c6dc8b"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:13.724905 14090 raft_consensus.cc:465] T c7a958fc685e41fb86108dfcb9c6dc8b P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 08:15:13.725703 15937 raft_consensus.cc:686] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 LEADER]: Becoming Leader. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:13.725786 14089 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "70e8bfaa82464d3ab3338a779457dc02"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:13.725955 14089 raft_consensus.cc:465] T 70e8bfaa82464d3ab3338a779457dc02 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 08:15:13.725934 16209 raft_consensus.cc:2781] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:15:13.726292 16209 raft_consensus.cc:686] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 LEADER]: Becoming Leader. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:13.726488 15937 consensus_queue.cc:237] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:13.726742 14089 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f27cc9b78d084ac791de35b29e08a09a"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:13.726893 14089 raft_consensus.cc:465] T f27cc9b78d084ac791de35b29e08a09a P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 08:15:13.727051 16209 consensus_queue.cc:237] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:13.727754 14089 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c244226a1c40488f949735420bfacd0b"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:13.727885 14089 raft_consensus.cc:465] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 08:15:13.728709 14089 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "32e141d0103848c1a2d473429cffcca9"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:13.728842 14089 raft_consensus.cc:482] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:13.728957 14089 raft_consensus.cc:3037] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.731595 14089 raft_consensus.cc:504] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:13.732290 14089 leader_election.cc:288] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 2 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:15:13.732815 13957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32e141d0103848c1a2d473429cffcca9" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:13.733019 13957 raft_consensus.cc:3037] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.733525 13829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32e141d0103848c1a2d473429cffcca9" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:13.733726 13829 raft_consensus.cc:3032] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:15:13.733832 13829 raft_consensus.cc:729] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:13.734112 13829 consensus_queue.cc:260] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:13.734423 13829 raft_consensus.cc:3037] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.735543 13957 raft_consensus.cc:2454] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 2.
I20240503 08:15:13.736066 14023 leader_election.cc:302] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:13.736452 16210 raft_consensus.cc:2781] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:15:13.736871 16210 raft_consensus.cc:686] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 LEADER]: Becoming Leader. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:13.737241 13829 raft_consensus.cc:2454] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 2.
I20240503 08:15:13.737146 16210 consensus_queue.cc:237] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:13.738571 14090 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "de418f9b6bb04763b88ccc3bbe6cd68f"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:13.738775 14090 raft_consensus.cc:482] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:13.738938 14090 raft_consensus.cc:3037] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.742012 14090 raft_consensus.cc:504] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:13.742938 13829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de418f9b6bb04763b88ccc3bbe6cd68f" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:13.743131 13829 raft_consensus.cc:3032] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:15:13.743206 13829 raft_consensus.cc:729] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:13.743261 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de418f9b6bb04763b88ccc3bbe6cd68f" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:13.743477 14219 raft_consensus.cc:3037] T de418f9b6bb04763b88ccc3bbe6cd68f P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.743499 13829 consensus_queue.cc:260] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:13.743816 13829 raft_consensus.cc:3037] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.747233 13829 raft_consensus.cc:2454] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 2.
I20240503 08:15:13.748054 14024 leader_election.cc:302] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:13.748292 16210 raft_consensus.cc:2781] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:15:13.748477 16210 raft_consensus.cc:686] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 LEADER]: Becoming Leader. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:13.748746 16210 consensus_queue.cc:237] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:13.750012 14219 raft_consensus.cc:2454] T de418f9b6bb04763b88ccc3bbe6cd68f P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 2.
I20240503 08:15:13.752030 14090 leader_election.cc:288] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 2 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:13.753214 14090 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d69d33529e1140e5bc3a0a561012dfa1"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:13.753464 14090 raft_consensus.cc:482] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:13.753661 14090 raft_consensus.cc:3037] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.755602 13689 catalog_manager.cc:5484] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 reported cstate change: term changed from 1 to 2, leader changed from 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65) to abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67). New cstate: current_term: 2 leader_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:13.756409 14090 raft_consensus.cc:504] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:13.757089 14090 leader_election.cc:288] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 2 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:13.757712 13957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d69d33529e1140e5bc3a0a561012dfa1" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:13.757953 13957 raft_consensus.cc:3032] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:15:13.758003 14089 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d09e60e9c309425da9d27ae1153e4510"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:13.758051 13957 raft_consensus.cc:729] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:13.758162 14089 raft_consensus.cc:482] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:13.758242 14089 raft_consensus.cc:3037] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.758297 13957 consensus_queue.cc:260] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:13.758657 13957 raft_consensus.cc:3037] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.760670 14089 raft_consensus.cc:504] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:13.760876 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d69d33529e1140e5bc3a0a561012dfa1" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:13.761227 14089 leader_election.cc:288] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 2 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:15:13.761548 14218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d09e60e9c309425da9d27ae1153e4510" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:13.761708 14218 raft_consensus.cc:3032] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:15:13.762014 15933 heartbeater.cc:503] Master 127.20.230.126:38135 was elected leader, sending a full tablet report...
I20240503 08:15:13.762125 14218 raft_consensus.cc:729] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Running, Role: LEADER
I20240503 08:15:13.762189 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d09e60e9c309425da9d27ae1153e4510" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:13.762246 14089 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4b680592b1774f09ba79789afa55a0f1"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:13.762349 13960 raft_consensus.cc:3037] T d09e60e9c309425da9d27ae1153e4510 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.762393 14089 raft_consensus.cc:482] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:13.762495 14089 raft_consensus.cc:3037] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.762440 14218 consensus_queue.cc:260] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:13.762924 14218 raft_consensus.cc:3037] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.764854 13960 raft_consensus.cc:2454] T d09e60e9c309425da9d27ae1153e4510 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 2.
I20240503 08:15:13.764969 14089 raft_consensus.cc:504] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:13.765576 14089 leader_election.cc:288] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 2 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:13.766005 14217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b680592b1774f09ba79789afa55a0f1" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:13.766166 14217 raft_consensus.cc:3037] T 4b680592b1774f09ba79789afa55a0f1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.766785 13829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b680592b1774f09ba79789afa55a0f1" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:13.766878 14023 leader_election.cc:302] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:13.766989 13829 raft_consensus.cc:3032] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:15:13.767060 13829 raft_consensus.cc:729] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:13.767300 13829 consensus_queue.cc:260] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:13.767625 13829 raft_consensus.cc:3037] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.768483 14217 raft_consensus.cc:2454] T 4b680592b1774f09ba79789afa55a0f1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 2.
I20240503 08:15:13.768761 14088 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cdf9885a1e7c4b8dbb160e370ca0f60c"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:13.768900 14088 raft_consensus.cc:482] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:13.769014 14088 raft_consensus.cc:3037] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.769430 15937 raft_consensus.cc:2781] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:15:13.769603 15937 raft_consensus.cc:686] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 LEADER]: Becoming Leader. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:13.769903 15937 consensus_queue.cc:237] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:13.770272 13829 raft_consensus.cc:2454] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 2.
I20240503 08:15:13.770334 14025 leader_election.cc:302] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:13.771101 16213 raft_consensus.cc:2781] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:15:13.771271 16213 raft_consensus.cc:686] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 LEADER]: Becoming Leader. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:13.771641 16213 consensus_queue.cc:237] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:13.772648 14088 raft_consensus.cc:504] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:13.773646 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdf9885a1e7c4b8dbb160e370ca0f60c" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:13.773890 14088 leader_election.cc:288] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 2 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:13.773906 13960 raft_consensus.cc:3032] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:15:13.773983 13960 raft_consensus.cc:729] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:13.774196 14218 raft_consensus.cc:2454] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 2.
I20240503 08:15:13.774230 13960 consensus_queue.cc:260] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:13.774546 13960 raft_consensus.cc:3037] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.775069 13829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdf9885a1e7c4b8dbb160e370ca0f60c" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:13.775280 13829 raft_consensus.cc:3037] T cdf9885a1e7c4b8dbb160e370ca0f60c P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.776006 14219 raft_consensus.cc:3037] T d69d33529e1140e5bc3a0a561012dfa1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.777022 13960 raft_consensus.cc:2454] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 2.
I20240503 08:15:13.777488 14023 leader_election.cc:302] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:13.778107 13957 raft_consensus.cc:2454] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 2.
I20240503 08:15:13.778676 14023 leader_election.cc:302] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:13.780321 16210 raft_consensus.cc:2781] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:15:13.780541 13829 raft_consensus.cc:2454] T cdf9885a1e7c4b8dbb160e370ca0f60c P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 2.
I20240503 08:15:13.780570 15937 raft_consensus.cc:2781] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:15:13.780761 15937 raft_consensus.cc:686] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 LEADER]: Becoming Leader. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:13.781023 15937 consensus_queue.cc:237] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:13.781219 14088 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c310dd5906a24299b6f62201b30b86db"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:13.781446 14088 raft_consensus.cc:465] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 08:15:13.783546 14219 raft_consensus.cc:2454] T d69d33529e1140e5bc3a0a561012dfa1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 2.
I20240503 08:15:13.783998 14088 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8bb1779c7ac64d599cde0665505b3dba"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:13.784209 14088 raft_consensus.cc:465] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 08:15:13.785076 14088 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "79014e600505478883fc90159784ce6c"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:13.785256 14088 raft_consensus.cc:465] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 08:15:13.786036 16210 raft_consensus.cc:686] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 LEADER]: Becoming Leader. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:13.786059 14088 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d2ba977460694815a38d415a14e1cd5f"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:13.786314 14088 raft_consensus.cc:465] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 08:15:13.786506 16210 consensus_queue.cc:237] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:13.787129 14088 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "150f0d2d17bc4a59a8d82982790b4594"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:13.787333 14088 raft_consensus.cc:465] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 08:15:13.788230 14088 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "06e7243793e9467daeed79f3b1f5be5a"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:13.788445 14088 raft_consensus.cc:482] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:13.788543 14088 raft_consensus.cc:3037] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.791381 14088 raft_consensus.cc:504] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:13.791880 14088 leader_election.cc:288] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 2 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:13.792389 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e7243793e9467daeed79f3b1f5be5a" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:13.792578 14219 raft_consensus.cc:3037] T 06e7243793e9467daeed79f3b1f5be5a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.792646 13829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e7243793e9467daeed79f3b1f5be5a" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:13.792825 13829 raft_consensus.cc:3032] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:15:13.792912 13829 raft_consensus.cc:729] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:13.793176 13829 consensus_queue.cc:260] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:13.793540 13829 raft_consensus.cc:3037] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.794751 14219 raft_consensus.cc:2454] T 06e7243793e9467daeed79f3b1f5be5a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 2.
I20240503 08:15:13.795238 14025 leader_election.cc:302] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:13.795500 14088 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "666be8071a904e6cbff984fa293d00b5"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:13.795655 14088 raft_consensus.cc:465] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 08:15:13.796347 13829 raft_consensus.cc:2454] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 2.
I20240503 08:15:13.796372 14088 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9cae16af5a034cc29b8bb410a758c332"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:13.796514 14088 raft_consensus.cc:465] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 08:15:13.797247 15937 raft_consensus.cc:2781] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:15:13.797289 14088 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "667abbd8fb6c49489127c12977d7a323"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:13.797442 15937 raft_consensus.cc:686] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 LEADER]: Becoming Leader. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:13.797487 14088 raft_consensus.cc:482] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:13.797643 14088 raft_consensus.cc:3037] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.797731 15937 consensus_queue.cc:237] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:13.800565 14088 raft_consensus.cc:504] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:13.801326 14088 leader_election.cc:288] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 2 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:13.801621 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "667abbd8fb6c49489127c12977d7a323" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:13.801820 14219 raft_consensus.cc:3032] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:15:13.801899 14219 raft_consensus.cc:729] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Running, Role: LEADER
I20240503 08:15:13.802101 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "667abbd8fb6c49489127c12977d7a323" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:13.802174 14219 consensus_queue.cc:260] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:13.802310 13960 raft_consensus.cc:3037] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.802515 14219 raft_consensus.cc:3037] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.803711 13689 catalog_manager.cc:5484] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0 reported cstate change: term changed from 1 to 2, leader changed from 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65) to abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67). New cstate: current_term: 2 leader_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } }
I20240503 08:15:13.804214 13689 catalog_manager.cc:5484] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0 reported cstate change: term changed from 1 to 2, leader changed from 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66) to abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67). New cstate: current_term: 2 leader_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } }
I20240503 08:15:13.804638 13689 catalog_manager.cc:5484] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0 reported cstate change: term changed from 1 to 2, leader changed from 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66) to abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67). New cstate: current_term: 2 leader_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } }
I20240503 08:15:13.804813 13960 raft_consensus.cc:2454] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 2.
I20240503 08:15:13.805104 14219 raft_consensus.cc:2454] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 2.
I20240503 08:15:13.805024 13689 catalog_manager.cc:5484] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0 reported cstate change: term changed from 1 to 2, leader changed from 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66) to abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67). New cstate: current_term: 2 leader_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:13.805400 13689 catalog_manager.cc:5484] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0 reported cstate change: term changed from 1 to 2, leader changed from 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65) to abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67). New cstate: current_term: 2 leader_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } }
I20240503 08:15:13.805601 14025 leader_election.cc:302] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:13.806049 13689 catalog_manager.cc:5484] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0 reported cstate change: term changed from 1 to 2, leader changed from 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68) to abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67). New cstate: current_term: 2 leader_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:13.806383 16210 raft_consensus.cc:2781] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:15:13.806457 13689 catalog_manager.cc:5484] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0 reported cstate change: term changed from 1 to 2, leader changed from 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65) to abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67). New cstate: current_term: 2 leader_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:13.807385 14090 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "94cfd3b764b148bb8b42e5a5cb7883b3"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:13.807446 16210 raft_consensus.cc:686] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 LEADER]: Becoming Leader. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:13.807554 14090 raft_consensus.cc:482] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:13.807674 14090 raft_consensus.cc:3037] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.807802 16210 consensus_queue.cc:237] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:13.810497 14090 raft_consensus.cc:504] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:13.811110 14090 leader_election.cc:288] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 2 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:15:13.811717 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94cfd3b764b148bb8b42e5a5cb7883b3" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:13.811914 13830 raft_consensus.cc:3032] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:15:13.812013 13830 raft_consensus.cc:729] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:13.812247 13830 consensus_queue.cc:260] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:13.812593 13830 raft_consensus.cc:3037] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.815272 13957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94cfd3b764b148bb8b42e5a5cb7883b3" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:13.815480 13957 raft_consensus.cc:3037] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.816586 13830 raft_consensus.cc:2454] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 2.
I20240503 08:15:13.817126 14024 leader_election.cc:302] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:13.817396 16222 raft_consensus.cc:2781] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:15:13.817528 16222 raft_consensus.cc:686] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 LEADER]: Becoming Leader. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:13.818006 16222 consensus_queue.cc:237] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:13.818140 13957 raft_consensus.cc:2454] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 2.
I20240503 08:15:13.819188 14090 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e82cb19fece24181b7607d9447ba1eb2"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:13.819386 14090 raft_consensus.cc:482] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:13.819481 14090 raft_consensus.cc:3037] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.822434 14090 raft_consensus.cc:504] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:13.823467 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82cb19fece24181b7607d9447ba1eb2" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:13.823632 13830 raft_consensus.cc:3032] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:15:13.823704 13830 raft_consensus.cc:729] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:13.823896 13830 consensus_queue.cc:260] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:13.824162 13830 raft_consensus.cc:3037] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.825889 13830 raft_consensus.cc:2454] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 2.
I20240503 08:15:13.826468 14024 leader_election.cc:302] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:13.826941 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82cb19fece24181b7607d9447ba1eb2" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:13.827061 16222 raft_consensus.cc:2781] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:15:13.827162 14219 raft_consensus.cc:3037] T e82cb19fece24181b7607d9447ba1eb2 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.827208 16222 raft_consensus.cc:686] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 LEADER]: Becoming Leader. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:13.827438 16222 consensus_queue.cc:237] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:13.827912 14090 leader_election.cc:288] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 2 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:13.829663 14219 raft_consensus.cc:2454] T e82cb19fece24181b7607d9447ba1eb2 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 2.
I20240503 08:15:13.832178 13689 catalog_manager.cc:5484] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 reported cstate change: term changed from 1 to 2, leader changed from 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65) to abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67). New cstate: current_term: 2 leader_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:13.832638 13689 catalog_manager.cc:5484] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0 reported cstate change: term changed from 1 to 2, leader changed from 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65) to abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67). New cstate: current_term: 2 leader_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:13.832974 13689 catalog_manager.cc:5484] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0 reported cstate change: term changed from 1 to 2, leader changed from 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68) to abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67). New cstate: current_term: 2 leader_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } }
I20240503 08:15:13.839170 13689 catalog_manager.cc:5484] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0 reported cstate change: term changed from 1 to 2, leader changed from 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65) to abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67). New cstate: current_term: 2 leader_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:13.849457 13957 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "51b32125655f4ef78dc49ba7174775da"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:13.849695 13957 raft_consensus.cc:482] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:13.849815 13957 raft_consensus.cc:3037] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.852176 13957 raft_consensus.cc:504] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:13.853703 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51b32125655f4ef78dc49ba7174775da" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:13.853914 13830 raft_consensus.cc:3037] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.856680 13957 leader_election.cc:288] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 2 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:13.857088 13830 raft_consensus.cc:2454] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 2.
I20240503 08:15:13.857519 13894 leader_election.cc:302] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:13.861429 16203 raft_consensus.cc:2781] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:15:13.861585 16203 raft_consensus.cc:686] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:13.863000 16203 consensus_queue.cc:237] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:13.864102 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51b32125655f4ef78dc49ba7174775da" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:13.864306 14219 raft_consensus.cc:3032] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:15:13.864403 14219 raft_consensus.cc:729] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Running, Role: LEADER
I20240503 08:15:13.864450 13957 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8bb1779c7ac64d599cde0665505b3dba"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:13.864593 13957 raft_consensus.cc:482] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:13.864710 13957 raft_consensus.cc:3037] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.864670 14219 consensus_queue.cc:260] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:13.865022 14219 raft_consensus.cc:3037] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.867278 13957 raft_consensus.cc:504] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:13.867584 14219 raft_consensus.cc:2454] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 2.
I20240503 08:15:13.869527 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb1779c7ac64d599cde0665505b3dba" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:13.869743 13830 raft_consensus.cc:3037] T 8bb1779c7ac64d599cde0665505b3dba P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.872591 13830 raft_consensus.cc:2454] T 8bb1779c7ac64d599cde0665505b3dba P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 2.
I20240503 08:15:13.873054 13894 leader_election.cc:302] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:13.873368 15670 raft_consensus.cc:2781] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:15:13.874037 13957 leader_election.cc:288] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 2 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:13.874536 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb1779c7ac64d599cde0665505b3dba" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:13.874718 14090 raft_consensus.cc:3032] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:15:13.874830 14090 raft_consensus.cc:729] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:13.875072 14090 consensus_queue.cc:260] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:13.875494 14090 raft_consensus.cc:3037] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.877420 14090 raft_consensus.cc:2454] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 2.
I20240503 08:15:13.879771 13957 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dd3d6353358d4b02af928cd9f9410aca"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:13.879967 13957 raft_consensus.cc:482] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:13.880061 13957 raft_consensus.cc:3037] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:13.879885 13689 catalog_manager.cc:5484] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 1 to 2, leader changed from 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68) to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 2 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } }
I20240503 08:15:13.882540 13957 raft_consensus.cc:504] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:13.883634 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd3d6353358d4b02af928cd9f9410aca" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:13.883845 13830 raft_consensus.cc:3037] T dd3d6353358d4b02af928cd9f9410aca P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:13.883862 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd3d6353358d4b02af928cd9f9410aca" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:13.884054 14090 raft_consensus.cc:3032] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:15:13.884150 14090 raft_consensus.cc:729] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:13.884406 14090 consensus_queue.cc:260] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:13.884693 14090 raft_consensus.cc:3037] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:13.886602 13830 raft_consensus.cc:2454] T dd3d6353358d4b02af928cd9f9410aca P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 3.
I20240503 08:15:13.886919 15670 raft_consensus.cc:686] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:13.887218 15670 consensus_queue.cc:237] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:13.888027 13957 leader_election.cc:288] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 3 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:13.889122 13960 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "70e8bfaa82464d3ab3338a779457dc02"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:13.889333 13960 raft_consensus.cc:482] T 70e8bfaa82464d3ab3338a779457dc02 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:13.889456 13960 raft_consensus.cc:3037] T 70e8bfaa82464d3ab3338a779457dc02 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.891064 13689 catalog_manager.cc:5484] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 1 to 2, leader changed from abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67) to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 2 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:13.892012 13960 raft_consensus.cc:504] T 70e8bfaa82464d3ab3338a779457dc02 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:13.892596 13894 leader_election.cc:302] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:13.893119 15670 raft_consensus.cc:2781] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:15:13.893151 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70e8bfaa82464d3ab3338a779457dc02" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:13.893268 15670 raft_consensus.cc:686] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:13.893348 14219 raft_consensus.cc:3037] T 70e8bfaa82464d3ab3338a779457dc02 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.893496 15670 consensus_queue.cc:237] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:13.894867 14088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70e8bfaa82464d3ab3338a779457dc02" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:13.895073 14088 raft_consensus.cc:3032] T 70e8bfaa82464d3ab3338a779457dc02 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:15:13.895144 14088 raft_consensus.cc:729] T 70e8bfaa82464d3ab3338a779457dc02 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:13.895388 14088 consensus_queue.cc:260] T 70e8bfaa82464d3ab3338a779457dc02 P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:13.895735 14088 raft_consensus.cc:3037] T 70e8bfaa82464d3ab3338a779457dc02 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.895972 14219 raft_consensus.cc:2454] T 70e8bfaa82464d3ab3338a779457dc02 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 2.
I20240503 08:15:13.896068 14090 raft_consensus.cc:2396] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 296f3a209cf5447aaff2b8ea96a3bc4f for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:13.896392 13895 leader_election.cc:302] T 70e8bfaa82464d3ab3338a779457dc02 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5761f1165461459f9aaa8cd5d8e9b0b9; no voters:
I20240503 08:15:13.896720 16203 raft_consensus.cc:2781] T 70e8bfaa82464d3ab3338a779457dc02 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:15:13.896816 13960 leader_election.cc:288] T 70e8bfaa82464d3ab3338a779457dc02 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 2 election: Requested vote from peers abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:13.896871 16203 raft_consensus.cc:686] T 70e8bfaa82464d3ab3338a779457dc02 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:13.897106 16203 consensus_queue.cc:237] T 70e8bfaa82464d3ab3338a779457dc02 P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:13.897826 13960 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a478c28e5d4646fdb956f5b65e5c6c94"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:13.898092 13960 raft_consensus.cc:482] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:13.898481 13960 raft_consensus.cc:3037] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.898905 14088 raft_consensus.cc:2454] T 70e8bfaa82464d3ab3338a779457dc02 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 2.
I20240503 08:15:13.906955 13960 raft_consensus.cc:504] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:13.908890 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a478c28e5d4646fdb956f5b65e5c6c94" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:13.909127 13830 raft_consensus.cc:3032] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:15:13.909202 13830 raft_consensus.cc:729] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:13.909467 13830 consensus_queue.cc:260] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:13.909807 13830 raft_consensus.cc:3037] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.913256 13830 raft_consensus.cc:2454] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 2.
I20240503 08:15:13.917997 13957 raft_consensus.cc:1226] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Rejecting Update request from peer 5a198ad76e4f462c8be0859184aeb2fc for earlier term 1. Current term is 2. Ops: []
I20240503 08:15:13.918674 13894 leader_election.cc:302] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:13.919745 16203 raft_consensus.cc:2781] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:15:13.919919 13960 leader_election.cc:288] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 2 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:13.920001 16203 raft_consensus.cc:686] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:13.920347 16203 consensus_queue.cc:237] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:13.920933 13957 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "769c4490062b4206951af2f4b9a6da05"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:13.921166 13957 raft_consensus.cc:465] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 08:15:13.922356 13957 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d09e60e9c309425da9d27ae1153e4510"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:13.922556 13957 raft_consensus.cc:482] T d09e60e9c309425da9d27ae1153e4510 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:13.922657 13957 raft_consensus.cc:3037] T d09e60e9c309425da9d27ae1153e4510 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:13.925650 13957 raft_consensus.cc:504] T d09e60e9c309425da9d27ae1153e4510 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:13.926451 13957 leader_election.cc:288] T d09e60e9c309425da9d27ae1153e4510 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 3 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:13.927168 14088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d09e60e9c309425da9d27ae1153e4510" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:13.927404 14088 raft_consensus.cc:3032] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:15:13.927479 14088 raft_consensus.cc:729] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:13.927752 14088 consensus_queue.cc:260] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:13.928069 14088 raft_consensus.cc:3037] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:13.931182 14088 raft_consensus.cc:2396] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 296f3a209cf5447aaff2b8ea96a3bc4f for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:13.932806 13957 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "84204cbcda32446ab708717e9633af69"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:13.933068 13957 raft_consensus.cc:465] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 08:15:13.934013 13960 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f27cc9b78d084ac791de35b29e08a09a"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:13.934186 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a478c28e5d4646fdb956f5b65e5c6c94" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:13.934260 13960 raft_consensus.cc:482] T f27cc9b78d084ac791de35b29e08a09a P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:13.934410 14219 raft_consensus.cc:3037] T a478c28e5d4646fdb956f5b65e5c6c94 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.934525 13960 raft_consensus.cc:3037] T f27cc9b78d084ac791de35b29e08a09a P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.937510 14219 raft_consensus.cc:2454] T a478c28e5d4646fdb956f5b65e5c6c94 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 2.
I20240503 08:15:13.937402 13960 raft_consensus.cc:504] T f27cc9b78d084ac791de35b29e08a09a P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:13.938488 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d09e60e9c309425da9d27ae1153e4510" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:13.938715 14219 raft_consensus.cc:3037] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:13.940222 14088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f27cc9b78d084ac791de35b29e08a09a" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:13.940439 14088 raft_consensus.cc:3032] T f27cc9b78d084ac791de35b29e08a09a P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:15:13.940526 14088 raft_consensus.cc:729] T f27cc9b78d084ac791de35b29e08a09a P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:13.940830 14088 consensus_queue.cc:260] T f27cc9b78d084ac791de35b29e08a09a P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:13.941253 14088 raft_consensus.cc:3037] T f27cc9b78d084ac791de35b29e08a09a P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.941514 14219 raft_consensus.cc:2454] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 3.
I20240503 08:15:13.942390 13895 leader_election.cc:302] T d09e60e9c309425da9d27ae1153e4510 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5761f1165461459f9aaa8cd5d8e9b0b9; no voters: abb3df711d3a46809274b5cfbd4ce4e0
I20240503 08:15:13.942646 16203 raft_consensus.cc:2781] T d09e60e9c309425da9d27ae1153e4510 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:15:13.942800 16203 raft_consensus.cc:686] T d09e60e9c309425da9d27ae1153e4510 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:13.943063 16203 consensus_queue.cc:237] T d09e60e9c309425da9d27ae1153e4510 P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:13.944530 14088 raft_consensus.cc:2454] T f27cc9b78d084ac791de35b29e08a09a P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 2.
I20240503 08:15:13.945050 13896 leader_election.cc:302] T f27cc9b78d084ac791de35b29e08a09a P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:13.945400 16232 raft_consensus.cc:2781] T f27cc9b78d084ac791de35b29e08a09a P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:15:13.945518 16232 raft_consensus.cc:686] T f27cc9b78d084ac791de35b29e08a09a P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:13.945820 16232 consensus_queue.cc:237] T f27cc9b78d084ac791de35b29e08a09a P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:13.952021 13689 catalog_manager.cc:5484] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 2 to 3, leader changed from abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67) to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 3 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:13.952824 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f27cc9b78d084ac791de35b29e08a09a" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:13.953073 14219 raft_consensus.cc:3037] T f27cc9b78d084ac791de35b29e08a09a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.955964 14219 raft_consensus.cc:2454] T f27cc9b78d084ac791de35b29e08a09a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 2.
I20240503 08:15:13.952073 13960 leader_election.cc:288] T f27cc9b78d084ac791de35b29e08a09a P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 2 election: Requested vote from peers abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:13.957787 13957 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c244226a1c40488f949735420bfacd0b"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:13.957955 13957 raft_consensus.cc:482] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:13.958065 13957 raft_consensus.cc:3037] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.961017 13957 raft_consensus.cc:504] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:13.962038 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c244226a1c40488f949735420bfacd0b" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:13.962246 13830 raft_consensus.cc:3037] T c244226a1c40488f949735420bfacd0b P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.964385 13830 raft_consensus.cc:2454] T c244226a1c40488f949735420bfacd0b P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 2.
I20240503 08:15:13.964730 13894 leader_election.cc:302] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:13.964610 13689 catalog_manager.cc:5484] T 70e8bfaa82464d3ab3338a779457dc02 P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 1 to 2, leader changed from abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67) to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 2 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } }
I20240503 08:15:13.965024 16203 raft_consensus.cc:2781] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:15:13.965162 16203 raft_consensus.cc:686] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:13.965126 13689 catalog_manager.cc:5484] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 1 to 2, leader changed from 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65) to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 2 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } }
I20240503 08:15:13.965417 16203 consensus_queue.cc:237] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:13.965533 13689 catalog_manager.cc:5484] T f27cc9b78d084ac791de35b29e08a09a P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 1 to 2, leader changed from abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67) to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 2 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } }
I20240503 08:15:13.966015 13689 catalog_manager.cc:5484] T d09e60e9c309425da9d27ae1153e4510 P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 2 to 3, leader changed from abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67) to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 3 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:13.967202 13957 leader_election.cc:288] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 2 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:13.969137 14088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c244226a1c40488f949735420bfacd0b" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:13.969393 14088 raft_consensus.cc:3032] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:15:13.969471 14088 raft_consensus.cc:729] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:13.969758 14088 consensus_queue.cc:260] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:13.970142 14088 raft_consensus.cc:3037] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.973062 14088 raft_consensus.cc:2454] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 2.
I20240503 08:15:13.977910 13957 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "32e141d0103848c1a2d473429cffcca9"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:13.978118 13957 raft_consensus.cc:482] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:13.978236 13957 raft_consensus.cc:3037] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:13.980767 13957 raft_consensus.cc:504] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:13.982211 13957 leader_election.cc:288] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 3 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:13.982403 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32e141d0103848c1a2d473429cffcca9" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:13.982597 13830 raft_consensus.cc:3037] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:13.985276 13830 raft_consensus.cc:2454] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 3.
I20240503 08:15:13.985664 13894 leader_election.cc:302] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:13.986006 16203 raft_consensus.cc:2781] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:15:13.986238 16203 raft_consensus.cc:686] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:13.986538 16203 consensus_queue.cc:237] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:13.991875 13689 catalog_manager.cc:5484] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 1 to 2, leader changed from abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67) to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 2 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:13.995546 13957 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d69d33529e1140e5bc3a0a561012dfa1"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:13.995771 13957 raft_consensus.cc:482] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:13.995904 13957 raft_consensus.cc:3037] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:13.998891 13957 raft_consensus.cc:504] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.000247 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d69d33529e1140e5bc3a0a561012dfa1" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:14.000474 14219 raft_consensus.cc:3037] T d69d33529e1140e5bc3a0a561012dfa1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.002561 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d69d33529e1140e5bc3a0a561012dfa1" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:14.002806 14090 raft_consensus.cc:3032] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:15:14.002928 14090 raft_consensus.cc:729] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:14.002974 14219 raft_consensus.cc:2454] T d69d33529e1140e5bc3a0a561012dfa1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 3.
I20240503 08:15:14.003166 14090 consensus_queue.cc:260] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.003093 13689 catalog_manager.cc:5484] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 2 to 3, leader changed from abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67) to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 3 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } }
I20240503 08:15:14.003386 13895 leader_election.cc:302] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5761f1165461459f9aaa8cd5d8e9b0b9; no voters:
I20240503 08:15:14.003443 14090 raft_consensus.cc:3037] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.003736 16203 raft_consensus.cc:2781] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:15:14.003855 16203 raft_consensus.cc:686] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:14.004086 16203 consensus_queue.cc:237] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.008141 14090 raft_consensus.cc:2396] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 296f3a209cf5447aaff2b8ea96a3bc4f for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.008664 13957 leader_election.cc:288] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 3 election: Requested vote from peers abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:14.009099 14088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32e141d0103848c1a2d473429cffcca9" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:14.009341 14088 raft_consensus.cc:3032] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:15:14.009465 14088 raft_consensus.cc:729] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:14.009431 13689 catalog_manager.cc:5484] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 2 to 3, leader changed from abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67) to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 3 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } }
I20240503 08:15:14.009814 14088 consensus_queue.cc:260] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:14.010226 14088 raft_consensus.cc:3037] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.012545 14088 raft_consensus.cc:2396] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 296f3a209cf5447aaff2b8ea96a3bc4f for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.012751 13957 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cdf9885a1e7c4b8dbb160e370ca0f60c"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:14.012943 13957 raft_consensus.cc:482] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.013058 13957 raft_consensus.cc:3037] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.015039 13957 raft_consensus.cc:504] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:14.015583 13957 leader_election.cc:288] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 3 election: Requested vote from peers abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:14.016089 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdf9885a1e7c4b8dbb160e370ca0f60c" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:14.016310 13830 raft_consensus.cc:3037] T cdf9885a1e7c4b8dbb160e370ca0f60c P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.018106 13957 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c310dd5906a24299b6f62201b30b86db"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:14.018316 13957 raft_consensus.cc:482] T c310dd5906a24299b6f62201b30b86db P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.018385 13830 raft_consensus.cc:2454] T cdf9885a1e7c4b8dbb160e370ca0f60c P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 3.
I20240503 08:15:14.018455 13957 raft_consensus.cc:3037] T c310dd5906a24299b6f62201b30b86db P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:14.018821 13894 leader_election.cc:302] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:14.019177 16232 raft_consensus.cc:2781] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:15:14.019351 16232 raft_consensus.cc:686] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:14.016731 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdf9885a1e7c4b8dbb160e370ca0f60c" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:14.019577 16232 consensus_queue.cc:237] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:14.019897 14090 raft_consensus.cc:3032] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:15:14.020041 14090 raft_consensus.cc:729] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:14.020347 14090 consensus_queue.cc:260] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:14.020653 14090 raft_consensus.cc:3037] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.022922 14090 raft_consensus.cc:2396] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 296f3a209cf5447aaff2b8ea96a3bc4f for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.023042 13689 catalog_manager.cc:5484] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 2 to 3, leader changed from abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67) to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 3 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } }
I20240503 08:15:14.023310 13957 raft_consensus.cc:504] T c310dd5906a24299b6f62201b30b86db P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.023950 13957 leader_election.cc:288] T c310dd5906a24299b6f62201b30b86db P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 2 election: Requested vote from peers abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:14.024405 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c310dd5906a24299b6f62201b30b86db" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:14.024611 14219 raft_consensus.cc:3037] T c310dd5906a24299b6f62201b30b86db P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:14.026687 14219 raft_consensus.cc:2454] T c310dd5906a24299b6f62201b30b86db P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 2.
I20240503 08:15:14.027163 13957 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f1e94dd86fc74c549979d30f0f705da7"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:14.027159 13895 leader_election.cc:302] T c310dd5906a24299b6f62201b30b86db P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5761f1165461459f9aaa8cd5d8e9b0b9; no voters:
I20240503 08:15:14.027204 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c310dd5906a24299b6f62201b30b86db" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:14.027366 13957 raft_consensus.cc:482] T f1e94dd86fc74c549979d30f0f705da7 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.027375 14090 raft_consensus.cc:3032] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:15:14.027456 13957 raft_consensus.cc:3037] T f1e94dd86fc74c549979d30f0f705da7 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:14.027465 14090 raft_consensus.cc:729] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:14.027647 16232 raft_consensus.cc:2781] T c310dd5906a24299b6f62201b30b86db P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:15:14.027715 14090 consensus_queue.cc:260] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.027825 16232 raft_consensus.cc:686] T c310dd5906a24299b6f62201b30b86db P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:14.028128 14090 raft_consensus.cc:3037] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:14.028086 16232 consensus_queue.cc:237] T c310dd5906a24299b6f62201b30b86db P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.030375 13957 raft_consensus.cc:504] T f1e94dd86fc74c549979d30f0f705da7 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.031039 14090 raft_consensus.cc:2454] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 2.
I20240503 08:15:14.031105 13957 leader_election.cc:288] T f1e94dd86fc74c549979d30f0f705da7 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 2 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:14.031591 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e94dd86fc74c549979d30f0f705da7" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:14.031816 13830 raft_consensus.cc:3032] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:15:14.031911 13830 raft_consensus.cc:729] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:14.032222 13957 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "150f0d2d17bc4a59a8d82982790b4594"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:14.032150 13830 consensus_queue.cc:260] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.032389 13957 raft_consensus.cc:482] T 150f0d2d17bc4a59a8d82982790b4594 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.032500 13957 raft_consensus.cc:3037] T 150f0d2d17bc4a59a8d82982790b4594 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:14.032529 13830 raft_consensus.cc:3037] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:14.034456 13830 raft_consensus.cc:2454] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 2.
I20240503 08:15:14.035001 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e94dd86fc74c549979d30f0f705da7" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:14.035065 13894 leader_election.cc:302] T f1e94dd86fc74c549979d30f0f705da7 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:14.035182 14219 raft_consensus.cc:3037] T f1e94dd86fc74c549979d30f0f705da7 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:14.035298 16232 raft_consensus.cc:2781] T f1e94dd86fc74c549979d30f0f705da7 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:15:14.035423 16232 raft_consensus.cc:686] T f1e94dd86fc74c549979d30f0f705da7 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:14.035665 16232 consensus_queue.cc:237] T f1e94dd86fc74c549979d30f0f705da7 P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.036159 13689 catalog_manager.cc:5484] T c310dd5906a24299b6f62201b30b86db P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 1 to 2, leader changed from abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67) to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 2 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } }
I20240503 08:15:14.038179 13957 raft_consensus.cc:504] T 150f0d2d17bc4a59a8d82982790b4594 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.038794 13957 leader_election.cc:288] T 150f0d2d17bc4a59a8d82982790b4594 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 2 election: Requested vote from peers abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:14.039386 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150f0d2d17bc4a59a8d82982790b4594" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:14.039568 14090 raft_consensus.cc:3032] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:15:14.039678 14090 raft_consensus.cc:729] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:14.039677 14218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150f0d2d17bc4a59a8d82982790b4594" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:14.039834 14218 raft_consensus.cc:3037] T 150f0d2d17bc4a59a8d82982790b4594 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:14.039816 13957 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d2ba977460694815a38d415a14e1cd5f"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:14.039980 13957 raft_consensus.cc:482] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.039932 14090 consensus_queue.cc:260] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.040066 13957 raft_consensus.cc:3037] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:14.040267 14090 raft_consensus.cc:3037] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:14.042335 14218 raft_consensus.cc:2454] T 150f0d2d17bc4a59a8d82982790b4594 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 2.
I20240503 08:15:14.042619 13957 raft_consensus.cc:504] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:14.042794 14090 raft_consensus.cc:2454] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 2.
I20240503 08:15:14.042963 13895 leader_election.cc:302] T 150f0d2d17bc4a59a8d82982790b4594 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5761f1165461459f9aaa8cd5d8e9b0b9; no voters:
I20240503 08:15:14.043234 13957 leader_election.cc:288] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 2 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:14.044140 14088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ba977460694815a38d415a14e1cd5f" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:14.044314 14088 raft_consensus.cc:3032] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:15:14.044379 14088 raft_consensus.cc:729] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:14.044476 13957 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "666be8071a904e6cbff984fa293d00b5"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:14.044644 13957 raft_consensus.cc:482] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.044615 14088 consensus_queue.cc:260] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:14.044775 13957 raft_consensus.cc:3037] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:14.044834 14219 raft_consensus.cc:2454] T f1e94dd86fc74c549979d30f0f705da7 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 2.
I20240503 08:15:14.044951 14088 raft_consensus.cc:3037] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:14.047060 14088 raft_consensus.cc:2454] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 2.
I20240503 08:15:14.047447 13896 leader_election.cc:302] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:14.047565 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ba977460694815a38d415a14e1cd5f" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:14.047727 15670 raft_consensus.cc:2781] T 150f0d2d17bc4a59a8d82982790b4594 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:15:14.047783 13830 raft_consensus.cc:3037] T d2ba977460694815a38d415a14e1cd5f P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:14.047879 15670 raft_consensus.cc:686] T 150f0d2d17bc4a59a8d82982790b4594 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:14.048110 15670 consensus_queue.cc:237] T 150f0d2d17bc4a59a8d82982790b4594 P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.048372 13957 raft_consensus.cc:504] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.049350 14088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "666be8071a904e6cbff984fa293d00b5" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:14.049575 14088 raft_consensus.cc:3032] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:15:14.049685 14088 raft_consensus.cc:729] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:14.049921 16236 raft_consensus.cc:2781] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:15:14.050022 14088 consensus_queue.cc:260] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.050410 14088 raft_consensus.cc:3037] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:14.050863 13829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "666be8071a904e6cbff984fa293d00b5" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:14.051038 13829 raft_consensus.cc:3037] T 666be8071a904e6cbff984fa293d00b5 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:14.053197 14088 raft_consensus.cc:2454] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 2.
I20240503 08:15:14.053576 13896 leader_election.cc:302] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:14.054809 13829 raft_consensus.cc:2454] T 666be8071a904e6cbff984fa293d00b5 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 2.
I20240503 08:15:14.055608 13830 raft_consensus.cc:2454] T d2ba977460694815a38d415a14e1cd5f P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 2.
I20240503 08:15:14.058072 13957 leader_election.cc:288] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 2 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:14.058179 16248 raft_consensus.cc:2781] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:15:14.059208 13960 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "667abbd8fb6c49489127c12977d7a323"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:14.059427 13960 raft_consensus.cc:482] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.059528 13960 raft_consensus.cc:3037] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.062234 13960 raft_consensus.cc:504] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.063762 13960 leader_election.cc:288] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 3 election: Requested vote from peers abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:14.064425 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "667abbd8fb6c49489127c12977d7a323" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:14.064673 14090 raft_consensus.cc:3032] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:15:14.064524 13689 catalog_manager.cc:5484] T f1e94dd86fc74c549979d30f0f705da7 P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 1 to 2, leader changed from 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65) to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 2 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } }
I20240503 08:15:14.064806 14090 raft_consensus.cc:729] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:14.064997 13960 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "94cfd3b764b148bb8b42e5a5cb7883b3"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:14.065164 13960 raft_consensus.cc:482] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.065279 13960 raft_consensus.cc:3037] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.065405 14090 consensus_queue.cc:260] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.065804 14090 raft_consensus.cc:3037] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.066746 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "667abbd8fb6c49489127c12977d7a323" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:14.066969 14219 raft_consensus.cc:3037] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.067406 16248 raft_consensus.cc:686] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:14.067770 13960 raft_consensus.cc:504] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.067742 16248 consensus_queue.cc:237] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.068820 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94cfd3b764b148bb8b42e5a5cb7883b3" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:14.069083 13830 raft_consensus.cc:3037] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.069326 14088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94cfd3b764b148bb8b42e5a5cb7883b3" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:14.069519 14088 raft_consensus.cc:3032] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:15:14.069548 14219 raft_consensus.cc:2454] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 3.
I20240503 08:15:14.069597 14088 raft_consensus.cc:729] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:14.069895 14088 consensus_queue.cc:260] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.070181 14088 raft_consensus.cc:3037] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.071888 13830 raft_consensus.cc:2454] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 3.
I20240503 08:15:14.072795 14088 raft_consensus.cc:2396] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 296f3a209cf5447aaff2b8ea96a3bc4f for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.073282 13895 leader_election.cc:302] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5761f1165461459f9aaa8cd5d8e9b0b9; no voters:
I20240503 08:15:14.073537 13960 leader_election.cc:288] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 3 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:14.073648 13894 leader_election.cc:302] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5a198ad76e4f462c8be0859184aeb2fc; no voters: abb3df711d3a46809274b5cfbd4ce4e0
I20240503 08:15:14.074152 16203 raft_consensus.cc:2781] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:15:14.074319 16203 raft_consensus.cc:686] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:14.074514 13960 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "377cd840a48f47238bdaa5ad2257e1f4"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:14.074566 16203 consensus_queue.cc:237] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.074717 13960 raft_consensus.cc:482] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.074818 13960 raft_consensus.cc:3037] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:14.077515 13960 raft_consensus.cc:504] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.081995 16232 raft_consensus.cc:2781] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:15:14.097851 13960 leader_election.cc:288] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 2 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:14.097899 14090 raft_consensus.cc:2396] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 296f3a209cf5447aaff2b8ea96a3bc4f for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.098022 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377cd840a48f47238bdaa5ad2257e1f4" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:14.098022 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377cd840a48f47238bdaa5ad2257e1f4" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:14.098222 14219 raft_consensus.cc:3037] T 377cd840a48f47238bdaa5ad2257e1f4 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:14.098223 13830 raft_consensus.cc:3032] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:15:14.098312 13830 raft_consensus.cc:729] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:14.098762 13830 consensus_queue.cc:260] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.098860 13960 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d7ebe785e08640cb866350eae5797e9f"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:14.099017 13960 raft_consensus.cc:465] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 08:15:14.099254 13830 raft_consensus.cc:3037] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:14.100859 14219 raft_consensus.cc:2454] T 377cd840a48f47238bdaa5ad2257e1f4 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 2.
I20240503 08:15:14.101310 13895 leader_election.cc:302] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5761f1165461459f9aaa8cd5d8e9b0b9; no voters:
I20240503 08:15:14.101620 16247 raft_consensus.cc:2781] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:15:14.101787 16247 raft_consensus.cc:686] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:14.102488 13830 raft_consensus.cc:2454] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 2.
I20240503 08:15:14.102198 16247 consensus_queue.cc:237] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.102824 16232 raft_consensus.cc:686] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:14.103180 16232 consensus_queue.cc:237] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.104933 16236 raft_consensus.cc:686] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:14.105613 16236 consensus_queue.cc:237] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:14.116927 13689 catalog_manager.cc:5484] T 150f0d2d17bc4a59a8d82982790b4594 P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 1 to 2, leader changed from abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67) to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 2 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } }
I20240503 08:15:14.117336 13689 catalog_manager.cc:5484] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 1 to 2, leader changed from abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67) to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 2 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:14.117684 13689 catalog_manager.cc:5484] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 1 to 2, leader changed from abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67) to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 2 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } }
I20240503 08:15:14.123335 13689 catalog_manager.cc:5484] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 2 to 3, leader changed from abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67) to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 3 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } }
I20240503 08:15:14.123664 13689 catalog_manager.cc:5484] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 1 to 2, leader changed from 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65) to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 2 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } }
I20240503 08:15:14.123979 13689 catalog_manager.cc:5484] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 2 to 3, leader changed from abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67) to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 3 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:14.132249 13830 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "045fa6c60d334659aab79f7ba4999661"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:14.132499 13830 raft_consensus.cc:482] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.132648 13830 raft_consensus.cc:3037] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.134689 13830 raft_consensus.cc:504] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.135545 13830 leader_election.cc:288] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 3 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:14.135962 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "045fa6c60d334659aab79f7ba4999661" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:14.136075 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "045fa6c60d334659aab79f7ba4999661" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:14.136164 14219 raft_consensus.cc:3037] T 045fa6c60d334659aab79f7ba4999661 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.136277 14090 raft_consensus.cc:3032] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:15:14.136387 14090 raft_consensus.cc:729] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:14.136672 14090 consensus_queue.cc:260] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.136998 14090 raft_consensus.cc:3037] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.138779 14219 raft_consensus.cc:2454] T 045fa6c60d334659aab79f7ba4999661 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 3.
I20240503 08:15:14.139123 13830 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c7a958fc685e41fb86108dfcb9c6dc8b"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:14.139273 13765 leader_election.cc:302] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:14.139299 13830 raft_consensus.cc:482] T c7a958fc685e41fb86108dfcb9c6dc8b P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.139436 13830 raft_consensus.cc:3037] T c7a958fc685e41fb86108dfcb9c6dc8b P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:14.139547 15593 raft_consensus.cc:2781] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:15:14.139603 14090 raft_consensus.cc:2396] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.139714 15593 raft_consensus.cc:686] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:14.139986 15593 consensus_queue.cc:237] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.141937 13830 raft_consensus.cc:504] T c7a958fc685e41fb86108dfcb9c6dc8b P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.142887 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a958fc685e41fb86108dfcb9c6dc8b" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:14.143066 14219 raft_consensus.cc:3037] T c7a958fc685e41fb86108dfcb9c6dc8b P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:14.145462 14219 raft_consensus.cc:2454] T c7a958fc685e41fb86108dfcb9c6dc8b P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 2.
I20240503 08:15:14.145612 13689 catalog_manager.cc:5484] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 2 to 3, leader changed from abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67) to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 3 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:14.145856 13830 leader_election.cc:288] T c7a958fc685e41fb86108dfcb9c6dc8b P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 2 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:14.145927 13765 leader_election.cc:302] T c7a958fc685e41fb86108dfcb9c6dc8b P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:14.146530 15593 raft_consensus.cc:2781] T c7a958fc685e41fb86108dfcb9c6dc8b P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:15:14.146768 13830 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "51b32125655f4ef78dc49ba7174775da"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:14.146787 15593 raft_consensus.cc:686] T c7a958fc685e41fb86108dfcb9c6dc8b P 5a198ad76e4f462c8be0859184aeb2fc [term 2 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:14.146960 13830 raft_consensus.cc:482] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.147087 13830 raft_consensus.cc:3037] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.147130 15593 consensus_queue.cc:237] T c7a958fc685e41fb86108dfcb9c6dc8b P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.147815 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a958fc685e41fb86108dfcb9c6dc8b" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:14.147990 14090 raft_consensus.cc:3032] T c7a958fc685e41fb86108dfcb9c6dc8b P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:15:14.148087 14090 raft_consensus.cc:729] T c7a958fc685e41fb86108dfcb9c6dc8b P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:14.148324 14090 consensus_queue.cc:260] T c7a958fc685e41fb86108dfcb9c6dc8b P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.148700 14090 raft_consensus.cc:3037] T c7a958fc685e41fb86108dfcb9c6dc8b P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:14.151226 13830 raft_consensus.cc:504] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:14.151335 14090 raft_consensus.cc:2454] T c7a958fc685e41fb86108dfcb9c6dc8b P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 2.
I20240503 08:15:14.151973 13830 leader_election.cc:288] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 3 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:15:14.152495 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51b32125655f4ef78dc49ba7174775da" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:14.152694 13960 raft_consensus.cc:3032] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:15:14.152693 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51b32125655f4ef78dc49ba7174775da" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:14.152801 13960 raft_consensus.cc:729] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:14.152916 14219 raft_consensus.cc:3037] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.153079 13960 consensus_queue.cc:260] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:14.153370 13960 raft_consensus.cc:3037] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.155417 14219 raft_consensus.cc:2454] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 3.
I20240503 08:15:14.155790 13765 leader_election.cc:302] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:14.155987 13960 raft_consensus.cc:2396] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.156018 15593 raft_consensus.cc:2781] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:15:14.156309 13830 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dd3d6353358d4b02af928cd9f9410aca"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:14.156361 15593 raft_consensus.cc:686] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc [term 3 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:14.156481 13830 raft_consensus.cc:482] T dd3d6353358d4b02af928cd9f9410aca P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.156584 13830 raft_consensus.cc:3037] T dd3d6353358d4b02af928cd9f9410aca P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.156610 15593 consensus_queue.cc:237] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:14.158464 13830 raft_consensus.cc:504] T dd3d6353358d4b02af928cd9f9410aca P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.158962 13830 leader_election.cc:288] T dd3d6353358d4b02af928cd9f9410aca P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 4 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:14.159318 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd3d6353358d4b02af928cd9f9410aca" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:14.159492 13960 raft_consensus.cc:3032] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:15:14.159596 13960 raft_consensus.cc:729] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:14.159719 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd3d6353358d4b02af928cd9f9410aca" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:14.159879 14090 raft_consensus.cc:3037] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.159847 13960 consensus_queue.cc:260] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.160131 13960 raft_consensus.cc:3037] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.161903 13830 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8bb1779c7ac64d599cde0665505b3dba"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:14.162042 13830 raft_consensus.cc:482] T 8bb1779c7ac64d599cde0665505b3dba P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.162166 13830 raft_consensus.cc:3037] T 8bb1779c7ac64d599cde0665505b3dba P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.162513 14090 raft_consensus.cc:2396] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.162664 13960 raft_consensus.cc:2396] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.163125 13763 leader_election.cc:302] T dd3d6353358d4b02af928cd9f9410aca P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc; no voters: 296f3a209cf5447aaff2b8ea96a3bc4f, abb3df711d3a46809274b5cfbd4ce4e0
I20240503 08:15:14.163444 15593 raft_consensus.cc:2726] T dd3d6353358d4b02af928cd9f9410aca P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 08:15:14.164224 13830 raft_consensus.cc:504] T 8bb1779c7ac64d599cde0665505b3dba P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.164764 13830 leader_election.cc:288] T 8bb1779c7ac64d599cde0665505b3dba P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 3 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:14.165100 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb1779c7ac64d599cde0665505b3dba" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:14.165271 13960 raft_consensus.cc:3032] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:15:14.165181 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb1779c7ac64d599cde0665505b3dba" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:14.165374 13960 raft_consensus.cc:729] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:14.165436 14090 raft_consensus.cc:3037] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.165598 13960 consensus_queue.cc:260] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.165846 13960 raft_consensus.cc:3037] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.167261 14090 raft_consensus.cc:2454] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 3.
I20240503 08:15:14.167497 13830 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "79014e600505478883fc90159784ce6c"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:14.167546 13960 raft_consensus.cc:2396] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.167665 13766 leader_election.cc:302] T 8bb1779c7ac64d599cde0665505b3dba P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:14.167668 13830 raft_consensus.cc:482] T 79014e600505478883fc90159784ce6c P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.167829 13830 raft_consensus.cc:3037] T 79014e600505478883fc90159784ce6c P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:14.167951 15593 raft_consensus.cc:2781] T 8bb1779c7ac64d599cde0665505b3dba P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:15:14.168092 15593 raft_consensus.cc:686] T 8bb1779c7ac64d599cde0665505b3dba P 5a198ad76e4f462c8be0859184aeb2fc [term 3 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:14.168319 15593 consensus_queue.cc:237] T 8bb1779c7ac64d599cde0665505b3dba P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.169689 13830 raft_consensus.cc:504] T 79014e600505478883fc90159784ce6c P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:14.170374 13830 leader_election.cc:288] T 79014e600505478883fc90159784ce6c P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 2 election: Requested vote from peers abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:14.171057 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79014e600505478883fc90159784ce6c" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:14.171219 14219 raft_consensus.cc:3037] T 79014e600505478883fc90159784ce6c P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:14.172211 13830 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a478c28e5d4646fdb956f5b65e5c6c94"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:14.172384 13830 raft_consensus.cc:482] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.172497 13830 raft_consensus.cc:3037] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.172633 13689 catalog_manager.cc:5484] T c7a958fc685e41fb86108dfcb9c6dc8b P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 1 to 2, leader changed from abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67) to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 2 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:14.173141 13689 catalog_manager.cc:5484] T 8bb1779c7ac64d599cde0665505b3dba P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 2 to 3, leader changed from 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66) to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 3 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:14.173411 14219 raft_consensus.cc:2454] T 79014e600505478883fc90159784ce6c P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 2.
I20240503 08:15:14.173596 13689 catalog_manager.cc:5484] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 2 to 3, leader changed from 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66) to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 3 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } }
I20240503 08:15:14.173882 13765 leader_election.cc:302] T 79014e600505478883fc90159784ce6c P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:14.174213 15593 raft_consensus.cc:2781] T 79014e600505478883fc90159784ce6c P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:15:14.174383 15593 raft_consensus.cc:686] T 79014e600505478883fc90159784ce6c P 5a198ad76e4f462c8be0859184aeb2fc [term 2 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:14.174906 13830 raft_consensus.cc:504] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:14.175539 13830 leader_election.cc:288] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 3 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:15:14.176708 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a478c28e5d4646fdb956f5b65e5c6c94" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:14.176877 13960 raft_consensus.cc:3032] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:15:14.176971 13960 raft_consensus.cc:729] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:14.177194 13960 consensus_queue.cc:260] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:14.177467 13960 raft_consensus.cc:3037] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.174677 15593 consensus_queue.cc:237] T 79014e600505478883fc90159784ce6c P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:14.180354 13830 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "769c4490062b4206951af2f4b9a6da05"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:14.180552 13830 raft_consensus.cc:482] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.180660 13830 raft_consensus.cc:3037] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:14.182454 13960 raft_consensus.cc:2396] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.183791 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79014e600505478883fc90159784ce6c" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:14.183995 14090 raft_consensus.cc:3032] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:15:14.184098 14090 raft_consensus.cc:729] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:14.184414 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a478c28e5d4646fdb956f5b65e5c6c94" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:14.184347 14090 consensus_queue.cc:260] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:14.184612 14219 raft_consensus.cc:3037] T a478c28e5d4646fdb956f5b65e5c6c94 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.184556 13830 raft_consensus.cc:504] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:14.184738 14090 raft_consensus.cc:3037] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:14.185145 13830 leader_election.cc:288] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 2 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:15:14.185482 14218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "769c4490062b4206951af2f4b9a6da05" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:14.185549 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "769c4490062b4206951af2f4b9a6da05" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:14.185660 14218 raft_consensus.cc:3037] T 769c4490062b4206951af2f4b9a6da05 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:14.185701 13960 raft_consensus.cc:3032] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:15:14.185808 13960 raft_consensus.cc:729] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:14.186051 13960 consensus_queue.cc:260] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:14.186374 13960 raft_consensus.cc:3037] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:14.186470 13830 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "de418f9b6bb04763b88ccc3bbe6cd68f"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:14.186604 13830 raft_consensus.cc:482] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.186715 13830 raft_consensus.cc:3037] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.187878 14090 raft_consensus.cc:2454] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 2.
I20240503 08:15:14.188302 14218 raft_consensus.cc:2454] T 769c4490062b4206951af2f4b9a6da05 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 2.
I20240503 08:15:14.188735 13765 leader_election.cc:302] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:14.188927 13960 raft_consensus.cc:2454] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 2.
I20240503 08:15:14.189045 16215 raft_consensus.cc:2781] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:15:14.190228 14218 raft_consensus.cc:1261] T 06e7243793e9467daeed79f3b1f5be5a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Refusing update from remote peer abb3df711d3a46809274b5cfbd4ce4e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:15:14.190827 16210 consensus_queue.cc:1035] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:15:14.191282 14219 raft_consensus.cc:2454] T a478c28e5d4646fdb956f5b65e5c6c94 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 3.
I20240503 08:15:14.191283 13830 raft_consensus.cc:504] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.191692 13765 leader_election.cc:302] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, 5a198ad76e4f462c8be0859184aeb2fc; no voters: 296f3a209cf5447aaff2b8ea96a3bc4f
I20240503 08:15:14.191879 13830 leader_election.cc:288] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 3 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:14.192708 13830 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d7ebe785e08640cb866350eae5797e9f"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:14.192936 13830 raft_consensus.cc:482] T d7ebe785e08640cb866350eae5797e9f P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.193059 13830 raft_consensus.cc:3037] T d7ebe785e08640cb866350eae5797e9f P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:14.194540 16065 heartbeater.cc:503] Master 127.20.230.126:38135 was elected leader, sending a full tablet report...
I20240503 08:15:14.196066 13830 raft_consensus.cc:504] T d7ebe785e08640cb866350eae5797e9f P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:14.196651 13830 leader_election.cc:288] T d7ebe785e08640cb866350eae5797e9f P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 2 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:15:14.197374 13830 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c244226a1c40488f949735420bfacd0b"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:14.197546 13830 raft_consensus.cc:482] T c244226a1c40488f949735420bfacd0b P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.197675 13830 raft_consensus.cc:3037] T c244226a1c40488f949735420bfacd0b P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.198446 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de418f9b6bb04763b88ccc3bbe6cd68f" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:14.198832 14090 raft_consensus.cc:3032] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:15:14.198915 14090 raft_consensus.cc:729] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:14.199244 15593 raft_consensus.cc:2781] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:15:14.199620 15593 raft_consensus.cc:686] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:14.200063 15593 consensus_queue.cc:237] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:14.200137 13689 catalog_manager.cc:5484] T 79014e600505478883fc90159784ce6c P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 1 to 2, leader changed from abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67) to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 2 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } }
I20240503 08:15:14.200238 13830 raft_consensus.cc:504] T c244226a1c40488f949735420bfacd0b P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.202013 13957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c244226a1c40488f949735420bfacd0b" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:14.202204 13957 raft_consensus.cc:3032] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:15:14.202297 13957 raft_consensus.cc:729] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:14.202456 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7ebe785e08640cb866350eae5797e9f" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:14.203277 13830 leader_election.cc:288] T c244226a1c40488f949735420bfacd0b P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 3 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:14.203454 13960 raft_consensus.cc:3032] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:15:14.203425 14088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c244226a1c40488f949735420bfacd0b" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:14.203553 13960 raft_consensus.cc:729] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:14.203608 14088 raft_consensus.cc:3037] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.204370 13830 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "32e141d0103848c1a2d473429cffcca9"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:14.204317 13960 consensus_queue.cc:260] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:14.204610 13830 raft_consensus.cc:482] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.199142 14090 consensus_queue.cc:260] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.204671 13960 raft_consensus.cc:3037] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:14.204861 14090 raft_consensus.cc:3037] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.205128 13830 raft_consensus.cc:3037] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.206317 14088 raft_consensus.cc:2454] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 3.
I20240503 08:15:14.207338 13960 raft_consensus.cc:2454] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 2.
I20240503 08:15:14.207832 14090 raft_consensus.cc:2396] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.207791 13763 leader_election.cc:302] T d7ebe785e08640cb866350eae5797e9f P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:14.202564 13957 consensus_queue.cc:260] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.208241 13957 raft_consensus.cc:3037] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.210295 13830 raft_consensus.cc:504] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:14.210480 16272 raft_consensus.cc:2781] T d7ebe785e08640cb866350eae5797e9f P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:15:14.211691 16272 raft_consensus.cc:686] T d7ebe785e08640cb866350eae5797e9f P 5a198ad76e4f462c8be0859184aeb2fc [term 2 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:14.212020 16272 consensus_queue.cc:237] T d7ebe785e08640cb866350eae5797e9f P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:14.213284 13830 leader_election.cc:288] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 4 election: Requested vote from peers abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:15:14.213949 13766 leader_election.cc:302] T c244226a1c40488f949735420bfacd0b P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:14.214217 13830 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cdf9885a1e7c4b8dbb160e370ca0f60c"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:14.214279 16215 raft_consensus.cc:686] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:14.214390 13830 raft_consensus.cc:482] T cdf9885a1e7c4b8dbb160e370ca0f60c P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.214497 13830 raft_consensus.cc:3037] T cdf9885a1e7c4b8dbb160e370ca0f60c P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.214565 16215 consensus_queue.cc:237] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:14.215560 13957 raft_consensus.cc:2396] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.215791 13829 raft_consensus.cc:1261] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Refusing update from remote peer abb3df711d3a46809274b5cfbd4ce4e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:15:14.216473 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de418f9b6bb04763b88ccc3bbe6cd68f" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:14.216524 16222 consensus_queue.cc:1035] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:15:14.216665 14219 raft_consensus.cc:3037] T de418f9b6bb04763b88ccc3bbe6cd68f P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.219259 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32e141d0103848c1a2d473429cffcca9" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:14.219453 13960 raft_consensus.cc:3032] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:15:14.219545 13960 raft_consensus.cc:729] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:14.219802 13960 consensus_queue.cc:260] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:14.220329 14218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7ebe785e08640cb866350eae5797e9f" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:14.220551 14218 raft_consensus.cc:3037] T d7ebe785e08640cb866350eae5797e9f P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:14.220616 13960 raft_consensus.cc:3037] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.221668 16271 raft_consensus.cc:2781] T c244226a1c40488f949735420bfacd0b P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:15:14.224903 13960 raft_consensus.cc:2396] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.225260 16271 raft_consensus.cc:686] T c244226a1c40488f949735420bfacd0b P 5a198ad76e4f462c8be0859184aeb2fc [term 3 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:14.225685 14219 raft_consensus.cc:2454] T de418f9b6bb04763b88ccc3bbe6cd68f P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 3.
I20240503 08:15:14.225817 14218 raft_consensus.cc:2454] T d7ebe785e08640cb866350eae5797e9f P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 2.
I20240503 08:15:14.226521 13765 leader_election.cc:302] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, 5a198ad76e4f462c8be0859184aeb2fc; no voters: abb3df711d3a46809274b5cfbd4ce4e0
I20240503 08:15:14.227159 16271 consensus_queue.cc:237] T c244226a1c40488f949735420bfacd0b P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.228315 15593 raft_consensus.cc:2781] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:15:14.228406 13830 raft_consensus.cc:504] T cdf9885a1e7c4b8dbb160e370ca0f60c P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:14.229225 15593 raft_consensus.cc:686] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [term 3 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:14.229444 14088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdf9885a1e7c4b8dbb160e370ca0f60c" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:14.229473 15593 consensus_queue.cc:237] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.229648 14088 raft_consensus.cc:3037] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.232479 14088 raft_consensus.cc:2396] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.232806 13689 catalog_manager.cc:5484] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 1 to 2, leader changed from 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66) to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 2 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } }
I20240503 08:15:14.232968 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32e141d0103848c1a2d473429cffcca9" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:14.233156 14090 raft_consensus.cc:3037] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.235581 14090 raft_consensus.cc:2396] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.235953 13766 leader_election.cc:302] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc; no voters: 296f3a209cf5447aaff2b8ea96a3bc4f, abb3df711d3a46809274b5cfbd4ce4e0
I20240503 08:15:14.236464 13830 leader_election.cc:288] T cdf9885a1e7c4b8dbb160e370ca0f60c P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 4 election: Requested vote from peers abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:15:14.236948 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdf9885a1e7c4b8dbb160e370ca0f60c" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:14.237213 13960 raft_consensus.cc:3032] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:15:14.237318 13960 raft_consensus.cc:729] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:14.237581 13960 consensus_queue.cc:260] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:14.237937 13960 raft_consensus.cc:3037] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.238720 16215 raft_consensus.cc:2726] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 08:15:14.240530 13960 raft_consensus.cc:2396] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.240788 13830 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4b680592b1774f09ba79789afa55a0f1"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:14.240998 13830 raft_consensus.cc:482] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.241102 13830 raft_consensus.cc:3037] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.242396 13763 leader_election.cc:302] T cdf9885a1e7c4b8dbb160e370ca0f60c P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc; no voters: 296f3a209cf5447aaff2b8ea96a3bc4f, abb3df711d3a46809274b5cfbd4ce4e0
I20240503 08:15:14.242312 13689 catalog_manager.cc:5484] T d7ebe785e08640cb866350eae5797e9f P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 1 to 2, leader changed from 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66) to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 2 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } }
I20240503 08:15:14.242915 13689 catalog_manager.cc:5484] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 2 to 3, leader changed from 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66) to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 3 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } }
I20240503 08:15:14.243271 15593 raft_consensus.cc:2726] T cdf9885a1e7c4b8dbb160e370ca0f60c P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 08:15:14.243451 13689 catalog_manager.cc:5484] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 2 to 3, leader changed from abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67) to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 3 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:14.243758 13830 raft_consensus.cc:504] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:14.243979 13689 catalog_manager.cc:5484] T c244226a1c40488f949735420bfacd0b P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 2 to 3, leader changed from 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66) to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 3 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:14.244503 13830 leader_election.cc:288] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 3 election: Requested vote from peers abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:14.244712 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b680592b1774f09ba79789afa55a0f1" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:14.244923 14090 raft_consensus.cc:3032] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:15:14.244994 14090 raft_consensus.cc:729] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:14.244941 14217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b680592b1774f09ba79789afa55a0f1" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:14.245287 14217 raft_consensus.cc:3037] T 4b680592b1774f09ba79789afa55a0f1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.245404 14090 consensus_queue.cc:260] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:14.245707 14090 raft_consensus.cc:3037] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.245726 13830 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f1e94dd86fc74c549979d30f0f705da7"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:14.245981 13830 raft_consensus.cc:482] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.246130 13830 raft_consensus.cc:3037] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.246944 13829 raft_consensus.cc:1261] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Refusing update from remote peer abb3df711d3a46809274b5cfbd4ce4e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:15:14.247437 16222 consensus_queue.cc:1035] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:15:14.248339 14090 raft_consensus.cc:2396] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.250478 13830 raft_consensus.cc:504] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.251405 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e94dd86fc74c549979d30f0f705da7" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:14.251603 14219 raft_consensus.cc:3037] T f1e94dd86fc74c549979d30f0f705da7 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.251874 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e94dd86fc74c549979d30f0f705da7" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:14.252071 13960 raft_consensus.cc:3032] T f1e94dd86fc74c549979d30f0f705da7 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:15:14.252117 14217 raft_consensus.cc:2454] T 4b680592b1774f09ba79789afa55a0f1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 3.
I20240503 08:15:14.252171 13960 raft_consensus.cc:729] T f1e94dd86fc74c549979d30f0f705da7 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:14.252400 13830 leader_election.cc:288] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 3 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:14.252457 13960 consensus_queue.cc:260] T f1e94dd86fc74c549979d30f0f705da7 P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.252794 13960 raft_consensus.cc:3037] T f1e94dd86fc74c549979d30f0f705da7 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.253708 14219 raft_consensus.cc:2454] T f1e94dd86fc74c549979d30f0f705da7 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 3.
I20240503 08:15:14.254205 13765 leader_election.cc:302] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, 5a198ad76e4f462c8be0859184aeb2fc; no voters: abb3df711d3a46809274b5cfbd4ce4e0
I20240503 08:15:14.254619 13765 leader_election.cc:302] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:14.255107 15593 raft_consensus.cc:2781] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:15:14.255168 16272 raft_consensus.cc:2781] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:15:14.255295 15593 raft_consensus.cc:686] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:14.255479 16272 raft_consensus.cc:686] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:14.255707 13960 raft_consensus.cc:2396] T f1e94dd86fc74c549979d30f0f705da7 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.255692 15593 consensus_queue.cc:237] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:14.256384 13830 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "06e7243793e9467daeed79f3b1f5be5a"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:14.256887 13830 raft_consensus.cc:482] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.255687 16272 consensus_queue.cc:237] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.257400 13830 raft_consensus.cc:3037] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.258245 14219 raft_consensus.cc:1261] T e82cb19fece24181b7607d9447ba1eb2 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Refusing update from remote peer abb3df711d3a46809274b5cfbd4ce4e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:15:14.258934 15937 consensus_queue.cc:1035] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:15:14.260011 13830 raft_consensus.cc:504] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.260942 13830 leader_election.cc:288] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 3 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:14.261340 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e7243793e9467daeed79f3b1f5be5a" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:14.261533 14219 raft_consensus.cc:3037] T 06e7243793e9467daeed79f3b1f5be5a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.261854 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e7243793e9467daeed79f3b1f5be5a" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:14.262197 14090 raft_consensus.cc:3032] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:15:14.262389 14090 raft_consensus.cc:729] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:14.262894 14090 consensus_queue.cc:260] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.263484 14090 raft_consensus.cc:3037] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.264286 14219 raft_consensus.cc:2454] T 06e7243793e9467daeed79f3b1f5be5a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 3.
I20240503 08:15:14.265302 13765 leader_election.cc:302] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:14.265594 13830 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "377cd840a48f47238bdaa5ad2257e1f4"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:14.265789 16281 raft_consensus.cc:2781] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:15:14.265810 13830 raft_consensus.cc:482] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.265933 16281 raft_consensus.cc:686] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [term 3 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:14.265915 13830 raft_consensus.cc:3037] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.266783 14090 raft_consensus.cc:2454] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 3.
I20240503 08:15:14.266218 16281 consensus_queue.cc:237] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.268739 13830 raft_consensus.cc:504] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.270154 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377cd840a48f47238bdaa5ad2257e1f4" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:14.270421 13960 raft_consensus.cc:3032] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:15:14.270558 13960 raft_consensus.cc:729] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:14.270615 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377cd840a48f47238bdaa5ad2257e1f4" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:14.270799 14219 raft_consensus.cc:3037] T 377cd840a48f47238bdaa5ad2257e1f4 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.271129 13960 consensus_queue.cc:260] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.271625 13960 raft_consensus.cc:3037] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.273537 14219 raft_consensus.cc:2454] T 377cd840a48f47238bdaa5ad2257e1f4 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 3.
I20240503 08:15:14.274029 13765 leader_election.cc:302] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:14.274291 16215 raft_consensus.cc:2781] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:15:14.274343 13830 leader_election.cc:288] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 3 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:14.274418 16215 raft_consensus.cc:686] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:14.274657 16215 consensus_queue.cc:237] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.275265 13828 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d2ba977460694815a38d415a14e1cd5f"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:14.275449 13828 raft_consensus.cc:482] T d2ba977460694815a38d415a14e1cd5f P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.275560 13828 raft_consensus.cc:3037] T d2ba977460694815a38d415a14e1cd5f P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.277125 13960 raft_consensus.cc:2396] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.278460 13828 raft_consensus.cc:504] T d2ba977460694815a38d415a14e1cd5f P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:14.279280 13828 leader_election.cc:288] T d2ba977460694815a38d415a14e1cd5f P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 3 election: Requested vote from peers abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:15:14.279366 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ba977460694815a38d415a14e1cd5f" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:14.279567 14090 raft_consensus.cc:3037] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.280121 13828 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9cae16af5a034cc29b8bb410a758c332"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:14.280315 13828 raft_consensus.cc:482] T 9cae16af5a034cc29b8bb410a758c332 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.280418 13828 raft_consensus.cc:3037] T 9cae16af5a034cc29b8bb410a758c332 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:14.282406 14090 raft_consensus.cc:2454] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 3.
I20240503 08:15:14.282821 13766 leader_election.cc:302] T d2ba977460694815a38d415a14e1cd5f P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:14.282871 13828 raft_consensus.cc:504] T 9cae16af5a034cc29b8bb410a758c332 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:14.283099 16281 raft_consensus.cc:2781] T d2ba977460694815a38d415a14e1cd5f P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:15:14.283232 16281 raft_consensus.cc:686] T d2ba977460694815a38d415a14e1cd5f P 5a198ad76e4f462c8be0859184aeb2fc [term 3 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:14.283444 16281 consensus_queue.cc:237] T d2ba977460694815a38d415a14e1cd5f P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:14.283766 13828 leader_election.cc:288] T 9cae16af5a034cc29b8bb410a758c332 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 2 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:14.284152 14088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cae16af5a034cc29b8bb410a758c332" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:14.284211 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cae16af5a034cc29b8bb410a758c332" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:14.284338 14088 raft_consensus.cc:3032] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:15:14.284410 14219 raft_consensus.cc:3037] T 9cae16af5a034cc29b8bb410a758c332 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:14.284421 14088 raft_consensus.cc:729] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:14.284694 14088 consensus_queue.cc:260] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:14.285032 14088 raft_consensus.cc:3037] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:14.286953 14219 raft_consensus.cc:2454] T 9cae16af5a034cc29b8bb410a758c332 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 2.
I20240503 08:15:14.287317 13765 leader_election.cc:302] T 9cae16af5a034cc29b8bb410a758c332 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:14.287390 14088 raft_consensus.cc:2454] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 2.
I20240503 08:15:14.288062 16292 raft_consensus.cc:2781] T 9cae16af5a034cc29b8bb410a758c332 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:15:14.288182 13828 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "666be8071a904e6cbff984fa293d00b5"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:14.288261 16292 raft_consensus.cc:686] T 9cae16af5a034cc29b8bb410a758c332 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:14.288475 13828 raft_consensus.cc:482] T 666be8071a904e6cbff984fa293d00b5 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.288662 13828 raft_consensus.cc:3037] T 666be8071a904e6cbff984fa293d00b5 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.288619 16292 consensus_queue.cc:237] T 9cae16af5a034cc29b8bb410a758c332 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:14.292032 13828 raft_consensus.cc:504] T 666be8071a904e6cbff984fa293d00b5 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.292649 13828 leader_election.cc:288] T 666be8071a904e6cbff984fa293d00b5 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 3 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:14.293062 13957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "666be8071a904e6cbff984fa293d00b5" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:14.293088 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ba977460694815a38d415a14e1cd5f" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:14.293316 13957 raft_consensus.cc:3032] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:15:14.293464 13957 raft_consensus.cc:729] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:14.293756 13957 consensus_queue.cc:260] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.293946 13830 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "94cfd3b764b148bb8b42e5a5cb7883b3"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:14.294008 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "666be8071a904e6cbff984fa293d00b5" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:14.294106 13830 raft_consensus.cc:482] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.294132 13957 raft_consensus.cc:3037] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.294159 14090 raft_consensus.cc:3037] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.294193 13830 raft_consensus.cc:3037] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.296938 13957 raft_consensus.cc:2396] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.297477 13960 raft_consensus.cc:3032] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:15:14.297605 13960 raft_consensus.cc:729] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:14.297925 13960 consensus_queue.cc:260] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:14.298260 13960 raft_consensus.cc:3037] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.300153 13830 raft_consensus.cc:504] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.301563 13957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94cfd3b764b148bb8b42e5a5cb7883b3" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:14.301751 13957 raft_consensus.cc:3032] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:15:14.301874 13957 raft_consensus.cc:729] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:14.302101 13957 consensus_queue.cc:260] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.302356 13957 raft_consensus.cc:3037] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.303650 13830 leader_election.cc:288] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 4 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:14.303617 14088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94cfd3b764b148bb8b42e5a5cb7883b3" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:14.303835 14088 raft_consensus.cc:3037] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.304562 13828 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "84204cbcda32446ab708717e9633af69"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:14.304735 13828 raft_consensus.cc:482] T 84204cbcda32446ab708717e9633af69 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.304821 13828 raft_consensus.cc:3037] T 84204cbcda32446ab708717e9633af69 P 5a198ad76e4f462c8be0859184aeb2fc [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:14.304903 13957 raft_consensus.cc:2396] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.306491 13960 raft_consensus.cc:2396] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.306746 14088 raft_consensus.cc:2396] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.307135 13766 leader_election.cc:302] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc; no voters: 296f3a209cf5447aaff2b8ea96a3bc4f, abb3df711d3a46809274b5cfbd4ce4e0
I20240503 08:15:14.307629 16272 raft_consensus.cc:2726] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 08:15:14.307618 13828 raft_consensus.cc:504] T 84204cbcda32446ab708717e9633af69 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:14.308682 14217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84204cbcda32446ab708717e9633af69" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:14.308880 14217 raft_consensus.cc:3037] T 84204cbcda32446ab708717e9633af69 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:14.310734 14090 raft_consensus.cc:2454] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 3.
I20240503 08:15:14.311113 13766 leader_election.cc:302] T 666be8071a904e6cbff984fa293d00b5 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc, abb3df711d3a46809274b5cfbd4ce4e0; no voters: 296f3a209cf5447aaff2b8ea96a3bc4f
I20240503 08:15:14.311372 14217 raft_consensus.cc:2454] T 84204cbcda32446ab708717e9633af69 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 2.
I20240503 08:15:14.311695 16281 raft_consensus.cc:2781] T 666be8071a904e6cbff984fa293d00b5 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:15:14.311844 16281 raft_consensus.cc:686] T 666be8071a904e6cbff984fa293d00b5 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:14.311870 13765 leader_election.cc:302] T 84204cbcda32446ab708717e9633af69 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:14.312094 13828 leader_election.cc:288] T 84204cbcda32446ab708717e9633af69 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 2 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:15:14.312227 16292 raft_consensus.cc:2781] T 84204cbcda32446ab708717e9633af69 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:15:14.312103 16281 consensus_queue.cc:237] T 666be8071a904e6cbff984fa293d00b5 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.312367 16292 raft_consensus.cc:686] T 84204cbcda32446ab708717e9633af69 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:14.312587 16292 consensus_queue.cc:237] T 84204cbcda32446ab708717e9633af69 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:14.313680 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84204cbcda32446ab708717e9633af69" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:14.313897 13960 raft_consensus.cc:3032] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:15:14.313987 13960 raft_consensus.cc:729] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:14.314440 13960 consensus_queue.cc:260] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:14.314780 13960 raft_consensus.cc:3037] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:14.317160 13960 raft_consensus.cc:2454] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 2.
I20240503 08:15:14.319733 13830 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e82cb19fece24181b7607d9447ba1eb2"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:14.319994 13830 raft_consensus.cc:482] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.320231 13830 raft_consensus.cc:3037] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.321661 13689 catalog_manager.cc:5484] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 2 to 3, leader changed from abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67) to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 3 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } }
I20240503 08:15:14.322275 13689 catalog_manager.cc:5484] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 2 to 3, leader changed from 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66) to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 3 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } }
I20240503 08:15:14.323215 13830 raft_consensus.cc:504] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.323365 14090 raft_consensus.cc:1261] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Refusing update from remote peer 296f3a209cf5447aaff2b8ea96a3bc4f: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240503 08:15:14.323479 14090 pending_rounds.cc:76] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0: Aborting all ops after (but not including) 3
I20240503 08:15:14.323565 14090 pending_rounds.cc:97] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 08:15:14.323721 14090 raft_consensus.cc:2866] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:15:14.324102 13830 leader_election.cc:288] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 3 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:14.324538 14088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82cb19fece24181b7607d9447ba1eb2" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:14.324710 14088 raft_consensus.cc:3032] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:15:14.324806 14088 raft_consensus.cc:729] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:14.325045 14088 consensus_queue.cc:260] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.325397 14088 raft_consensus.cc:3037] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.325395 16232 consensus_queue.cc:1035] T d09e60e9c309425da9d27ae1153e4510 P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:15:14.326601 14217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82cb19fece24181b7607d9447ba1eb2" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:14.326881 14217 raft_consensus.cc:3037] T e82cb19fece24181b7607d9447ba1eb2 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.330420 14217 raft_consensus.cc:2454] T e82cb19fece24181b7607d9447ba1eb2 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 3.
I20240503 08:15:14.330981 13765 leader_election.cc:302] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:14.331172 14088 raft_consensus.cc:2454] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 3.
I20240503 08:15:14.332067 15593 raft_consensus.cc:2781] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:15:14.332231 15593 raft_consensus.cc:686] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:14.332491 15593 consensus_queue.cc:237] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.341722 13689 catalog_manager.cc:5484] T d2ba977460694815a38d415a14e1cd5f P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 2 to 3, leader changed from 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66) to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 3 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } }
I20240503 08:15:14.342370 13689 catalog_manager.cc:5484] T 666be8071a904e6cbff984fa293d00b5 P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 2 to 3, leader changed from 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66) to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 3 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:14.342921 13689 catalog_manager.cc:5484] T 84204cbcda32446ab708717e9633af69 P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 1 to 2, leader changed from 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66) to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 2 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } }
I20240503 08:15:14.343413 13689 catalog_manager.cc:5484] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 2 to 3, leader changed from abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67) to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 3 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:14.343861 13689 catalog_manager.cc:5484] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 2 to 3, leader changed from 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66) to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 3 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } }
I20240503 08:15:14.344323 13689 catalog_manager.cc:5484] T 9cae16af5a034cc29b8bb410a758c332 P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 1 to 2, leader changed from abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67) to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 2 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } }
I20240503 08:15:14.344759 13689 catalog_manager.cc:5484] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 2 to 3, leader changed from abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67) to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 3 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:14.355113 14217 raft_consensus.cc:1261] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Refusing update from remote peer 296f3a209cf5447aaff2b8ea96a3bc4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 08:15:14.356599 15670 consensus_queue.cc:1035] T d09e60e9c309425da9d27ae1153e4510 P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:15:14.364105 14217 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "045fa6c60d334659aab79f7ba4999661"
dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
from {username='slave'} at 127.0.0.1:58176
I20240503 08:15:14.364370 14217 raft_consensus.cc:482] T 045fa6c60d334659aab79f7ba4999661 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.364480 14217 raft_consensus.cc:3037] T 045fa6c60d334659aab79f7ba4999661 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.367228 14217 raft_consensus.cc:504] T 045fa6c60d334659aab79f7ba4999661 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.368049 14217 leader_election.cc:288] T 045fa6c60d334659aab79f7ba4999661 P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 4 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:14.368625 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "045fa6c60d334659aab79f7ba4999661" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:14.368774 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "045fa6c60d334659aab79f7ba4999661" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:14.368870 13830 raft_consensus.cc:3032] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:15:14.368947 13830 raft_consensus.cc:729] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:14.368983 14090 raft_consensus.cc:3037] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.369035 14219 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "667abbd8fb6c49489127c12977d7a323"
dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
from {username='slave'} at 127.0.0.1:58176
I20240503 08:15:14.369210 14219 raft_consensus.cc:482] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.369313 14219 raft_consensus.cc:3037] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.369218 13830 consensus_queue.cc:260] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.369494 13830 raft_consensus.cc:3037] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.371565 14219 raft_consensus.cc:504] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.371676 14090 raft_consensus.cc:2396] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5761f1165461459f9aaa8cd5d8e9b0b9 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.372220 13830 raft_consensus.cc:2396] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5761f1165461459f9aaa8cd5d8e9b0b9 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.372215 14219 leader_election.cc:288] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 4 election: Requested vote from peers abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:15:14.372644 14154 leader_election.cc:302] T 045fa6c60d334659aab79f7ba4999661 P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9; no voters: 5a198ad76e4f462c8be0859184aeb2fc, abb3df711d3a46809274b5cfbd4ce4e0
I20240503 08:15:14.372915 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "667abbd8fb6c49489127c12977d7a323" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:14.373073 16069 raft_consensus.cc:2726] T 045fa6c60d334659aab79f7ba4999661 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 08:15:14.373133 13960 raft_consensus.cc:3032] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:15:14.373224 13960 raft_consensus.cc:729] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:14.373473 13960 consensus_queue.cc:260] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.373539 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "667abbd8fb6c49489127c12977d7a323" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:14.373780 13960 raft_consensus.cc:3037] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.373863 14090 raft_consensus.cc:3037] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.375864 14219 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "51b32125655f4ef78dc49ba7174775da"
dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
from {username='slave'} at 127.0.0.1:58176
I20240503 08:15:14.376031 14219 raft_consensus.cc:482] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.376113 14219 raft_consensus.cc:3037] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.376314 14090 raft_consensus.cc:2396] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5761f1165461459f9aaa8cd5d8e9b0b9 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.376555 13960 raft_consensus.cc:2396] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5761f1165461459f9aaa8cd5d8e9b0b9 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.376976 14153 leader_election.cc:302] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9; no voters: 296f3a209cf5447aaff2b8ea96a3bc4f, abb3df711d3a46809274b5cfbd4ce4e0
I20240503 08:15:14.377311 16069 raft_consensus.cc:2726] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 08:15:14.378726 14219 raft_consensus.cc:504] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:14.379334 14219 leader_election.cc:288] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 4 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:14.379683 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51b32125655f4ef78dc49ba7174775da" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:14.379854 13960 raft_consensus.cc:3037] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.379840 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51b32125655f4ef78dc49ba7174775da" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:14.380038 13830 raft_consensus.cc:3032] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:15:14.380215 13830 raft_consensus.cc:729] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:14.380470 14219 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "79014e600505478883fc90159784ce6c"
dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
from {username='slave'} at 127.0.0.1:58176
I20240503 08:15:14.380659 14219 raft_consensus.cc:482] T 79014e600505478883fc90159784ce6c P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.380580 13830 consensus_queue.cc:260] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:14.380864 14219 raft_consensus.cc:3037] T 79014e600505478883fc90159784ce6c P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.380913 13830 raft_consensus.cc:3037] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.382483 13960 raft_consensus.cc:2396] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5761f1165461459f9aaa8cd5d8e9b0b9 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.383430 14219 raft_consensus.cc:504] T 79014e600505478883fc90159784ce6c P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:14.383781 13830 raft_consensus.cc:2396] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5761f1165461459f9aaa8cd5d8e9b0b9 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.384240 14154 leader_election.cc:302] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9; no voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5a198ad76e4f462c8be0859184aeb2fc
I20240503 08:15:14.385022 16069 raft_consensus.cc:2726] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 08:15:14.389585 14219 leader_election.cc:288] T 79014e600505478883fc90159784ce6c P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 3 election: Requested vote from peers abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:14.390434 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79014e600505478883fc90159784ce6c" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:14.390667 13830 raft_consensus.cc:3032] T 79014e600505478883fc90159784ce6c P 5a198ad76e4f462c8be0859184aeb2fc [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:15:14.390756 13830 raft_consensus.cc:729] T 79014e600505478883fc90159784ce6c P 5a198ad76e4f462c8be0859184aeb2fc [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:14.391050 13830 consensus_queue.cc:260] T 79014e600505478883fc90159784ce6c P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:14.391268 14219 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c7a958fc685e41fb86108dfcb9c6dc8b"
dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
from {username='slave'} at 127.0.0.1:58176
I20240503 08:15:14.391336 13830 raft_consensus.cc:3037] T 79014e600505478883fc90159784ce6c P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.391470 14219 raft_consensus.cc:482] T c7a958fc685e41fb86108dfcb9c6dc8b P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.391572 14219 raft_consensus.cc:3037] T c7a958fc685e41fb86108dfcb9c6dc8b P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.393357 14219 raft_consensus.cc:504] T c7a958fc685e41fb86108dfcb9c6dc8b P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.393905 14219 leader_election.cc:288] T c7a958fc685e41fb86108dfcb9c6dc8b P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 3 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:14.394556 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79014e600505478883fc90159784ce6c" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:14.394731 14219 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "70e8bfaa82464d3ab3338a779457dc02"
dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
from {username='slave'} at 127.0.0.1:58176
I20240503 08:15:14.394820 14090 raft_consensus.cc:3037] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.394891 14219 raft_consensus.cc:482] T 70e8bfaa82464d3ab3338a779457dc02 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.394999 14219 raft_consensus.cc:3037] T 70e8bfaa82464d3ab3338a779457dc02 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.396692 14219 raft_consensus.cc:504] T 70e8bfaa82464d3ab3338a779457dc02 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.397161 14219 leader_election.cc:288] T 70e8bfaa82464d3ab3338a779457dc02 P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 3 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:14.397567 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70e8bfaa82464d3ab3338a779457dc02" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:14.397650 14090 raft_consensus.cc:2454] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5761f1165461459f9aaa8cd5d8e9b0b9 in term 3.
I20240503 08:15:14.397948 13960 raft_consensus.cc:3032] T 70e8bfaa82464d3ab3338a779457dc02 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:15:14.398047 13960 raft_consensus.cc:729] T 70e8bfaa82464d3ab3338a779457dc02 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:14.398383 14219 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d7ebe785e08640cb866350eae5797e9f"
dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
from {username='slave'} at 127.0.0.1:58176
I20240503 08:15:14.398303 13960 consensus_queue.cc:260] T 70e8bfaa82464d3ab3338a779457dc02 P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.398483 13828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a958fc685e41fb86108dfcb9c6dc8b" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:14.398605 13960 raft_consensus.cc:3037] T 70e8bfaa82464d3ab3338a779457dc02 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.398619 14219 raft_consensus.cc:482] T d7ebe785e08640cb866350eae5797e9f P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.398692 13828 raft_consensus.cc:3032] T c7a958fc685e41fb86108dfcb9c6dc8b P 5a198ad76e4f462c8be0859184aeb2fc [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:15:14.398713 14219 raft_consensus.cc:3037] T d7ebe785e08640cb866350eae5797e9f P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.398784 13828 raft_consensus.cc:729] T c7a958fc685e41fb86108dfcb9c6dc8b P 5a198ad76e4f462c8be0859184aeb2fc [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:14.399021 13828 consensus_queue.cc:260] T c7a958fc685e41fb86108dfcb9c6dc8b P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.399331 13828 raft_consensus.cc:3037] T c7a958fc685e41fb86108dfcb9c6dc8b P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.401180 13960 raft_consensus.cc:2396] T 70e8bfaa82464d3ab3338a779457dc02 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5761f1165461459f9aaa8cd5d8e9b0b9 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.401803 14219 raft_consensus.cc:504] T d7ebe785e08640cb866350eae5797e9f P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:14.402622 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7ebe785e08640cb866350eae5797e9f" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:14.402786 14219 leader_election.cc:288] T d7ebe785e08640cb866350eae5797e9f P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 3 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:14.402837 13960 raft_consensus.cc:3037] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.403235 13829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7ebe785e08640cb866350eae5797e9f" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:14.403422 13829 raft_consensus.cc:3032] T d7ebe785e08640cb866350eae5797e9f P 5a198ad76e4f462c8be0859184aeb2fc [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:15:14.403494 13829 raft_consensus.cc:729] T d7ebe785e08640cb866350eae5797e9f P 5a198ad76e4f462c8be0859184aeb2fc [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:14.403632 14219 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a478c28e5d4646fdb956f5b65e5c6c94"
dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
from {username='slave'} at 127.0.0.1:58176
I20240503 08:15:14.403786 14219 raft_consensus.cc:482] T a478c28e5d4646fdb956f5b65e5c6c94 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.403748 13829 consensus_queue.cc:260] T d7ebe785e08640cb866350eae5797e9f P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:14.403883 14219 raft_consensus.cc:3037] T a478c28e5d4646fdb956f5b65e5c6c94 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.404016 13829 raft_consensus.cc:3037] T d7ebe785e08640cb866350eae5797e9f P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.405483 13960 raft_consensus.cc:2454] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5761f1165461459f9aaa8cd5d8e9b0b9 in term 3.
I20240503 08:15:14.405848 14153 leader_election.cc:302] T d7ebe785e08640cb866350eae5797e9f P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5761f1165461459f9aaa8cd5d8e9b0b9; no voters:
I20240503 08:15:14.406105 16069 raft_consensus.cc:2781] T d7ebe785e08640cb866350eae5797e9f P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:15:14.406560 14219 raft_consensus.cc:504] T a478c28e5d4646fdb956f5b65e5c6c94 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:14.407354 13829 raft_consensus.cc:2396] T d7ebe785e08640cb866350eae5797e9f P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5761f1165461459f9aaa8cd5d8e9b0b9 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.402184 14089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a958fc685e41fb86108dfcb9c6dc8b" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:14.407549 13827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a478c28e5d4646fdb956f5b65e5c6c94" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:14.407734 14089 raft_consensus.cc:3037] T c7a958fc685e41fb86108dfcb9c6dc8b P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.407786 13827 raft_consensus.cc:3032] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:15:14.407877 13827 raft_consensus.cc:729] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:14.408144 13827 consensus_queue.cc:260] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:14.408428 13827 raft_consensus.cc:3037] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.410477 14089 raft_consensus.cc:2454] T c7a958fc685e41fb86108dfcb9c6dc8b P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5761f1165461459f9aaa8cd5d8e9b0b9 in term 3.
I20240503 08:15:14.402318 14088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70e8bfaa82464d3ab3338a779457dc02" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:14.410781 14088 raft_consensus.cc:3037] T 70e8bfaa82464d3ab3338a779457dc02 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.411168 13827 raft_consensus.cc:2396] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5761f1165461459f9aaa8cd5d8e9b0b9 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.412091 14219 leader_election.cc:288] T a478c28e5d4646fdb956f5b65e5c6c94 P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 4 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:15:14.412942 16069 raft_consensus.cc:686] T d7ebe785e08640cb866350eae5797e9f P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 LEADER]: Becoming Leader. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Running, Role: LEADER
I20240503 08:15:14.413053 14219 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "769c4490062b4206951af2f4b9a6da05"
dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
from {username='slave'} at 127.0.0.1:58176
I20240503 08:15:14.413226 14219 raft_consensus.cc:482] T 769c4490062b4206951af2f4b9a6da05 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.413324 14219 raft_consensus.cc:3037] T 769c4490062b4206951af2f4b9a6da05 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.413486 14088 raft_consensus.cc:2454] T 70e8bfaa82464d3ab3338a779457dc02 P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5761f1165461459f9aaa8cd5d8e9b0b9 in term 3.
I20240503 08:15:14.413482 16069 consensus_queue.cc:237] T d7ebe785e08640cb866350eae5797e9f P 5761f1165461459f9aaa8cd5d8e9b0b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:14.414062 14156 leader_election.cc:302] T c7a958fc685e41fb86108dfcb9c6dc8b P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:14.414408 14156 leader_election.cc:302] T 70e8bfaa82464d3ab3338a779457dc02 P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, abb3df711d3a46809274b5cfbd4ce4e0; no voters: 296f3a209cf5447aaff2b8ea96a3bc4f
I20240503 08:15:14.415028 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a478c28e5d4646fdb956f5b65e5c6c94" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:14.415288 13960 raft_consensus.cc:3037] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.415884 14219 raft_consensus.cc:504] T 769c4490062b4206951af2f4b9a6da05 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:14.416152 13830 raft_consensus.cc:2396] T 79014e600505478883fc90159784ce6c P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5761f1165461459f9aaa8cd5d8e9b0b9 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.416800 13827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "769c4490062b4206951af2f4b9a6da05" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:14.416985 13827 raft_consensus.cc:3032] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:15:14.417086 13827 raft_consensus.cc:729] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:14.417070 13957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "769c4490062b4206951af2f4b9a6da05" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:14.417251 13957 raft_consensus.cc:3037] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.417325 13827 consensus_queue.cc:260] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:14.417620 13827 raft_consensus.cc:3037] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.418865 13960 raft_consensus.cc:2396] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5761f1165461459f9aaa8cd5d8e9b0b9 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.419750 13957 raft_consensus.cc:2454] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5761f1165461459f9aaa8cd5d8e9b0b9 in term 3.
I20240503 08:15:14.420253 14217 raft_consensus.cc:1261] T d69d33529e1140e5bc3a0a561012dfa1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Refusing update from remote peer 296f3a209cf5447aaff2b8ea96a3bc4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 08:15:14.420677 13827 raft_consensus.cc:2396] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5761f1165461459f9aaa8cd5d8e9b0b9 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.420859 15670 consensus_queue.cc:1035] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:15:14.420962 13828 raft_consensus.cc:2396] T c7a958fc685e41fb86108dfcb9c6dc8b P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5761f1165461459f9aaa8cd5d8e9b0b9 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.421170 16316 raft_consensus.cc:2781] T c7a958fc685e41fb86108dfcb9c6dc8b P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:15:14.421535 14156 leader_election.cc:302] T 79014e600505478883fc90159784ce6c P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, abb3df711d3a46809274b5cfbd4ce4e0; no voters: 5a198ad76e4f462c8be0859184aeb2fc
I20240503 08:15:14.422405 14153 leader_election.cc:302] T a478c28e5d4646fdb956f5b65e5c6c94 P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9; no voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5a198ad76e4f462c8be0859184aeb2fc
I20240503 08:15:14.427774 14219 leader_election.cc:288] T 769c4490062b4206951af2f4b9a6da05 P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 3 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:15:14.428552 16315 raft_consensus.cc:2781] T 70e8bfaa82464d3ab3338a779457dc02 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:15:14.434283 16321 raft_consensus.cc:2781] T 79014e600505478883fc90159784ce6c P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:15:14.434724 14153 leader_election.cc:302] T 769c4490062b4206951af2f4b9a6da05 P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5761f1165461459f9aaa8cd5d8e9b0b9; no voters: 5a198ad76e4f462c8be0859184aeb2fc
I20240503 08:15:14.435505 16320 raft_consensus.cc:2726] T a478c28e5d4646fdb956f5b65e5c6c94 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 08:15:14.438968 14090 raft_consensus.cc:1261] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Refusing update from remote peer 296f3a209cf5447aaff2b8ea96a3bc4f: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240503 08:15:14.439069 14090 pending_rounds.cc:76] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0: Aborting all ops after (but not including) 3
I20240503 08:15:14.439149 14090 pending_rounds.cc:97] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 08:15:14.439214 14090 raft_consensus.cc:2866] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:15:14.436003 14219 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f27cc9b78d084ac791de35b29e08a09a"
dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
from {username='slave'} at 127.0.0.1:58176
I20240503 08:15:14.439512 14219 raft_consensus.cc:482] T f27cc9b78d084ac791de35b29e08a09a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.439710 14219 raft_consensus.cc:3037] T f27cc9b78d084ac791de35b29e08a09a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.439898 16247 consensus_queue.cc:1035] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:15:14.442579 14219 raft_consensus.cc:504] T f27cc9b78d084ac791de35b29e08a09a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.443284 16320 raft_consensus.cc:2781] T 769c4490062b4206951af2f4b9a6da05 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:15:14.443794 16320 raft_consensus.cc:686] T 769c4490062b4206951af2f4b9a6da05 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 LEADER]: Becoming Leader. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Running, Role: LEADER
I20240503 08:15:14.444491 13957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f27cc9b78d084ac791de35b29e08a09a" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:14.444686 13957 raft_consensus.cc:3032] T f27cc9b78d084ac791de35b29e08a09a P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:15:14.444790 13957 raft_consensus.cc:729] T f27cc9b78d084ac791de35b29e08a09a P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:14.445003 13957 consensus_queue.cc:260] T f27cc9b78d084ac791de35b29e08a09a P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.445297 13957 raft_consensus.cc:3037] T f27cc9b78d084ac791de35b29e08a09a P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.448026 13957 raft_consensus.cc:2396] T f27cc9b78d084ac791de35b29e08a09a P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5761f1165461459f9aaa8cd5d8e9b0b9 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.449858 16320 consensus_queue.cc:237] T 769c4490062b4206951af2f4b9a6da05 P 5761f1165461459f9aaa8cd5d8e9b0b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:14.450353 16315 raft_consensus.cc:686] T 70e8bfaa82464d3ab3338a779457dc02 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 LEADER]: Becoming Leader. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Running, Role: LEADER
I20240503 08:15:14.450242 13689 catalog_manager.cc:5484] T d7ebe785e08640cb866350eae5797e9f P 5761f1165461459f9aaa8cd5d8e9b0b9 reported cstate change: term changed from 2 to 3, leader changed from 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65) to 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68). New cstate: current_term: 3 leader_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } }
I20240503 08:15:14.450862 16315 consensus_queue.cc:237] T 70e8bfaa82464d3ab3338a779457dc02 P 5761f1165461459f9aaa8cd5d8e9b0b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.454032 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f27cc9b78d084ac791de35b29e08a09a" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:14.454279 14090 raft_consensus.cc:3037] T f27cc9b78d084ac791de35b29e08a09a P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.457211 14090 raft_consensus.cc:2454] T f27cc9b78d084ac791de35b29e08a09a P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5761f1165461459f9aaa8cd5d8e9b0b9 in term 3.
I20240503 08:15:14.457670 14156 leader_election.cc:302] T f27cc9b78d084ac791de35b29e08a09a P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, abb3df711d3a46809274b5cfbd4ce4e0; no voters: 296f3a209cf5447aaff2b8ea96a3bc4f
I20240503 08:15:14.458838 16322 raft_consensus.cc:2781] T f27cc9b78d084ac791de35b29e08a09a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:15:14.459117 16322 raft_consensus.cc:686] T f27cc9b78d084ac791de35b29e08a09a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 LEADER]: Becoming Leader. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Running, Role: LEADER
I20240503 08:15:14.448043 16321 raft_consensus.cc:686] T 79014e600505478883fc90159784ce6c P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 LEADER]: Becoming Leader. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Running, Role: LEADER
I20240503 08:15:14.459565 16321 consensus_queue.cc:237] T 79014e600505478883fc90159784ce6c P 5761f1165461459f9aaa8cd5d8e9b0b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:14.449553 14219 leader_election.cc:288] T f27cc9b78d084ac791de35b29e08a09a P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 3 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:14.445827 16316 raft_consensus.cc:686] T c7a958fc685e41fb86108dfcb9c6dc8b P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 LEADER]: Becoming Leader. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Running, Role: LEADER
I20240503 08:15:14.460960 16316 consensus_queue.cc:237] T c7a958fc685e41fb86108dfcb9c6dc8b P 5761f1165461459f9aaa8cd5d8e9b0b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.459570 16322 consensus_queue.cc:237] T f27cc9b78d084ac791de35b29e08a09a P 5761f1165461459f9aaa8cd5d8e9b0b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.462460 14219 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "de418f9b6bb04763b88ccc3bbe6cd68f"
dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
from {username='slave'} at 127.0.0.1:58176
I20240503 08:15:14.462673 14219 raft_consensus.cc:482] T de418f9b6bb04763b88ccc3bbe6cd68f P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.462772 14219 raft_consensus.cc:3037] T de418f9b6bb04763b88ccc3bbe6cd68f P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.465543 14219 raft_consensus.cc:504] T de418f9b6bb04763b88ccc3bbe6cd68f P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.466800 14219 leader_election.cc:288] T de418f9b6bb04763b88ccc3bbe6cd68f P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 4 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:14.467294 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de418f9b6bb04763b88ccc3bbe6cd68f" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:14.467505 13830 raft_consensus.cc:3032] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:15:14.467593 13830 raft_consensus.cc:729] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:14.467851 13830 consensus_queue.cc:260] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.468070 13830 raft_consensus.cc:3037] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.469916 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de418f9b6bb04763b88ccc3bbe6cd68f" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:14.470111 14090 raft_consensus.cc:3037] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.472309 14090 raft_consensus.cc:2396] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5761f1165461459f9aaa8cd5d8e9b0b9 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.473453 13830 raft_consensus.cc:2396] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5761f1165461459f9aaa8cd5d8e9b0b9 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.473918 14154 leader_election.cc:302] T de418f9b6bb04763b88ccc3bbe6cd68f P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9; no voters: 5a198ad76e4f462c8be0859184aeb2fc, abb3df711d3a46809274b5cfbd4ce4e0
I20240503 08:15:14.474170 14219 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d69d33529e1140e5bc3a0a561012dfa1"
dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
from {username='slave'} at 127.0.0.1:58176
I20240503 08:15:14.474417 14219 raft_consensus.cc:482] T d69d33529e1140e5bc3a0a561012dfa1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.474594 14219 raft_consensus.cc:3037] T d69d33529e1140e5bc3a0a561012dfa1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.478273 14219 raft_consensus.cc:504] T d69d33529e1140e5bc3a0a561012dfa1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.479854 13957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d69d33529e1140e5bc3a0a561012dfa1" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:14.480083 13957 raft_consensus.cc:3032] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:15:14.480172 13957 raft_consensus.cc:729] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:14.480403 13957 consensus_queue.cc:260] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.480777 13957 raft_consensus.cc:3037] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.483155 13957 raft_consensus.cc:2454] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5761f1165461459f9aaa8cd5d8e9b0b9 in term 4.
I20240503 08:15:14.484448 14153 leader_election.cc:302] T d69d33529e1140e5bc3a0a561012dfa1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5761f1165461459f9aaa8cd5d8e9b0b9; no voters:
I20240503 08:15:14.484694 16316 raft_consensus.cc:2781] T d69d33529e1140e5bc3a0a561012dfa1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:15:14.484874 16316 raft_consensus.cc:686] T d69d33529e1140e5bc3a0a561012dfa1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 LEADER]: Becoming Leader. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Running, Role: LEADER
I20240503 08:15:14.485124 16316 consensus_queue.cc:237] T d69d33529e1140e5bc3a0a561012dfa1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.485376 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d69d33529e1140e5bc3a0a561012dfa1" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:14.485563 14090 raft_consensus.cc:3037] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.485837 13687 catalog_manager.cc:5484] T c7a958fc685e41fb86108dfcb9c6dc8b P 5761f1165461459f9aaa8cd5d8e9b0b9 reported cstate change: term changed from 2 to 3, leader changed from 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65) to 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68). New cstate: current_term: 3 leader_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:14.486351 13687 catalog_manager.cc:5484] T 769c4490062b4206951af2f4b9a6da05 P 5761f1165461459f9aaa8cd5d8e9b0b9 reported cstate change: term changed from 2 to 3, leader changed from 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65) to 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68). New cstate: current_term: 3 leader_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } }
I20240503 08:15:14.486858 13687 catalog_manager.cc:5484] T 79014e600505478883fc90159784ce6c P 5761f1165461459f9aaa8cd5d8e9b0b9 reported cstate change: term changed from 2 to 3, leader changed from 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65) to 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68). New cstate: current_term: 3 leader_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } }
I20240503 08:15:14.487233 13687 catalog_manager.cc:5484] T 70e8bfaa82464d3ab3338a779457dc02 P 5761f1165461459f9aaa8cd5d8e9b0b9 reported cstate change: term changed from 2 to 3, leader changed from 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66) to 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68). New cstate: current_term: 3 leader_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } }
I20240503 08:15:14.487668 14090 raft_consensus.cc:2454] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5761f1165461459f9aaa8cd5d8e9b0b9 in term 4.
I20240503 08:15:14.488026 14219 leader_election.cc:288] T d69d33529e1140e5bc3a0a561012dfa1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 4 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:14.474182 16315 raft_consensus.cc:2726] T de418f9b6bb04763b88ccc3bbe6cd68f P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 08:15:14.489434 14219 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d09e60e9c309425da9d27ae1153e4510"
dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
from {username='slave'} at 127.0.0.1:58176
I20240503 08:15:14.489681 14219 raft_consensus.cc:482] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.489856 14219 raft_consensus.cc:3037] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.493078 14219 raft_consensus.cc:504] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.494364 13957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d09e60e9c309425da9d27ae1153e4510" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:14.494522 13957 raft_consensus.cc:3032] T d09e60e9c309425da9d27ae1153e4510 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:15:14.494617 13957 raft_consensus.cc:729] T d09e60e9c309425da9d27ae1153e4510 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:14.494863 13957 consensus_queue.cc:260] T d09e60e9c309425da9d27ae1153e4510 P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.495146 13957 raft_consensus.cc:3037] T d09e60e9c309425da9d27ae1153e4510 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.497347 13957 raft_consensus.cc:2454] T d09e60e9c309425da9d27ae1153e4510 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5761f1165461459f9aaa8cd5d8e9b0b9 in term 4.
I20240503 08:15:14.498566 14217 raft_consensus.cc:1261] T c310dd5906a24299b6f62201b30b86db P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Refusing update from remote peer 296f3a209cf5447aaff2b8ea96a3bc4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:15:14.499289 16247 consensus_queue.cc:1035] T c310dd5906a24299b6f62201b30b86db P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:15:14.500653 14153 leader_election.cc:302] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5761f1165461459f9aaa8cd5d8e9b0b9; no voters:
I20240503 08:15:14.500972 16316 raft_consensus.cc:2781] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:15:14.501111 16316 raft_consensus.cc:686] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 LEADER]: Becoming Leader. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Running, Role: LEADER
I20240503 08:15:14.501353 16316 consensus_queue.cc:237] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.503239 14090 raft_consensus.cc:1261] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Refusing update from remote peer 296f3a209cf5447aaff2b8ea96a3bc4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:15:14.503836 16247 consensus_queue.cc:1035] T c310dd5906a24299b6f62201b30b86db P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:15:14.506783 14219 leader_election.cc:288] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 4 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:14.507579 14088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d09e60e9c309425da9d27ae1153e4510" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:14.507794 14088 raft_consensus.cc:3037] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.508402 14219 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4b680592b1774f09ba79789afa55a0f1"
dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
from {username='slave'} at 127.0.0.1:58176
I20240503 08:15:14.508569 14219 raft_consensus.cc:482] T 4b680592b1774f09ba79789afa55a0f1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.508697 14219 raft_consensus.cc:3037] T 4b680592b1774f09ba79789afa55a0f1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.510434 14088 raft_consensus.cc:2454] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5761f1165461459f9aaa8cd5d8e9b0b9 in term 4.
I20240503 08:15:14.511253 14219 raft_consensus.cc:504] T 4b680592b1774f09ba79789afa55a0f1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:14.511919 14219 leader_election.cc:288] T 4b680592b1774f09ba79789afa55a0f1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 4 election: Requested vote from peers abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:14.512382 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b680592b1774f09ba79789afa55a0f1" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:14.512621 13830 raft_consensus.cc:3032] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:15:14.512714 13830 raft_consensus.cc:729] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:14.512991 13830 consensus_queue.cc:260] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:14.513267 13830 raft_consensus.cc:3037] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.514751 14088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b680592b1774f09ba79789afa55a0f1" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:14.514920 14088 raft_consensus.cc:3037] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.516016 13830 raft_consensus.cc:2396] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5761f1165461459f9aaa8cd5d8e9b0b9 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.518206 14219 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "377cd840a48f47238bdaa5ad2257e1f4"
dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
from {username='slave'} at 127.0.0.1:58176
I20240503 08:15:14.518400 14219 raft_consensus.cc:482] T 377cd840a48f47238bdaa5ad2257e1f4 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.518504 14219 raft_consensus.cc:3037] T 377cd840a48f47238bdaa5ad2257e1f4 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.518788 14088 raft_consensus.cc:2396] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5761f1165461459f9aaa8cd5d8e9b0b9 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.519191 14156 leader_election.cc:302] T 4b680592b1774f09ba79789afa55a0f1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9; no voters: 5a198ad76e4f462c8be0859184aeb2fc, abb3df711d3a46809274b5cfbd4ce4e0
I20240503 08:15:14.519485 16315 raft_consensus.cc:2726] T 4b680592b1774f09ba79789afa55a0f1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 08:15:14.521493 14219 raft_consensus.cc:504] T 377cd840a48f47238bdaa5ad2257e1f4 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.522490 13957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377cd840a48f47238bdaa5ad2257e1f4" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:14.522469 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377cd840a48f47238bdaa5ad2257e1f4" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:14.522675 13957 raft_consensus.cc:3037] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.522701 13830 raft_consensus.cc:3032] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:15:14.522768 13830 raft_consensus.cc:729] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:14.521844 13689 catalog_manager.cc:5484] T d69d33529e1140e5bc3a0a561012dfa1 P 5761f1165461459f9aaa8cd5d8e9b0b9 reported cstate change: term changed from 3 to 4, leader changed from 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66) to 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68). New cstate: current_term: 4 leader_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } }
I20240503 08:15:14.523367 13689 catalog_manager.cc:5484] T f27cc9b78d084ac791de35b29e08a09a P 5761f1165461459f9aaa8cd5d8e9b0b9 reported cstate change: term changed from 2 to 3, leader changed from 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66) to 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68). New cstate: current_term: 3 leader_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } }
I20240503 08:15:14.524386 13830 consensus_queue.cc:260] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.524674 13830 raft_consensus.cc:3037] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.525297 13957 raft_consensus.cc:2396] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5761f1165461459f9aaa8cd5d8e9b0b9 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.525800 14219 leader_election.cc:288] T 377cd840a48f47238bdaa5ad2257e1f4 P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 4 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:15:14.526669 14219 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "06e7243793e9467daeed79f3b1f5be5a"
dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
from {username='slave'} at 127.0.0.1:58176
I20240503 08:15:14.526839 14219 raft_consensus.cc:482] T 06e7243793e9467daeed79f3b1f5be5a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.526958 14219 raft_consensus.cc:3037] T 06e7243793e9467daeed79f3b1f5be5a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.529369 14219 raft_consensus.cc:504] T 06e7243793e9467daeed79f3b1f5be5a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.530517 14088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e7243793e9467daeed79f3b1f5be5a" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:14.530728 14088 raft_consensus.cc:3037] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.531423 13687 catalog_manager.cc:5484] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 reported cstate change: term changed from 3 to 4, leader changed from 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66) to 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68). New cstate: current_term: 4 leader_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:14.532773 13828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e7243793e9467daeed79f3b1f5be5a" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:14.532943 13828 raft_consensus.cc:3032] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:15:14.533026 13828 raft_consensus.cc:729] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:14.533223 14088 raft_consensus.cc:2454] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5761f1165461459f9aaa8cd5d8e9b0b9 in term 4.
I20240503 08:15:14.533289 13828 consensus_queue.cc:260] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.533571 13828 raft_consensus.cc:3037] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.533690 14156 leader_election.cc:302] T 06e7243793e9467daeed79f3b1f5be5a P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:14.533947 14219 leader_election.cc:288] T 06e7243793e9467daeed79f3b1f5be5a P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 4 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:14.535203 14219 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "150f0d2d17bc4a59a8d82982790b4594"
dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
from {username='slave'} at 127.0.0.1:58176
I20240503 08:15:14.535374 14219 raft_consensus.cc:482] T 150f0d2d17bc4a59a8d82982790b4594 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.535529 14219 raft_consensus.cc:3037] T 150f0d2d17bc4a59a8d82982790b4594 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.536216 13828 raft_consensus.cc:2396] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5761f1165461459f9aaa8cd5d8e9b0b9 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 08:15:14.536545 16322 raft_consensus.cc:2781] T 06e7243793e9467daeed79f3b1f5be5a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:15:14.536744 16322 raft_consensus.cc:686] T 06e7243793e9467daeed79f3b1f5be5a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 LEADER]: Becoming Leader. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Running, Role: LEADER
I20240503 08:15:14.537024 16322 consensus_queue.cc:237] T 06e7243793e9467daeed79f3b1f5be5a P 5761f1165461459f9aaa8cd5d8e9b0b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.538162 14219 raft_consensus.cc:504] T 150f0d2d17bc4a59a8d82982790b4594 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.538798 14219 leader_election.cc:288] T 150f0d2d17bc4a59a8d82982790b4594 P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 3 election: Requested vote from peers abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:15:14.539106 14088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150f0d2d17bc4a59a8d82982790b4594" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:14.539268 14088 raft_consensus.cc:3037] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.539858 13687 catalog_manager.cc:5484] T 06e7243793e9467daeed79f3b1f5be5a P 5761f1165461459f9aaa8cd5d8e9b0b9 reported cstate change: term changed from 3 to 4, leader changed from 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65) to 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68). New cstate: current_term: 4 leader_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:14.541721 14088 raft_consensus.cc:2454] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5761f1165461459f9aaa8cd5d8e9b0b9 in term 3.
I20240503 08:15:14.542500 13957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150f0d2d17bc4a59a8d82982790b4594" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:14.542704 13957 raft_consensus.cc:3032] T 150f0d2d17bc4a59a8d82982790b4594 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:15:14.542791 13957 raft_consensus.cc:729] T 150f0d2d17bc4a59a8d82982790b4594 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:14.543056 13957 consensus_queue.cc:260] T 150f0d2d17bc4a59a8d82982790b4594 P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.543323 13957 raft_consensus.cc:3037] T 150f0d2d17bc4a59a8d82982790b4594 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.545718 13830 raft_consensus.cc:2396] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5761f1165461459f9aaa8cd5d8e9b0b9 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.545780 13957 raft_consensus.cc:2396] T 150f0d2d17bc4a59a8d82982790b4594 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5761f1165461459f9aaa8cd5d8e9b0b9 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.546094 14156 leader_election.cc:302] T 150f0d2d17bc4a59a8d82982790b4594 P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:14.546165 14154 leader_election.cc:302] T 377cd840a48f47238bdaa5ad2257e1f4 P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9; no voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5a198ad76e4f462c8be0859184aeb2fc
I20240503 08:15:14.546510 16315 raft_consensus.cc:2781] T 150f0d2d17bc4a59a8d82982790b4594 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:15:14.546663 16315 raft_consensus.cc:686] T 150f0d2d17bc4a59a8d82982790b4594 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 LEADER]: Becoming Leader. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Running, Role: LEADER
I20240503 08:15:14.546715 16316 raft_consensus.cc:2726] T 377cd840a48f47238bdaa5ad2257e1f4 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 08:15:14.546885 16315 consensus_queue.cc:237] T 150f0d2d17bc4a59a8d82982790b4594 P 5761f1165461459f9aaa8cd5d8e9b0b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.547109 14219 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9cae16af5a034cc29b8bb410a758c332"
dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
from {username='slave'} at 127.0.0.1:58176
I20240503 08:15:14.547273 14219 raft_consensus.cc:482] T 9cae16af5a034cc29b8bb410a758c332 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.547396 14219 raft_consensus.cc:3037] T 9cae16af5a034cc29b8bb410a758c332 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.550695 13689 catalog_manager.cc:5484] T 150f0d2d17bc4a59a8d82982790b4594 P 5761f1165461459f9aaa8cd5d8e9b0b9 reported cstate change: term changed from 2 to 3, leader changed from 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66) to 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68). New cstate: current_term: 3 leader_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } }
I20240503 08:15:14.551489 14219 raft_consensus.cc:504] T 9cae16af5a034cc29b8bb410a758c332 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:14.552336 14088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cae16af5a034cc29b8bb410a758c332" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:14.552549 14088 raft_consensus.cc:3037] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.554962 14088 raft_consensus.cc:2454] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5761f1165461459f9aaa8cd5d8e9b0b9 in term 3.
I20240503 08:15:14.555536 14156 leader_election.cc:302] T 9cae16af5a034cc29b8bb410a758c332 P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:14.555835 16315 raft_consensus.cc:2781] T 9cae16af5a034cc29b8bb410a758c332 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:15:14.555979 16315 raft_consensus.cc:686] T 9cae16af5a034cc29b8bb410a758c332 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 LEADER]: Becoming Leader. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Running, Role: LEADER
I20240503 08:15:14.556218 16315 consensus_queue.cc:237] T 9cae16af5a034cc29b8bb410a758c332 P 5761f1165461459f9aaa8cd5d8e9b0b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:14.557376 13957 raft_consensus.cc:1261] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Refusing update from remote peer 5a198ad76e4f462c8be0859184aeb2fc: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240503 08:15:14.557507 13957 pending_rounds.cc:76] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f: Aborting all ops after (but not including) 3
I20240503 08:15:14.557582 13957 pending_rounds.cc:97] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 08:15:14.557723 13957 raft_consensus.cc:2866] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:15:14.558422 16292 consensus_queue.cc:1035] T 8bb1779c7ac64d599cde0665505b3dba P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:15:14.562729 14088 raft_consensus.cc:1261] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Refusing update from remote peer 5a198ad76e4f462c8be0859184aeb2fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 08:15:14.564319 15593 consensus_queue.cc:1035] T 8bb1779c7ac64d599cde0665505b3dba P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:15:14.569077 14219 leader_election.cc:288] T 9cae16af5a034cc29b8bb410a758c332 P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 3 election: Requested vote from peers abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:14.569275 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cae16af5a034cc29b8bb410a758c332" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:14.569478 13830 raft_consensus.cc:3032] T 9cae16af5a034cc29b8bb410a758c332 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:15:14.569782 13830 raft_consensus.cc:729] T 9cae16af5a034cc29b8bb410a758c332 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:14.570083 13830 consensus_queue.cc:260] T 9cae16af5a034cc29b8bb410a758c332 P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:14.570331 13830 raft_consensus.cc:3037] T 9cae16af5a034cc29b8bb410a758c332 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.571465 13681 catalog_manager.cc:5484] T 9cae16af5a034cc29b8bb410a758c332 P 5761f1165461459f9aaa8cd5d8e9b0b9 reported cstate change: term changed from 2 to 3, leader changed from 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65) to 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68). New cstate: current_term: 3 leader_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } }
I20240503 08:15:14.571782 14219 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f1e94dd86fc74c549979d30f0f705da7"
dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
from {username='slave'} at 127.0.0.1:58176
I20240503 08:15:14.572006 14219 raft_consensus.cc:482] T f1e94dd86fc74c549979d30f0f705da7 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.572109 14219 raft_consensus.cc:3037] T f1e94dd86fc74c549979d30f0f705da7 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.575388 13830 raft_consensus.cc:2396] T 9cae16af5a034cc29b8bb410a758c332 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5761f1165461459f9aaa8cd5d8e9b0b9 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.576114 14219 raft_consensus.cc:504] T f1e94dd86fc74c549979d30f0f705da7 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.577315 13957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e94dd86fc74c549979d30f0f705da7" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:14.577553 13957 raft_consensus.cc:3037] T f1e94dd86fc74c549979d30f0f705da7 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.577636 14219 leader_election.cc:288] T f1e94dd86fc74c549979d30f0f705da7 P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 4 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:14.578176 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e94dd86fc74c549979d30f0f705da7" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:14.578364 13830 raft_consensus.cc:3032] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:15:14.578436 13830 raft_consensus.cc:729] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:14.578680 13830 consensus_queue.cc:260] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.578907 13830 raft_consensus.cc:3037] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.578826 14219 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c310dd5906a24299b6f62201b30b86db"
dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
from {username='slave'} at 127.0.0.1:58176
I20240503 08:15:14.579046 14219 raft_consensus.cc:482] T c310dd5906a24299b6f62201b30b86db P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.579190 14219 raft_consensus.cc:3037] T c310dd5906a24299b6f62201b30b86db P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.580157 13957 raft_consensus.cc:2396] T f1e94dd86fc74c549979d30f0f705da7 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5761f1165461459f9aaa8cd5d8e9b0b9 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.580814 13830 raft_consensus.cc:2396] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5761f1165461459f9aaa8cd5d8e9b0b9 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.581205 14154 leader_election.cc:302] T f1e94dd86fc74c549979d30f0f705da7 P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9; no voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5a198ad76e4f462c8be0859184aeb2fc
I20240503 08:15:14.581611 16315 raft_consensus.cc:2726] T f1e94dd86fc74c549979d30f0f705da7 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 08:15:14.581704 14219 raft_consensus.cc:504] T c310dd5906a24299b6f62201b30b86db P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.582847 13957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c310dd5906a24299b6f62201b30b86db" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:14.583026 13957 raft_consensus.cc:3032] T c310dd5906a24299b6f62201b30b86db P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:15:14.582991 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c310dd5906a24299b6f62201b30b86db" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:14.583119 13957 raft_consensus.cc:729] T c310dd5906a24299b6f62201b30b86db P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:14.583207 14090 raft_consensus.cc:3037] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.583359 13957 consensus_queue.cc:260] T c310dd5906a24299b6f62201b30b86db P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.583722 13957 raft_consensus.cc:3037] T c310dd5906a24299b6f62201b30b86db P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.585896 14090 raft_consensus.cc:2454] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5761f1165461459f9aaa8cd5d8e9b0b9 in term 3.
I20240503 08:15:14.586179 13957 raft_consensus.cc:2454] T c310dd5906a24299b6f62201b30b86db P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5761f1165461459f9aaa8cd5d8e9b0b9 in term 3.
I20240503 08:15:14.586349 14156 leader_election.cc:302] T c310dd5906a24299b6f62201b30b86db P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:14.586644 16315 raft_consensus.cc:2781] T c310dd5906a24299b6f62201b30b86db P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:15:14.586771 16315 raft_consensus.cc:686] T c310dd5906a24299b6f62201b30b86db P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 LEADER]: Becoming Leader. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Running, Role: LEADER
I20240503 08:15:14.587045 16315 consensus_queue.cc:237] T c310dd5906a24299b6f62201b30b86db P 5761f1165461459f9aaa8cd5d8e9b0b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.588439 14219 leader_election.cc:288] T c310dd5906a24299b6f62201b30b86db P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 3 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:14.589694 14217 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "84204cbcda32446ab708717e9633af69"
dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
from {username='slave'} at 127.0.0.1:58176
I20240503 08:15:14.589877 14217 raft_consensus.cc:482] T 84204cbcda32446ab708717e9633af69 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.589984 14217 raft_consensus.cc:3037] T 84204cbcda32446ab708717e9633af69 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.590132 13686 catalog_manager.cc:5484] T c310dd5906a24299b6f62201b30b86db P 5761f1165461459f9aaa8cd5d8e9b0b9 reported cstate change: term changed from 2 to 3, leader changed from 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66) to 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68). New cstate: current_term: 3 leader_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } }
I20240503 08:15:14.592294 14217 raft_consensus.cc:504] T 84204cbcda32446ab708717e9633af69 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:14.592901 14217 leader_election.cc:288] T 84204cbcda32446ab708717e9633af69 P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 3 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:15:14.593570 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84204cbcda32446ab708717e9633af69" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:14.593672 13957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84204cbcda32446ab708717e9633af69" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:14.593819 13830 raft_consensus.cc:3032] T 84204cbcda32446ab708717e9633af69 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:15:14.593850 13957 raft_consensus.cc:3037] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.593909 13830 raft_consensus.cc:729] T 84204cbcda32446ab708717e9633af69 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:14.593933 14219 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e82cb19fece24181b7607d9447ba1eb2"
dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
from {username='slave'} at 127.0.0.1:58176
I20240503 08:15:14.594192 14219 raft_consensus.cc:482] T e82cb19fece24181b7607d9447ba1eb2 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.594166 13830 consensus_queue.cc:260] T 84204cbcda32446ab708717e9633af69 P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:14.594420 14219 raft_consensus.cc:3037] T e82cb19fece24181b7607d9447ba1eb2 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.594446 13830 raft_consensus.cc:3037] T 84204cbcda32446ab708717e9633af69 P 5a198ad76e4f462c8be0859184aeb2fc [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:14.596398 13957 raft_consensus.cc:2454] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5761f1165461459f9aaa8cd5d8e9b0b9 in term 3.
I20240503 08:15:14.596856 14153 leader_election.cc:302] T 84204cbcda32446ab708717e9633af69 P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5761f1165461459f9aaa8cd5d8e9b0b9; no voters:
I20240503 08:15:14.596980 13830 raft_consensus.cc:2396] T 84204cbcda32446ab708717e9633af69 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5761f1165461459f9aaa8cd5d8e9b0b9 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.597146 16322 raft_consensus.cc:2781] T 84204cbcda32446ab708717e9633af69 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:15:14.597275 16322 raft_consensus.cc:686] T 84204cbcda32446ab708717e9633af69 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 LEADER]: Becoming Leader. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Running, Role: LEADER
I20240503 08:15:14.597537 16322 consensus_queue.cc:237] T 84204cbcda32446ab708717e9633af69 P 5761f1165461459f9aaa8cd5d8e9b0b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:14.603796 14219 raft_consensus.cc:504] T e82cb19fece24181b7607d9447ba1eb2 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.604405 14219 leader_election.cc:288] T e82cb19fece24181b7607d9447ba1eb2 P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 4 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:14.604962 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82cb19fece24181b7607d9447ba1eb2" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:14.605206 14090 raft_consensus.cc:3037] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.605926 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82cb19fece24181b7607d9447ba1eb2" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:14.606179 13830 raft_consensus.cc:3032] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:15:14.606292 13830 raft_consensus.cc:729] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:14.606616 13830 consensus_queue.cc:260] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.606992 13830 raft_consensus.cc:3037] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.609146 14090 raft_consensus.cc:2454] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5761f1165461459f9aaa8cd5d8e9b0b9 in term 4.
I20240503 08:15:14.609586 14156 leader_election.cc:302] T e82cb19fece24181b7607d9447ba1eb2 P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:14.609886 16315 raft_consensus.cc:2781] T e82cb19fece24181b7607d9447ba1eb2 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:15:14.610050 13830 raft_consensus.cc:2396] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5761f1165461459f9aaa8cd5d8e9b0b9 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 08:15:14.610042 16315 raft_consensus.cc:686] T e82cb19fece24181b7607d9447ba1eb2 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 LEADER]: Becoming Leader. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Running, Role: LEADER
I20240503 08:15:14.610276 16315 consensus_queue.cc:237] T e82cb19fece24181b7607d9447ba1eb2 P 5761f1165461459f9aaa8cd5d8e9b0b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.613627 13957 raft_consensus.cc:1261] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Refusing update from remote peer 5a198ad76e4f462c8be0859184aeb2fc: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240503 08:15:14.613740 13957 pending_rounds.cc:76] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f: Aborting all ops after (but not including) 3
I20240503 08:15:14.613811 13957 pending_rounds.cc:97] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 08:15:14.613904 13957 raft_consensus.cc:2866] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:15:14.614497 15593 consensus_queue.cc:1035] T c244226a1c40488f949735420bfacd0b P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:15:14.618443 14090 raft_consensus.cc:1261] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Refusing update from remote peer 5a198ad76e4f462c8be0859184aeb2fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 08:15:14.619004 16215 consensus_queue.cc:1035] T c244226a1c40488f949735420bfacd0b P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:15:14.618275 13689 catalog_manager.cc:5484] T 84204cbcda32446ab708717e9633af69 P 5761f1165461459f9aaa8cd5d8e9b0b9 reported cstate change: term changed from 2 to 3, leader changed from 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65) to 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68). New cstate: current_term: 3 leader_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } }
I20240503 08:15:14.624501 13689 catalog_manager.cc:5484] T e82cb19fece24181b7607d9447ba1eb2 P 5761f1165461459f9aaa8cd5d8e9b0b9 reported cstate change: term changed from 3 to 4, leader changed from 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65) to 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68). New cstate: current_term: 4 leader_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:14.630343 15737 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:14.637679 14088 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "045fa6c60d334659aab79f7ba4999661"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:14.637972 14088 raft_consensus.cc:482] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.638159 14088 raft_consensus.cc:3037] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:14.641297 14088 raft_consensus.cc:504] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.642449 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "045fa6c60d334659aab79f7ba4999661" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:14.642756 13830 raft_consensus.cc:3037] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:14.650002 13939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:14.653988 13830 raft_consensus.cc:2396] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate abb3df711d3a46809274b5cfbd4ce4e0 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 08:15:14.655133 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "045fa6c60d334659aab79f7ba4999661" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:14.655328 14219 raft_consensus.cc:3037] T 045fa6c60d334659aab79f7ba4999661 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:14.658246 16196 heartbeater.cc:503] Master 127.20.230.126:38135 was elected leader, sending a full tablet report...
I20240503 08:15:14.658912 16000 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:14.657990 14219 raft_consensus.cc:2454] T 045fa6c60d334659aab79f7ba4999661 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 5.
I20240503 08:15:14.660692 14025 leader_election.cc:302] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, abb3df711d3a46809274b5cfbd4ce4e0; no voters: 5a198ad76e4f462c8be0859184aeb2fc
I20240503 08:15:14.661015 16210 raft_consensus.cc:2781] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:15:14.661172 16210 raft_consensus.cc:686] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 LEADER]: Becoming Leader. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:14.661463 16210 consensus_queue.cc:237] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.664381 13689 catalog_manager.cc:5484] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 reported cstate change: term changed from 3 to 5, leader changed from 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65) to abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67). New cstate: current_term: 5 leader_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:14.664927 14088 leader_election.cc:288] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 5 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:14.665958 14090 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dd3d6353358d4b02af928cd9f9410aca"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:14.666142 14090 raft_consensus.cc:482] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.666234 14090 raft_consensus.cc:3037] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:14.668747 14090 raft_consensus.cc:504] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.669669 14198 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:15:14.671168 13957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd3d6353358d4b02af928cd9f9410aca" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:14.671386 13957 raft_consensus.cc:3037] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:14.674088 13957 raft_consensus.cc:2396] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate abb3df711d3a46809274b5cfbd4ce4e0 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 08:15:14.674508 14090 leader_election.cc:288] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 5 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:15:14.675438 14090 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c7a958fc685e41fb86108dfcb9c6dc8b"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:14.675611 14090 raft_consensus.cc:482] T c7a958fc685e41fb86108dfcb9c6dc8b P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.675734 14090 raft_consensus.cc:3037] T c7a958fc685e41fb86108dfcb9c6dc8b P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.678272 14090 raft_consensus.cc:504] T c7a958fc685e41fb86108dfcb9c6dc8b P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.679780 13828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a958fc685e41fb86108dfcb9c6dc8b" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:14.679977 13828 raft_consensus.cc:3037] T c7a958fc685e41fb86108dfcb9c6dc8b P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.681030 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a958fc685e41fb86108dfcb9c6dc8b" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:14.681236 14219 raft_consensus.cc:3032] T c7a958fc685e41fb86108dfcb9c6dc8b P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:15:14.681313 14219 raft_consensus.cc:729] T c7a958fc685e41fb86108dfcb9c6dc8b P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Running, Role: LEADER
I20240503 08:15:14.682658 13828 raft_consensus.cc:2396] T c7a958fc685e41fb86108dfcb9c6dc8b P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate abb3df711d3a46809274b5cfbd4ce4e0 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.687068 14090 leader_election.cc:288] T c7a958fc685e41fb86108dfcb9c6dc8b P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 4 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:14.688144 14089 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "70e8bfaa82464d3ab3338a779457dc02"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:14.688313 14089 raft_consensus.cc:482] T 70e8bfaa82464d3ab3338a779457dc02 P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.688421 14089 raft_consensus.cc:3037] T 70e8bfaa82464d3ab3338a779457dc02 P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.688345 14219 consensus_queue.cc:260] T c7a958fc685e41fb86108dfcb9c6dc8b P 5761f1165461459f9aaa8cd5d8e9b0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.688645 14219 raft_consensus.cc:3037] T c7a958fc685e41fb86108dfcb9c6dc8b P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.690975 14089 raft_consensus.cc:504] T 70e8bfaa82464d3ab3338a779457dc02 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.691423 14219 raft_consensus.cc:2396] T c7a958fc685e41fb86108dfcb9c6dc8b P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate abb3df711d3a46809274b5cfbd4ce4e0 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.691972 14217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70e8bfaa82464d3ab3338a779457dc02" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:14.692168 14217 raft_consensus.cc:3032] T 70e8bfaa82464d3ab3338a779457dc02 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:15:14.692253 14217 raft_consensus.cc:729] T 70e8bfaa82464d3ab3338a779457dc02 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Running, Role: LEADER
I20240503 08:15:14.692555 14089 leader_election.cc:288] T 70e8bfaa82464d3ab3338a779457dc02 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 4 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:14.692512 14217 consensus_queue.cc:260] T 70e8bfaa82464d3ab3338a779457dc02 P 5761f1165461459f9aaa8cd5d8e9b0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.692787 14217 raft_consensus.cc:3037] T 70e8bfaa82464d3ab3338a779457dc02 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.693455 14089 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f27cc9b78d084ac791de35b29e08a09a"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:14.693622 14089 raft_consensus.cc:482] T f27cc9b78d084ac791de35b29e08a09a P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.693713 14089 raft_consensus.cc:3037] T f27cc9b78d084ac791de35b29e08a09a P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.695602 14217 raft_consensus.cc:2396] T 70e8bfaa82464d3ab3338a779457dc02 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate abb3df711d3a46809274b5cfbd4ce4e0 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.696094 14089 raft_consensus.cc:504] T f27cc9b78d084ac791de35b29e08a09a P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.697104 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f27cc9b78d084ac791de35b29e08a09a" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:14.697345 13960 raft_consensus.cc:3037] T f27cc9b78d084ac791de35b29e08a09a P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.697467 14217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f27cc9b78d084ac791de35b29e08a09a" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:14.697675 14217 raft_consensus.cc:3032] T f27cc9b78d084ac791de35b29e08a09a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:15:14.697748 14217 raft_consensus.cc:729] T f27cc9b78d084ac791de35b29e08a09a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Running, Role: LEADER
I20240503 08:15:14.698089 14217 consensus_queue.cc:260] T f27cc9b78d084ac791de35b29e08a09a P 5761f1165461459f9aaa8cd5d8e9b0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.698354 14217 raft_consensus.cc:3037] T f27cc9b78d084ac791de35b29e08a09a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.700304 13960 raft_consensus.cc:2396] T f27cc9b78d084ac791de35b29e08a09a P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate abb3df711d3a46809274b5cfbd4ce4e0 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.701059 14217 raft_consensus.cc:2396] T f27cc9b78d084ac791de35b29e08a09a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate abb3df711d3a46809274b5cfbd4ce4e0 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.701458 14025 leader_election.cc:302] T f27cc9b78d084ac791de35b29e08a09a P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: abb3df711d3a46809274b5cfbd4ce4e0; no voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5761f1165461459f9aaa8cd5d8e9b0b9
I20240503 08:15:14.701746 15937 raft_consensus.cc:2726] T f27cc9b78d084ac791de35b29e08a09a P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 08:15:14.702165 13957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70e8bfaa82464d3ab3338a779457dc02" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:14.702432 13957 raft_consensus.cc:3037] T 70e8bfaa82464d3ab3338a779457dc02 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.705277 13957 raft_consensus.cc:2396] T 70e8bfaa82464d3ab3338a779457dc02 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate abb3df711d3a46809274b5cfbd4ce4e0 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.705418 14089 leader_election.cc:288] T f27cc9b78d084ac791de35b29e08a09a P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 4 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:14.706696 14090 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c244226a1c40488f949735420bfacd0b"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:14.706908 14090 raft_consensus.cc:482] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.707005 14090 raft_consensus.cc:3037] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.709793 14090 raft_consensus.cc:504] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.710850 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c244226a1c40488f949735420bfacd0b" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:14.717695 13960 raft_consensus.cc:3037] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.719875 14090 leader_election.cc:288] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 4 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:14.720341 13960 raft_consensus.cc:2454] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 4.
I20240503 08:15:14.726106 14023 leader_election.cc:302] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:14.726382 15937 raft_consensus.cc:2781] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:15:14.726522 15937 raft_consensus.cc:686] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 LEADER]: Becoming Leader. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:14.726783 15937 consensus_queue.cc:237] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.729676 13689 catalog_manager.cc:5484] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 reported cstate change: term changed from 3 to 4, leader changed from 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65) to abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67). New cstate: current_term: 4 leader_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:14.734452 14023 leader_election.cc:302] T 70e8bfaa82464d3ab3338a779457dc02 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: abb3df711d3a46809274b5cfbd4ce4e0; no voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5761f1165461459f9aaa8cd5d8e9b0b9
I20240503 08:15:14.734918 16210 raft_consensus.cc:2726] T 70e8bfaa82464d3ab3338a779457dc02 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 08:15:14.717391 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd3d6353358d4b02af928cd9f9410aca" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:14.735456 13830 raft_consensus.cc:3037] T dd3d6353358d4b02af928cd9f9410aca P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:14.735427 14090 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "32e141d0103848c1a2d473429cffcca9"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:14.735610 14090 raft_consensus.cc:482] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.735743 14090 raft_consensus.cc:3037] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:14.747299 14090 raft_consensus.cc:504] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:14.749612 13830 raft_consensus.cc:2454] T dd3d6353358d4b02af928cd9f9410aca P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 5.
I20240503 08:15:14.752262 13828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32e141d0103848c1a2d473429cffcca9" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:14.752740 13828 raft_consensus.cc:3037] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:14.755741 13828 raft_consensus.cc:2454] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 5.
I20240503 08:15:14.756577 13957 raft_consensus.cc:1261] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Refusing update from remote peer 5a198ad76e4f462c8be0859184aeb2fc: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240503 08:15:14.756762 13957 pending_rounds.cc:76] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f: Aborting all ops after (but not including) 3
I20240503 08:15:14.756893 13957 pending_rounds.cc:97] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 08:15:14.757026 13957 raft_consensus.cc:2866] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:15:14.757791 15593 consensus_queue.cc:1035] T 666be8071a904e6cbff984fa293d00b5 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:15:14.759256 13957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32e141d0103848c1a2d473429cffcca9" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:14.759479 13957 raft_consensus.cc:3037] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:14.762054 14090 leader_election.cc:288] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 5 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:15:14.763268 14088 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "de418f9b6bb04763b88ccc3bbe6cd68f"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:14.763505 14088 raft_consensus.cc:482] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.763671 14088 raft_consensus.cc:3037] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:14.763996 14024 leader_election.cc:302] T c7a958fc685e41fb86108dfcb9c6dc8b P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: abb3df711d3a46809274b5cfbd4ce4e0; no voters: 5761f1165461459f9aaa8cd5d8e9b0b9, 5a198ad76e4f462c8be0859184aeb2fc
I20240503 08:15:14.764452 14024 leader_election.cc:302] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc, abb3df711d3a46809274b5cfbd4ce4e0; no voters: 296f3a209cf5447aaff2b8ea96a3bc4f
I20240503 08:15:14.764746 15937 raft_consensus.cc:2726] T c7a958fc685e41fb86108dfcb9c6dc8b P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 08:15:14.764870 14024 leader_election.cc:302] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:14.764930 15937 raft_consensus.cc:2781] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:15:14.765098 16210 raft_consensus.cc:2781] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:15:14.766613 13957 raft_consensus.cc:2396] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate abb3df711d3a46809274b5cfbd4ce4e0 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 08:15:14.771791 14089 raft_consensus.cc:1261] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Refusing update from remote peer 5a198ad76e4f462c8be0859184aeb2fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 08:15:14.772372 16215 consensus_queue.cc:1035] T 666be8071a904e6cbff984fa293d00b5 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:15:14.784102 14088 raft_consensus.cc:504] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.784772 14088 leader_election.cc:288] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 5 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:14.785221 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c244226a1c40488f949735420bfacd0b" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:14.785437 13830 raft_consensus.cc:3032] T c244226a1c40488f949735420bfacd0b P 5a198ad76e4f462c8be0859184aeb2fc [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:15:14.785523 13830 raft_consensus.cc:729] T c244226a1c40488f949735420bfacd0b P 5a198ad76e4f462c8be0859184aeb2fc [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:14.786548 14088 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d69d33529e1140e5bc3a0a561012dfa1"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:14.786768 14088 raft_consensus.cc:482] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.786671 14217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de418f9b6bb04763b88ccc3bbe6cd68f" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:14.786904 14088 raft_consensus.cc:3037] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:14.786983 14217 raft_consensus.cc:3037] T de418f9b6bb04763b88ccc3bbe6cd68f P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:14.789623 14217 raft_consensus.cc:2454] T de418f9b6bb04763b88ccc3bbe6cd68f P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 5.
I20240503 08:15:14.789722 14088 raft_consensus.cc:504] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.790673 13957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d69d33529e1140e5bc3a0a561012dfa1" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:14.790843 13957 raft_consensus.cc:3037] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:14.791115 14025 leader_election.cc:302] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:14.791391 16362 raft_consensus.cc:2781] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:15:14.791491 16362 raft_consensus.cc:686] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 LEADER]: Becoming Leader. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:14.791738 16362 consensus_queue.cc:237] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.793432 13957 raft_consensus.cc:2454] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 5.
I20240503 08:15:14.793901 14090 raft_consensus.cc:1261] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Refusing update from remote peer 5a198ad76e4f462c8be0859184aeb2fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 08:15:14.794032 14023 leader_election.cc:302] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:14.794423 15937 raft_consensus.cc:686] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 LEADER]: Becoming Leader. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:14.794541 16360 raft_consensus.cc:2781] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:15:14.794662 13830 consensus_queue.cc:260] T c244226a1c40488f949735420bfacd0b P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.794886 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d69d33529e1140e5bc3a0a561012dfa1" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:14.795017 13830 raft_consensus.cc:3037] T c244226a1c40488f949735420bfacd0b P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.794987 16215 consensus_queue.cc:1035] T d2ba977460694815a38d415a14e1cd5f P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:15:14.795101 14219 raft_consensus.cc:3032] T d69d33529e1140e5bc3a0a561012dfa1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:15:14.795178 14219 raft_consensus.cc:729] T d69d33529e1140e5bc3a0a561012dfa1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Running, Role: LEADER
I20240503 08:15:14.795462 14219 consensus_queue.cc:260] T d69d33529e1140e5bc3a0a561012dfa1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.795735 14219 raft_consensus.cc:3037] T d69d33529e1140e5bc3a0a561012dfa1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:14.795894 13828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de418f9b6bb04763b88ccc3bbe6cd68f" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:14.796308 13828 raft_consensus.cc:3037] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:14.797827 13830 raft_consensus.cc:2454] T c244226a1c40488f949735420bfacd0b P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 4.
I20240503 08:15:14.798499 14219 raft_consensus.cc:2396] T d69d33529e1140e5bc3a0a561012dfa1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate abb3df711d3a46809274b5cfbd4ce4e0 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:15:14.794462 14088 leader_election.cc:288] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 5 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:14.798159 16360 raft_consensus.cc:686] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 LEADER]: Becoming Leader. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:14.799049 16360 consensus_queue.cc:237] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.799618 14088 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d09e60e9c309425da9d27ae1153e4510"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:14.799887 14088 raft_consensus.cc:482] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.800056 14088 raft_consensus.cc:3037] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:14.794746 15937 consensus_queue.cc:237] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.802772 14088 raft_consensus.cc:504] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.803845 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d09e60e9c309425da9d27ae1153e4510" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:14.804021 14219 raft_consensus.cc:3032] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:15:14.804054 13828 raft_consensus.cc:2396] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate abb3df711d3a46809274b5cfbd4ce4e0 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 08:15:14.804091 14219 raft_consensus.cc:729] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Running, Role: LEADER
I20240503 08:15:14.804303 13957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d09e60e9c309425da9d27ae1153e4510" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:14.804333 14219 consensus_queue.cc:260] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.804509 13957 raft_consensus.cc:3037] T d09e60e9c309425da9d27ae1153e4510 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:14.804637 14219 raft_consensus.cc:3037] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:14.807288 13957 raft_consensus.cc:2454] T d09e60e9c309425da9d27ae1153e4510 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 5.
I20240503 08:15:14.807440 14219 raft_consensus.cc:2396] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate abb3df711d3a46809274b5cfbd4ce4e0 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:15:14.807440 16210 raft_consensus.cc:686] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 LEADER]: Becoming Leader. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:14.807682 16210 consensus_queue.cc:237] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:14.807801 14023 leader_election.cc:302] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:14.810168 16361 raft_consensus.cc:2781] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:15:14.810562 16361 raft_consensus.cc:686] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 LEADER]: Becoming Leader. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:14.810889 16361 consensus_queue.cc:237] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.812256 14088 leader_election.cc:288] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 5 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:15:14.814020 14088 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4b680592b1774f09ba79789afa55a0f1"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:14.814282 14088 raft_consensus.cc:482] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.814447 14088 raft_consensus.cc:3037] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:14.814632 13957 raft_consensus.cc:1261] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Refusing update from remote peer 5a198ad76e4f462c8be0859184aeb2fc: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240503 08:15:14.814736 13957 pending_rounds.cc:76] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f: Aborting all ops after (but not including) 3
I20240503 08:15:14.814811 13957 pending_rounds.cc:97] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 08:15:14.814877 13957 raft_consensus.cc:2866] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:15:14.815443 16215 consensus_queue.cc:1035] T d2ba977460694815a38d415a14e1cd5f P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:15:14.818964 14088 raft_consensus.cc:504] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:14.820238 14217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b680592b1774f09ba79789afa55a0f1" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:14.820493 14217 raft_consensus.cc:3037] T 4b680592b1774f09ba79789afa55a0f1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:14.821112 13828 raft_consensus.cc:1261] T d7ebe785e08640cb866350eae5797e9f P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Refusing update from remote peer 5761f1165461459f9aaa8cd5d8e9b0b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240503 08:15:14.821271 13828 pending_rounds.cc:76] T d7ebe785e08640cb866350eae5797e9f P 5a198ad76e4f462c8be0859184aeb2fc: Aborting all ops after (but not including) 3
I20240503 08:15:14.821391 13828 pending_rounds.cc:97] T d7ebe785e08640cb866350eae5797e9f P 5a198ad76e4f462c8be0859184aeb2fc: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 08:15:14.821583 13828 raft_consensus.cc:2866] T d7ebe785e08640cb866350eae5797e9f P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:15:14.822285 16322 consensus_queue.cc:1035] T d7ebe785e08640cb866350eae5797e9f P 5761f1165461459f9aaa8cd5d8e9b0b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:15:14.823380 14217 raft_consensus.cc:2454] T 4b680592b1774f09ba79789afa55a0f1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 5.
I20240503 08:15:14.824263 14088 leader_election.cc:288] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 5 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:14.825304 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b680592b1774f09ba79789afa55a0f1" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:14.825528 13830 raft_consensus.cc:3037] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:14.826150 14025 leader_election.cc:302] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:14.826467 16361 raft_consensus.cc:2781] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:15:14.826663 16361 raft_consensus.cc:686] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 LEADER]: Becoming Leader. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:14.826922 16361 consensus_queue.cc:237] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:14.829044 13957 raft_consensus.cc:1261] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Refusing update from remote peer 5761f1165461459f9aaa8cd5d8e9b0b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 08:15:14.829586 16316 consensus_queue.cc:1035] T d7ebe785e08640cb866350eae5797e9f P 5761f1165461459f9aaa8cd5d8e9b0b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:15:14.830654 13830 raft_consensus.cc:2396] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate abb3df711d3a46809274b5cfbd4ce4e0 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 08:15:14.833726 14088 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cdf9885a1e7c4b8dbb160e370ca0f60c"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:14.834340 14088 raft_consensus.cc:482] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.834939 14088 raft_consensus.cc:3037] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:14.839001 13689 catalog_manager.cc:5484] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0 reported cstate change: term changed from 3 to 5, leader changed from 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66) to abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67). New cstate: current_term: 5 leader_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:14.840811 14088 raft_consensus.cc:504] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:14.841894 13957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdf9885a1e7c4b8dbb160e370ca0f60c" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:14.842092 13957 raft_consensus.cc:3037] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:14.845444 13957 raft_consensus.cc:2396] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate abb3df711d3a46809274b5cfbd4ce4e0 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 08:15:14.846163 14088 leader_election.cc:288] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 5 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:14.848035 13828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdf9885a1e7c4b8dbb160e370ca0f60c" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:14.848304 13828 raft_consensus.cc:3037] T cdf9885a1e7c4b8dbb160e370ca0f60c P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:14.850960 14088 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c310dd5906a24299b6f62201b30b86db"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:14.851804 14088 raft_consensus.cc:482] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.852046 14088 raft_consensus.cc:3037] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.852843 13810 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:15:14.853482 16131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:14.856102 13828 raft_consensus.cc:2454] T cdf9885a1e7c4b8dbb160e370ca0f60c P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 5.
I20240503 08:15:14.856952 14088 raft_consensus.cc:504] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.857883 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c310dd5906a24299b6f62201b30b86db" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:14.858081 13960 raft_consensus.cc:3037] T c310dd5906a24299b6f62201b30b86db P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.859726 14024 leader_election.cc:302] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc, abb3df711d3a46809274b5cfbd4ce4e0; no voters: 296f3a209cf5447aaff2b8ea96a3bc4f
I20240503 08:15:14.860034 16361 raft_consensus.cc:2781] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:15:14.860296 16361 raft_consensus.cc:686] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 LEADER]: Becoming Leader. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:14.860709 16361 consensus_queue.cc:237] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:14.862190 14088 leader_election.cc:288] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 4 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:14.866433 13686 catalog_manager.cc:5484] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0 reported cstate change: term changed from 3 to 5, leader changed from 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66) to abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67). New cstate: current_term: 5 leader_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } }
I20240503 08:15:14.866904 13686 catalog_manager.cc:5484] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0 reported cstate change: term changed from 4 to 5, leader changed from 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68) to abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67). New cstate: current_term: 5 leader_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:14.867398 13686 catalog_manager.cc:5484] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0 reported cstate change: term changed from 3 to 5, leader changed from 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65) to abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67). New cstate: current_term: 5 leader_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } }
I20240503 08:15:14.867863 13686 catalog_manager.cc:5484] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0 reported cstate change: term changed from 4 to 5, leader changed from 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68) to abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67). New cstate: current_term: 5 leader_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } }
I20240503 08:15:14.868386 13686 catalog_manager.cc:5484] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0 reported cstate change: term changed from 3 to 5, leader changed from 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65) to abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67). New cstate: current_term: 5 leader_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:14.869781 14068 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:14.872248 15868 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:14.874485 14090 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8bb1779c7ac64d599cde0665505b3dba"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:14.874656 14090 raft_consensus.cc:482] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.874754 14090 raft_consensus.cc:3037] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.875303 14217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c310dd5906a24299b6f62201b30b86db" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:14.875594 14217 raft_consensus.cc:3032] T c310dd5906a24299b6f62201b30b86db P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:15:14.875692 14217 raft_consensus.cc:729] T c310dd5906a24299b6f62201b30b86db P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Running, Role: LEADER
I20240503 08:15:14.864156 13960 raft_consensus.cc:2454] T c310dd5906a24299b6f62201b30b86db P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 4.
I20240503 08:15:14.876387 14023 leader_election.cc:302] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:14.876638 16361 raft_consensus.cc:2781] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:15:14.876788 16361 raft_consensus.cc:686] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 LEADER]: Becoming Leader. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:14.877039 16361 consensus_queue.cc:237] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.882223 14217 consensus_queue.cc:260] T c310dd5906a24299b6f62201b30b86db P 5761f1165461459f9aaa8cd5d8e9b0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.882511 14217 raft_consensus.cc:3037] T c310dd5906a24299b6f62201b30b86db P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.885072 13686 catalog_manager.cc:5484] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0 reported cstate change: term changed from 3 to 5, leader changed from 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66) to abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67). New cstate: current_term: 5 leader_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } }
I20240503 08:15:14.885422 14217 raft_consensus.cc:2396] T c310dd5906a24299b6f62201b30b86db P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate abb3df711d3a46809274b5cfbd4ce4e0 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 08:15:14.886152 13686 catalog_manager.cc:5484] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 reported cstate change: term changed from 3 to 4, leader changed from 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68) to abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67). New cstate: current_term: 4 leader_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } }
I20240503 08:15:14.896585 14090 raft_consensus.cc:504] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.897713 13828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb1779c7ac64d599cde0665505b3dba" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:14.897742 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb1779c7ac64d599cde0665505b3dba" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:14.897949 13828 raft_consensus.cc:3032] T 8bb1779c7ac64d599cde0665505b3dba P 5a198ad76e4f462c8be0859184aeb2fc [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:15:14.897951 13960 raft_consensus.cc:3037] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.898038 13828 raft_consensus.cc:729] T 8bb1779c7ac64d599cde0665505b3dba P 5a198ad76e4f462c8be0859184aeb2fc [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:14.898276 13828 consensus_queue.cc:260] T 8bb1779c7ac64d599cde0665505b3dba P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.898615 13828 raft_consensus.cc:3037] T 8bb1779c7ac64d599cde0665505b3dba P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.900655 13960 raft_consensus.cc:2454] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 4.
I20240503 08:15:14.901065 14023 leader_election.cc:302] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:14.901204 13828 raft_consensus.cc:2454] T 8bb1779c7ac64d599cde0665505b3dba P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 4.
I20240503 08:15:14.901329 16210 raft_consensus.cc:2781] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:15:14.901455 16210 raft_consensus.cc:686] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 LEADER]: Becoming Leader. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:14.901685 16210 consensus_queue.cc:237] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.904261 14090 leader_election.cc:288] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 4 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:15:14.904461 13686 catalog_manager.cc:5484] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 reported cstate change: term changed from 3 to 4, leader changed from 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65) to abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67). New cstate: current_term: 4 leader_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:14.908771 14090 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "79014e600505478883fc90159784ce6c"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:14.909054 14090 raft_consensus.cc:482] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.909219 14090 raft_consensus.cc:3037] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.912039 14090 raft_consensus.cc:504] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:14.915336 14090 leader_election.cc:288] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 4 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:14.915562 13828 raft_consensus.cc:1261] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Refusing update from remote peer 5761f1165461459f9aaa8cd5d8e9b0b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20240503 08:15:14.915697 13828 pending_rounds.cc:76] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc: Aborting all ops after (but not including) 4
I20240503 08:15:14.915778 13828 pending_rounds.cc:97] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20240503 08:15:14.915840 13828 raft_consensus.cc:2866] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:15:14.916070 14217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79014e600505478883fc90159784ce6c" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:14.916285 14217 raft_consensus.cc:3032] T 79014e600505478883fc90159784ce6c P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:15:14.916399 14217 raft_consensus.cc:729] T 79014e600505478883fc90159784ce6c P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Running, Role: LEADER
I20240503 08:15:14.916726 14217 consensus_queue.cc:260] T 79014e600505478883fc90159784ce6c P 5761f1165461459f9aaa8cd5d8e9b0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:14.916910 16316 consensus_queue.cc:1035] T 06e7243793e9467daeed79f3b1f5be5a P 5761f1165461459f9aaa8cd5d8e9b0b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:15:14.917078 14217 raft_consensus.cc:3037] T 79014e600505478883fc90159784ce6c P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.921010 14088 raft_consensus.cc:1261] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Refusing update from remote peer 5761f1165461459f9aaa8cd5d8e9b0b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240503 08:15:14.921427 13828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79014e600505478883fc90159784ce6c" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:14.921631 13828 raft_consensus.cc:3037] T 79014e600505478883fc90159784ce6c P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.921694 16322 consensus_queue.cc:1035] T 06e7243793e9467daeed79f3b1f5be5a P 5761f1165461459f9aaa8cd5d8e9b0b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:15:14.923069 14217 raft_consensus.cc:2396] T 79014e600505478883fc90159784ce6c P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate abb3df711d3a46809274b5cfbd4ce4e0 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.924341 13828 raft_consensus.cc:2396] T 79014e600505478883fc90159784ce6c P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate abb3df711d3a46809274b5cfbd4ce4e0 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:14.925697 13828 raft_consensus.cc:1261] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Refusing update from remote peer 5761f1165461459f9aaa8cd5d8e9b0b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240503 08:15:14.925887 13828 pending_rounds.cc:76] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc: Aborting all ops after (but not including) 3
I20240503 08:15:14.926002 13828 pending_rounds.cc:97] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 08:15:14.926146 13828 raft_consensus.cc:2866] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:15:14.926268 14024 leader_election.cc:302] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: abb3df711d3a46809274b5cfbd4ce4e0; no voters: 5761f1165461459f9aaa8cd5d8e9b0b9, 5a198ad76e4f462c8be0859184aeb2fc
I20240503 08:15:14.926635 16210 raft_consensus.cc:2726] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 08:15:14.926865 16322 consensus_queue.cc:1035] T 769c4490062b4206951af2f4b9a6da05 P 5761f1165461459f9aaa8cd5d8e9b0b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:15:14.928768 14090 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d2ba977460694815a38d415a14e1cd5f"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:14.929047 14090 raft_consensus.cc:482] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.929172 14090 raft_consensus.cc:3037] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.933825 13960 raft_consensus.cc:1261] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Refusing update from remote peer 5761f1165461459f9aaa8cd5d8e9b0b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 08:15:14.935104 16322 consensus_queue.cc:1035] T 769c4490062b4206951af2f4b9a6da05 P 5761f1165461459f9aaa8cd5d8e9b0b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:15:14.942715 14088 raft_consensus.cc:1261] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Refusing update from remote peer 5761f1165461459f9aaa8cd5d8e9b0b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 08:15:14.943274 16322 consensus_queue.cc:1035] T 150f0d2d17bc4a59a8d82982790b4594 P 5761f1165461459f9aaa8cd5d8e9b0b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:15:14.950531 14090 raft_consensus.cc:504] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:14.953788 13957 raft_consensus.cc:1261] T 150f0d2d17bc4a59a8d82982790b4594 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Refusing update from remote peer 5761f1165461459f9aaa8cd5d8e9b0b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240503 08:15:14.953897 13957 pending_rounds.cc:76] T 150f0d2d17bc4a59a8d82982790b4594 P 296f3a209cf5447aaff2b8ea96a3bc4f: Aborting all ops after (but not including) 3
I20240503 08:15:14.953977 13957 pending_rounds.cc:97] T 150f0d2d17bc4a59a8d82982790b4594 P 296f3a209cf5447aaff2b8ea96a3bc4f: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 08:15:14.954043 13957 raft_consensus.cc:2866] T 150f0d2d17bc4a59a8d82982790b4594 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:15:14.954593 16322 consensus_queue.cc:1035] T 150f0d2d17bc4a59a8d82982790b4594 P 5761f1165461459f9aaa8cd5d8e9b0b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:15:14.958609 14090 leader_election.cc:288] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 4 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:15:14.958546 13828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ba977460694815a38d415a14e1cd5f" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:14.958781 13828 raft_consensus.cc:3032] T d2ba977460694815a38d415a14e1cd5f P 5a198ad76e4f462c8be0859184aeb2fc [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:15:14.958869 13828 raft_consensus.cc:729] T d2ba977460694815a38d415a14e1cd5f P 5a198ad76e4f462c8be0859184aeb2fc [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:14.959137 13828 consensus_queue.cc:260] T d2ba977460694815a38d415a14e1cd5f P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:14.959517 13828 raft_consensus.cc:3037] T d2ba977460694815a38d415a14e1cd5f P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.961304 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ba977460694815a38d415a14e1cd5f" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:14.961479 13960 raft_consensus.cc:3037] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.962666 14090 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "150f0d2d17bc4a59a8d82982790b4594"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:14.964633 13960 raft_consensus.cc:2454] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 4.
I20240503 08:15:14.967084 14023 leader_election.cc:302] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:14.967413 16210 raft_consensus.cc:2781] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:15:14.967530 16210 raft_consensus.cc:686] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 LEADER]: Becoming Leader. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:14.967761 16210 consensus_queue.cc:237] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:14.969723 13828 raft_consensus.cc:2454] T d2ba977460694815a38d415a14e1cd5f P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 4.
I20240503 08:15:14.970659 14090 raft_consensus.cc:482] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.970778 14090 raft_consensus.cc:3037] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.973395 14090 raft_consensus.cc:504] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.973866 14087 raft_consensus.cc:1226] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Rejecting Update request from peer 5761f1165461459f9aaa8cd5d8e9b0b9 for earlier term 3. Current term is 4. Ops: []
I20240503 08:15:14.974566 16315 consensus_queue.cc:1046] T 150f0d2d17bc4a59a8d82982790b4594 P 5761f1165461459f9aaa8cd5d8e9b0b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:15:14.974686 14090 leader_election.cc:288] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 4 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:14.974835 16315 raft_consensus.cc:3032] T 150f0d2d17bc4a59a8d82982790b4594 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:15:14.974933 16315 raft_consensus.cc:729] T 150f0d2d17bc4a59a8d82982790b4594 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Running, Role: LEADER
I20240503 08:15:14.975178 16315 consensus_queue.cc:260] T 150f0d2d17bc4a59a8d82982790b4594 P 5761f1165461459f9aaa8cd5d8e9b0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.975404 13828 raft_consensus.cc:1261] T 84204cbcda32446ab708717e9633af69 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Refusing update from remote peer 5761f1165461459f9aaa8cd5d8e9b0b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240503 08:15:14.975564 16315 raft_consensus.cc:3037] T 150f0d2d17bc4a59a8d82982790b4594 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.975658 14217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150f0d2d17bc4a59a8d82982790b4594" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:14.975939 14090 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "06e7243793e9467daeed79f3b1f5be5a"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:14.976083 14090 raft_consensus.cc:482] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:14.976181 14090 raft_consensus.cc:3037] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:14.978169 13828 pending_rounds.cc:76] T 84204cbcda32446ab708717e9633af69 P 5a198ad76e4f462c8be0859184aeb2fc: Aborting all ops after (but not including) 3
I20240503 08:15:14.978250 13828 pending_rounds.cc:97] T 84204cbcda32446ab708717e9633af69 P 5a198ad76e4f462c8be0859184aeb2fc: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 08:15:14.978330 13828 raft_consensus.cc:2866] T 84204cbcda32446ab708717e9633af69 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:15:14.978870 16322 consensus_queue.cc:1035] T 84204cbcda32446ab708717e9633af69 P 5761f1165461459f9aaa8cd5d8e9b0b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:15:14.979588 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150f0d2d17bc4a59a8d82982790b4594" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:14.979815 13960 raft_consensus.cc:3037] T 150f0d2d17bc4a59a8d82982790b4594 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:14.983549 14217 raft_consensus.cc:2454] T 150f0d2d17bc4a59a8d82982790b4594 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 4.
I20240503 08:15:14.983850 13686 catalog_manager.cc:5484] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 reported cstate change: term changed from 3 to 4, leader changed from 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65) to abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67). New cstate: current_term: 4 leader_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } }
I20240503 08:15:14.984339 13957 raft_consensus.cc:1261] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Refusing update from remote peer 5761f1165461459f9aaa8cd5d8e9b0b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 08:15:14.984951 16316 consensus_queue.cc:1035] T 84204cbcda32446ab708717e9633af69 P 5761f1165461459f9aaa8cd5d8e9b0b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:15:14.986183 14090 raft_consensus.cc:504] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.987435 14025 leader_election.cc:302] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:14.987709 16361 raft_consensus.cc:2781] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:15:14.987823 16361 raft_consensus.cc:686] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 LEADER]: Becoming Leader. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:14.988049 16361 consensus_queue.cc:237] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:14.988978 13828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e7243793e9467daeed79f3b1f5be5a" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:14.989178 13828 raft_consensus.cc:3037] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:14.989751 14217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e7243793e9467daeed79f3b1f5be5a" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:14.990057 14217 raft_consensus.cc:3032] T 06e7243793e9467daeed79f3b1f5be5a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:15:14.990167 14217 raft_consensus.cc:729] T 06e7243793e9467daeed79f3b1f5be5a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Running, Role: LEADER
I20240503 08:15:14.990478 14217 consensus_queue.cc:260] T 06e7243793e9467daeed79f3b1f5be5a P 5761f1165461459f9aaa8cd5d8e9b0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:14.990542 13689 catalog_manager.cc:5484] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 reported cstate change: term changed from 3 to 4, leader changed from 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68) to abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67). New cstate: current_term: 4 leader_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } }
I20240503 08:15:14.991824 14217 raft_consensus.cc:3037] T 06e7243793e9467daeed79f3b1f5be5a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:14.991839 13830 raft_consensus.cc:1261] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Refusing update from remote peer 5761f1165461459f9aaa8cd5d8e9b0b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20240503 08:15:14.991921 13830 pending_rounds.cc:76] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc: Aborting all ops after (but not including) 4
I20240503 08:15:14.991994 13830 pending_rounds.cc:97] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20240503 08:15:14.992060 13830 raft_consensus.cc:2866] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:15:14.992609 16322 consensus_queue.cc:1035] T e82cb19fece24181b7607d9447ba1eb2 P 5761f1165461459f9aaa8cd5d8e9b0b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:15:14.996552 14087 raft_consensus.cc:1261] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Refusing update from remote peer 5761f1165461459f9aaa8cd5d8e9b0b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240503 08:15:14.997483 16316 consensus_queue.cc:1035] T e82cb19fece24181b7607d9447ba1eb2 P 5761f1165461459f9aaa8cd5d8e9b0b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:15:15.000622 14217 raft_consensus.cc:2454] T 06e7243793e9467daeed79f3b1f5be5a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 5.
I20240503 08:15:15.001039 14025 leader_election.cc:302] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:15.001286 16210 raft_consensus.cc:2781] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:15:15.001411 16210 raft_consensus.cc:686] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 LEADER]: Becoming Leader. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:15.001622 16210 consensus_queue.cc:237] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.002219 14090 leader_election.cc:288] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 5 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:15.003957 13828 raft_consensus.cc:2454] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 5.
I20240503 08:15:15.005125 13689 catalog_manager.cc:5484] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 reported cstate change: term changed from 4 to 5, leader changed from 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68) to abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67). New cstate: current_term: 5 leader_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:15.006213 14090 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "666be8071a904e6cbff984fa293d00b5"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:15.006471 14090 raft_consensus.cc:482] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.006616 14090 raft_consensus.cc:3037] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:15.007465 13960 raft_consensus.cc:2454] T 150f0d2d17bc4a59a8d82982790b4594 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 4.
I20240503 08:15:15.010283 14090 raft_consensus.cc:504] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.011436 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "666be8071a904e6cbff984fa293d00b5" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:15.011698 13960 raft_consensus.cc:3037] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:15.014616 13828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "666be8071a904e6cbff984fa293d00b5" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:15.014811 13828 raft_consensus.cc:3032] T 666be8071a904e6cbff984fa293d00b5 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:15:15.014895 13828 raft_consensus.cc:729] T 666be8071a904e6cbff984fa293d00b5 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:15.015132 13828 consensus_queue.cc:260] T 666be8071a904e6cbff984fa293d00b5 P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.015515 13828 raft_consensus.cc:3037] T 666be8071a904e6cbff984fa293d00b5 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:15.019294 13828 raft_consensus.cc:2454] T 666be8071a904e6cbff984fa293d00b5 P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 4.
I20240503 08:15:15.019567 14087 raft_consensus.cc:1261] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Refusing update from remote peer 5761f1165461459f9aaa8cd5d8e9b0b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 08:15:15.019816 14024 leader_election.cc:302] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:15.020094 16361 raft_consensus.cc:2781] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:15:15.020237 16361 raft_consensus.cc:686] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 LEADER]: Becoming Leader. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:15.020319 16315 consensus_queue.cc:1035] T 9cae16af5a034cc29b8bb410a758c332 P 5761f1165461459f9aaa8cd5d8e9b0b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:15:15.020483 16361 consensus_queue.cc:237] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.020805 14090 leader_election.cc:288] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 4 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:15.021926 13960 raft_consensus.cc:2454] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 4.
I20240503 08:15:15.023213 14087 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9cae16af5a034cc29b8bb410a758c332"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:15.024380 13689 catalog_manager.cc:5484] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 reported cstate change: term changed from 3 to 4, leader changed from 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65) to abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67). New cstate: current_term: 4 leader_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:15.029351 14087 raft_consensus.cc:482] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.029479 14087 raft_consensus.cc:3037] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:15.031948 14087 raft_consensus.cc:504] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:15.032709 14087 leader_election.cc:288] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 4 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:15.033298 14217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cae16af5a034cc29b8bb410a758c332" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:15.033507 14217 raft_consensus.cc:3032] T 9cae16af5a034cc29b8bb410a758c332 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:15:15.033572 14217 raft_consensus.cc:729] T 9cae16af5a034cc29b8bb410a758c332 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Running, Role: LEADER
I20240503 08:15:15.033844 14217 consensus_queue.cc:260] T 9cae16af5a034cc29b8bb410a758c332 P 5761f1165461459f9aaa8cd5d8e9b0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:15.034109 14217 raft_consensus.cc:3037] T 9cae16af5a034cc29b8bb410a758c332 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:15.034266 14087 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "667abbd8fb6c49489127c12977d7a323"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:15.034433 14087 raft_consensus.cc:482] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.034539 14087 raft_consensus.cc:3037] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.035607 13828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cae16af5a034cc29b8bb410a758c332" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:15.035827 13828 raft_consensus.cc:3037] T 9cae16af5a034cc29b8bb410a758c332 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:15.036787 14217 raft_consensus.cc:2454] T 9cae16af5a034cc29b8bb410a758c332 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 4.
I20240503 08:15:15.036939 14087 raft_consensus.cc:504] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:15.037212 14025 leader_election.cc:302] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:15.037504 14087 leader_election.cc:288] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 5 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:15.037680 16361 raft_consensus.cc:2781] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:15:15.037844 16361 raft_consensus.cc:686] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 LEADER]: Becoming Leader. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:15.038043 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "667abbd8fb6c49489127c12977d7a323" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:15.038105 16361 consensus_queue.cc:237] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:15.038262 13960 raft_consensus.cc:3037] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.038543 13828 raft_consensus.cc:2454] T 9cae16af5a034cc29b8bb410a758c332 P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 4.
I20240503 08:15:15.039535 14217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "667abbd8fb6c49489127c12977d7a323" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:15.039803 14217 raft_consensus.cc:3037] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.040820 13960 raft_consensus.cc:2396] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate abb3df711d3a46809274b5cfbd4ce4e0 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 08:15:15.040971 14087 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "94cfd3b764b148bb8b42e5a5cb7883b3"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:15.041252 14087 raft_consensus.cc:482] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.041417 14087 raft_consensus.cc:3037] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.042186 13689 catalog_manager.cc:5484] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 reported cstate change: term changed from 3 to 4, leader changed from 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68) to abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67). New cstate: current_term: 4 leader_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } }
I20240503 08:15:15.043082 14217 raft_consensus.cc:2454] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 5.
I20240503 08:15:15.044237 14087 raft_consensus.cc:504] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.044374 14025 leader_election.cc:302] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, abb3df711d3a46809274b5cfbd4ce4e0; no voters: 296f3a209cf5447aaff2b8ea96a3bc4f
I20240503 08:15:15.044709 16210 raft_consensus.cc:2781] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:15:15.044880 16210 raft_consensus.cc:686] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 LEADER]: Becoming Leader. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:15.045029 14087 leader_election.cc:288] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 5 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:15:15.045153 16210 consensus_queue.cc:237] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:15.045413 13828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94cfd3b764b148bb8b42e5a5cb7883b3" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:15.045616 13828 raft_consensus.cc:3037] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.045830 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94cfd3b764b148bb8b42e5a5cb7883b3" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:15.045938 14090 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e82cb19fece24181b7607d9447ba1eb2"
dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
from {username='slave'} at 127.0.0.1:40742
I20240503 08:15:15.046011 13960 raft_consensus.cc:3037] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.046103 14090 raft_consensus.cc:482] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.046211 14090 raft_consensus.cc:3037] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.048350 13828 raft_consensus.cc:2454] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 5.
I20240503 08:15:15.048653 13960 raft_consensus.cc:2396] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate abb3df711d3a46809274b5cfbd4ce4e0 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 08:15:15.048888 14024 leader_election.cc:302] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:15.049057 14090 raft_consensus.cc:504] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.049273 16361 raft_consensus.cc:2781] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:15:15.049444 16361 raft_consensus.cc:686] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 LEADER]: Becoming Leader. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:15.049702 16361 consensus_queue.cc:237] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.050468 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82cb19fece24181b7607d9447ba1eb2" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:15.050674 14219 raft_consensus.cc:3032] T e82cb19fece24181b7607d9447ba1eb2 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:15:15.050621 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82cb19fece24181b7607d9447ba1eb2" candidate_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:15.050753 14219 raft_consensus.cc:729] T e82cb19fece24181b7607d9447ba1eb2 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Running, Role: LEADER
I20240503 08:15:15.050796 13830 raft_consensus.cc:3037] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.051036 14219 consensus_queue.cc:260] T e82cb19fece24181b7607d9447ba1eb2 P 5761f1165461459f9aaa8cd5d8e9b0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.051409 14219 raft_consensus.cc:3037] T e82cb19fece24181b7607d9447ba1eb2 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.053360 13830 raft_consensus.cc:2454] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 5.
I20240503 08:15:15.054044 14219 raft_consensus.cc:2454] T e82cb19fece24181b7607d9447ba1eb2 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb3df711d3a46809274b5cfbd4ce4e0 in term 5.
I20240503 08:15:15.054780 14024 leader_election.cc:302] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:15.055086 16362 raft_consensus.cc:2781] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:15:15.055239 16362 raft_consensus.cc:686] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 LEADER]: Becoming Leader. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:15.055456 16362 consensus_queue.cc:237] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.056758 14090 leader_election.cc:288] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0 [CANDIDATE]: Term 5 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:15.061194 13689 catalog_manager.cc:5484] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0 reported cstate change: term changed from 3 to 5, leader changed from 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66) to abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67). New cstate: current_term: 5 leader_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } }
I20240503 08:15:15.061923 13689 catalog_manager.cc:5484] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0 reported cstate change: term changed from 4 to 5, leader changed from 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68) to abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67). New cstate: current_term: 5 leader_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:15.062351 13689 catalog_manager.cc:5484] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0 reported cstate change: term changed from 3 to 5, leader changed from 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66) to abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67). New cstate: current_term: 5 leader_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:15.064334 13960 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "51b32125655f4ef78dc49ba7174775da"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:15.064568 13960 raft_consensus.cc:482] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.064697 13960 raft_consensus.cc:3037] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.067183 13960 raft_consensus.cc:504] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:15.067938 13960 leader_election.cc:288] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 5 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:15.068269 14217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51b32125655f4ef78dc49ba7174775da" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:15.068760 14217 raft_consensus.cc:3037] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.068883 13828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51b32125655f4ef78dc49ba7174775da" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:15.069100 13828 raft_consensus.cc:3037] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.069000 13960 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8bb1779c7ac64d599cde0665505b3dba"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:15.070008 13960 raft_consensus.cc:482] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.070132 13960 raft_consensus.cc:3037] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.071738 14217 raft_consensus.cc:2454] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 5.
I20240503 08:15:15.071825 13828 raft_consensus.cc:2396] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 296f3a209cf5447aaff2b8ea96a3bc4f for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 08:15:15.072252 13895 leader_election.cc:302] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5761f1165461459f9aaa8cd5d8e9b0b9; no voters: 5a198ad76e4f462c8be0859184aeb2fc
I20240503 08:15:15.072540 16232 raft_consensus.cc:2781] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:15:15.072862 16232 raft_consensus.cc:686] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:15.073165 16232 consensus_queue.cc:237] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:15.075292 13960 raft_consensus.cc:504] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.075917 13960 leader_election.cc:288] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 5 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:15.076318 13828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb1779c7ac64d599cde0665505b3dba" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:15.076391 14087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb1779c7ac64d599cde0665505b3dba" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:15.076521 13828 raft_consensus.cc:3037] T 8bb1779c7ac64d599cde0665505b3dba P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.076594 14087 raft_consensus.cc:3032] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:15:15.076695 14087 raft_consensus.cc:729] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:15.076944 14087 consensus_queue.cc:260] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.076927 13689 catalog_manager.cc:5484] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 3 to 5, leader changed from 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65) to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 5 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } }
I20240503 08:15:15.077251 14087 raft_consensus.cc:3037] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.078866 13828 raft_consensus.cc:2454] T 8bb1779c7ac64d599cde0665505b3dba P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 5.
I20240503 08:15:15.079964 14087 raft_consensus.cc:2396] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 296f3a209cf5447aaff2b8ea96a3bc4f for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:15:15.080304 13894 leader_election.cc:302] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:15.080874 16232 raft_consensus.cc:2781] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:15:15.081143 16232 raft_consensus.cc:686] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:15.081436 16232 consensus_queue.cc:237] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.081846 13960 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dd3d6353358d4b02af928cd9f9410aca"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:15.082093 13960 raft_consensus.cc:482] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.082295 13960 raft_consensus.cc:3037] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.084669 13960 raft_consensus.cc:504] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.085723 13828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd3d6353358d4b02af928cd9f9410aca" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:15.085588 13689 catalog_manager.cc:5484] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 4 to 5, leader changed from abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67) to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 5 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:15.085939 13828 raft_consensus.cc:3037] T dd3d6353358d4b02af928cd9f9410aca P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.086024 13960 leader_election.cc:288] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 6 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:15.086536 14087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd3d6353358d4b02af928cd9f9410aca" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:15.086741 14087 raft_consensus.cc:3032] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:15:15.086829 14087 raft_consensus.cc:729] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:15.087080 14087 consensus_queue.cc:260] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.087363 14087 raft_consensus.cc:3037] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.087821 13960 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "70e8bfaa82464d3ab3338a779457dc02"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:15.087983 13960 raft_consensus.cc:482] T 70e8bfaa82464d3ab3338a779457dc02 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.088088 13960 raft_consensus.cc:3037] T 70e8bfaa82464d3ab3338a779457dc02 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.088461 13828 raft_consensus.cc:2454] T dd3d6353358d4b02af928cd9f9410aca P 5a198ad76e4f462c8be0859184aeb2fc [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 6.
I20240503 08:15:15.089124 13894 leader_election.cc:302] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:15.089423 16232 raft_consensus.cc:2781] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 FOLLOWER]: Leader election won for term 6
I20240503 08:15:15.089558 16232 raft_consensus.cc:686] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:15.089884 16232 consensus_queue.cc:237] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.090869 13960 raft_consensus.cc:504] T 70e8bfaa82464d3ab3338a779457dc02 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:15.091176 14087 raft_consensus.cc:2396] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 296f3a209cf5447aaff2b8ea96a3bc4f for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:15:15.091897 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70e8bfaa82464d3ab3338a779457dc02" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:15.092130 14090 raft_consensus.cc:3037] T 70e8bfaa82464d3ab3338a779457dc02 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.094072 13960 leader_election.cc:288] T 70e8bfaa82464d3ab3338a779457dc02 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 5 election: Requested vote from peers abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:15.094563 14217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70e8bfaa82464d3ab3338a779457dc02" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:15.094822 14090 raft_consensus.cc:2454] T 70e8bfaa82464d3ab3338a779457dc02 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 5.
I20240503 08:15:15.094959 14217 raft_consensus.cc:3037] T 70e8bfaa82464d3ab3338a779457dc02 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.095348 13896 leader_election.cc:302] T 70e8bfaa82464d3ab3338a779457dc02 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:15.095611 16389 raft_consensus.cc:2781] T 70e8bfaa82464d3ab3338a779457dc02 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:15:15.096293 13960 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a478c28e5d4646fdb956f5b65e5c6c94"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:15.096488 13960 raft_consensus.cc:482] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.096581 13960 raft_consensus.cc:3037] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.097945 14217 raft_consensus.cc:2396] T 70e8bfaa82464d3ab3338a779457dc02 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 296f3a209cf5447aaff2b8ea96a3bc4f for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 08:15:15.099610 13960 raft_consensus.cc:504] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:15.100569 13960 leader_election.cc:288] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 5 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:15.100735 13828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a478c28e5d4646fdb956f5b65e5c6c94" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:15.100993 13828 raft_consensus.cc:3037] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.100904 14217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a478c28e5d4646fdb956f5b65e5c6c94" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:15.101259 14217 raft_consensus.cc:3037] T a478c28e5d4646fdb956f5b65e5c6c94 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.101598 13960 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "769c4490062b4206951af2f4b9a6da05"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:15.101881 13960 raft_consensus.cc:482] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.102017 13960 raft_consensus.cc:3037] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:15.103837 14217 raft_consensus.cc:2454] T a478c28e5d4646fdb956f5b65e5c6c94 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 5.
I20240503 08:15:15.104243 13895 leader_election.cc:302] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5761f1165461459f9aaa8cd5d8e9b0b9; no voters:
I20240503 08:15:15.104194 13828 raft_consensus.cc:2396] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 296f3a209cf5447aaff2b8ea96a3bc4f for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 08:15:15.104754 13960 raft_consensus.cc:504] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:15.105403 13960 leader_election.cc:288] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 4 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:15.105893 13828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "769c4490062b4206951af2f4b9a6da05" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:15.106396 13960 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d09e60e9c309425da9d27ae1153e4510"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:15.106611 13960 raft_consensus.cc:482] T d09e60e9c309425da9d27ae1153e4510 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.106729 13960 raft_consensus.cc:3037] T d09e60e9c309425da9d27ae1153e4510 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.106650 13828 raft_consensus.cc:3037] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:15.109460 13960 raft_consensus.cc:504] T d09e60e9c309425da9d27ae1153e4510 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.110533 13960 leader_election.cc:288] T d09e60e9c309425da9d27ae1153e4510 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 6 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:15.110639 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d09e60e9c309425da9d27ae1153e4510" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:15.110841 14090 raft_consensus.cc:3032] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:15:15.110937 14090 raft_consensus.cc:729] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:15.111207 14090 consensus_queue.cc:260] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.111515 14090 raft_consensus.cc:3037] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.114398 14090 raft_consensus.cc:2396] T d09e60e9c309425da9d27ae1153e4510 P abb3df711d3a46809274b5cfbd4ce4e0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 296f3a209cf5447aaff2b8ea96a3bc4f for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:15:15.114499 13960 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "84204cbcda32446ab708717e9633af69"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:15.114706 13960 raft_consensus.cc:482] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.114841 13960 raft_consensus.cc:3037] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:15.115518 13828 raft_consensus.cc:2454] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 4.
I20240503 08:15:15.115989 13894 leader_election.cc:302] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:15.117568 13960 raft_consensus.cc:504] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:15.118247 13960 leader_election.cc:288] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 4 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:15.122233 16396 raft_consensus.cc:2781] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:15:15.123356 16389 raft_consensus.cc:686] T 70e8bfaa82464d3ab3338a779457dc02 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:15.123899 16389 consensus_queue.cc:237] T 70e8bfaa82464d3ab3338a779457dc02 P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:15.123538 16232 raft_consensus.cc:2781] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:15:15.124701 13689 catalog_manager.cc:5484] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 5 to 6, leader changed from abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67) to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 6 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:15.125216 16232 raft_consensus.cc:686] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:15.125813 16396 raft_consensus.cc:686] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:15.126336 16396 consensus_queue.cc:237] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:15.125996 16232 consensus_queue.cc:237] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:15.128224 13960 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f27cc9b78d084ac791de35b29e08a09a"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:15.128413 13960 raft_consensus.cc:482] T f27cc9b78d084ac791de35b29e08a09a P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.128510 13960 raft_consensus.cc:3037] T f27cc9b78d084ac791de35b29e08a09a P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.132761 14217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "769c4490062b4206951af2f4b9a6da05" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:15.133002 14217 raft_consensus.cc:3032] T 769c4490062b4206951af2f4b9a6da05 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:15:15.133106 14217 raft_consensus.cc:729] T 769c4490062b4206951af2f4b9a6da05 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Running, Role: LEADER
I20240503 08:15:15.133152 13828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84204cbcda32446ab708717e9633af69" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:15.133448 13828 raft_consensus.cc:3037] T 84204cbcda32446ab708717e9633af69 P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:15.133414 14217 consensus_queue.cc:260] T 769c4490062b4206951af2f4b9a6da05 P 5761f1165461459f9aaa8cd5d8e9b0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:15.133819 14217 raft_consensus.cc:3037] T 769c4490062b4206951af2f4b9a6da05 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:15.134199 13960 raft_consensus.cc:504] T f27cc9b78d084ac791de35b29e08a09a P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:15.134790 13960 leader_election.cc:288] T f27cc9b78d084ac791de35b29e08a09a P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 5 election: Requested vote from peers abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:15.135149 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f27cc9b78d084ac791de35b29e08a09a" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:15.135329 14090 raft_consensus.cc:3037] T f27cc9b78d084ac791de35b29e08a09a P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.136864 14217 raft_consensus.cc:2454] T 769c4490062b4206951af2f4b9a6da05 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 4.
I20240503 08:15:15.137305 14090 raft_consensus.cc:2454] T f27cc9b78d084ac791de35b29e08a09a P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 5.
I20240503 08:15:15.137699 14217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d09e60e9c309425da9d27ae1153e4510" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:15.137970 13896 leader_election.cc:302] T f27cc9b78d084ac791de35b29e08a09a P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:15.138036 14217 raft_consensus.cc:3037] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.137964 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84204cbcda32446ab708717e9633af69" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:15.138192 14219 raft_consensus.cc:3032] T 84204cbcda32446ab708717e9633af69 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:15:15.138281 14219 raft_consensus.cc:729] T 84204cbcda32446ab708717e9633af69 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Running, Role: LEADER
I20240503 08:15:15.138531 14219 consensus_queue.cc:260] T 84204cbcda32446ab708717e9633af69 P 5761f1165461459f9aaa8cd5d8e9b0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:15.138909 14219 raft_consensus.cc:3037] T 84204cbcda32446ab708717e9633af69 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:15.140058 13828 raft_consensus.cc:2454] T 84204cbcda32446ab708717e9633af69 P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 4.
I20240503 08:15:15.140719 16396 raft_consensus.cc:2781] T f27cc9b78d084ac791de35b29e08a09a P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:15:15.141005 14217 raft_consensus.cc:2396] T d09e60e9c309425da9d27ae1153e4510 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 296f3a209cf5447aaff2b8ea96a3bc4f for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:15:15.141036 16396 raft_consensus.cc:686] T f27cc9b78d084ac791de35b29e08a09a P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:15.141517 14219 raft_consensus.cc:2454] T 84204cbcda32446ab708717e9633af69 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 4.
I20240503 08:15:15.141845 14218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f27cc9b78d084ac791de35b29e08a09a" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:15.142009 14218 raft_consensus.cc:3037] T f27cc9b78d084ac791de35b29e08a09a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.141571 16396 consensus_queue.cc:237] T f27cc9b78d084ac791de35b29e08a09a P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:15.144379 14218 raft_consensus.cc:2396] T f27cc9b78d084ac791de35b29e08a09a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 296f3a209cf5447aaff2b8ea96a3bc4f for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 08:15:15.144809 13894 leader_election.cc:302] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:15.144695 13689 catalog_manager.cc:5484] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 3 to 4, leader changed from 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68) to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 4 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } }
I20240503 08:15:15.145291 16389 raft_consensus.cc:2781] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:15:15.145171 13689 catalog_manager.cc:5484] T 70e8bfaa82464d3ab3338a779457dc02 P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 3 to 5, leader changed from 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68) to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 5 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } }
I20240503 08:15:15.145498 16389 raft_consensus.cc:686] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:15.145613 13689 catalog_manager.cc:5484] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 3 to 5, leader changed from 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65) to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 5 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } }
I20240503 08:15:15.145757 13895 leader_election.cc:302] T d09e60e9c309425da9d27ae1153e4510 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f; no voters: 5761f1165461459f9aaa8cd5d8e9b0b9, abb3df711d3a46809274b5cfbd4ce4e0
I20240503 08:15:15.145788 16389 consensus_queue.cc:237] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:15.146574 16232 raft_consensus.cc:2726] T d09e60e9c309425da9d27ae1153e4510 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240503 08:15:15.146878 13960 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c244226a1c40488f949735420bfacd0b"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:15.147034 13960 raft_consensus.cc:482] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.147133 13960 raft_consensus.cc:3037] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.150075 13960 raft_consensus.cc:504] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.151021 13828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c244226a1c40488f949735420bfacd0b" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:15.151198 13828 raft_consensus.cc:3037] T c244226a1c40488f949735420bfacd0b P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.151176 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c244226a1c40488f949735420bfacd0b" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:15.151405 14090 raft_consensus.cc:3032] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:15:15.151479 14090 raft_consensus.cc:729] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:15.151773 14090 consensus_queue.cc:260] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.152060 14090 raft_consensus.cc:3037] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.154461 13828 raft_consensus.cc:2454] T c244226a1c40488f949735420bfacd0b P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 5.
I20240503 08:15:15.154963 14090 raft_consensus.cc:2396] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 296f3a209cf5447aaff2b8ea96a3bc4f for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:15:15.155009 13894 leader_election.cc:302] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:15.155457 16389 raft_consensus.cc:2781] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:15:15.155654 16389 raft_consensus.cc:686] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:15.155737 13960 leader_election.cc:288] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 5 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:15.156034 16389 consensus_queue.cc:237] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.156796 13960 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "32e141d0103848c1a2d473429cffcca9"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:15.157034 13960 raft_consensus.cc:482] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.157159 13960 raft_consensus.cc:3037] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.157845 13689 catalog_manager.cc:5484] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 3 to 4, leader changed from 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68) to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 4 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } }
I20240503 08:15:15.158265 13689 catalog_manager.cc:5484] T f27cc9b78d084ac791de35b29e08a09a P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 3 to 5, leader changed from 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68) to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 5 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } }
I20240503 08:15:15.159579 13960 raft_consensus.cc:504] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:15.160022 13960 leader_election.cc:288] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 6 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:15.160442 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32e141d0103848c1a2d473429cffcca9" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:15.160624 14090 raft_consensus.cc:3032] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:15:15.160691 14090 raft_consensus.cc:729] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:15.160943 14090 consensus_queue.cc:260] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:15.161108 13960 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d69d33529e1140e5bc3a0a561012dfa1"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:15.161206 14090 raft_consensus.cc:3037] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.161244 13960 raft_consensus.cc:482] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.161340 13960 raft_consensus.cc:3037] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.162729 13828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32e141d0103848c1a2d473429cffcca9" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:15.162925 13828 raft_consensus.cc:3037] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.163820 14090 raft_consensus.cc:2396] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 296f3a209cf5447aaff2b8ea96a3bc4f for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:15:15.164148 13960 raft_consensus.cc:504] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:15.164702 13960 leader_election.cc:288] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 6 election: Requested vote from peers abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:15.164907 13828 raft_consensus.cc:2454] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 6.
I20240503 08:15:15.165048 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d69d33529e1140e5bc3a0a561012dfa1" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:15.165143 14218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d69d33529e1140e5bc3a0a561012dfa1" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:15.165341 14218 raft_consensus.cc:3037] T d69d33529e1140e5bc3a0a561012dfa1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.165334 13894 leader_election.cc:302] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5a198ad76e4f462c8be0859184aeb2fc; no voters: abb3df711d3a46809274b5cfbd4ce4e0
I20240503 08:15:15.165575 16389 raft_consensus.cc:2781] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 FOLLOWER]: Leader election won for term 6
I20240503 08:15:15.165724 16389 raft_consensus.cc:686] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:15.165964 16389 consensus_queue.cc:237] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:15.166170 13689 catalog_manager.cc:5484] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 4 to 5, leader changed from abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67) to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 5 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:15.167047 14090 raft_consensus.cc:3032] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:15:15.167126 14090 raft_consensus.cc:729] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:15.167348 14090 consensus_queue.cc:260] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:15.167594 14090 raft_consensus.cc:3037] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.167886 14218 raft_consensus.cc:2396] T d69d33529e1140e5bc3a0a561012dfa1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 296f3a209cf5447aaff2b8ea96a3bc4f for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:15:15.169032 13960 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cdf9885a1e7c4b8dbb160e370ca0f60c"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:15.169243 13960 raft_consensus.cc:482] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.169365 13960 raft_consensus.cc:3037] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.170586 14090 raft_consensus.cc:2396] T d69d33529e1140e5bc3a0a561012dfa1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 296f3a209cf5447aaff2b8ea96a3bc4f for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:15:15.170993 13896 leader_election.cc:302] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f; no voters: 5761f1165461459f9aaa8cd5d8e9b0b9, abb3df711d3a46809274b5cfbd4ce4e0
I20240503 08:15:15.171300 16389 raft_consensus.cc:2726] T d69d33529e1140e5bc3a0a561012dfa1 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240503 08:15:15.171849 13960 raft_consensus.cc:504] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:15.172672 13960 leader_election.cc:288] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 6 election: Requested vote from peers abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:15.173122 13828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdf9885a1e7c4b8dbb160e370ca0f60c" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:15.172807 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdf9885a1e7c4b8dbb160e370ca0f60c" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:15.173306 13828 raft_consensus.cc:3037] T cdf9885a1e7c4b8dbb160e370ca0f60c P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.173614 14090 raft_consensus.cc:3032] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:15:15.173530 13689 catalog_manager.cc:5484] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 5 to 6, leader changed from abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67) to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 6 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } }
I20240503 08:15:15.173725 14090 raft_consensus.cc:729] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:15.174052 14090 consensus_queue.cc:260] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:15.174360 14090 raft_consensus.cc:3037] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.175900 13828 raft_consensus.cc:2454] T cdf9885a1e7c4b8dbb160e370ca0f60c P 5a198ad76e4f462c8be0859184aeb2fc [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 6.
I20240503 08:15:15.176739 13960 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c310dd5906a24299b6f62201b30b86db"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:15.176831 14090 raft_consensus.cc:2396] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 296f3a209cf5447aaff2b8ea96a3bc4f for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:15:15.177088 13894 leader_election.cc:302] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:15.177320 16389 raft_consensus.cc:2781] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 FOLLOWER]: Leader election won for term 6
I20240503 08:15:15.177317 13960 raft_consensus.cc:482] T c310dd5906a24299b6f62201b30b86db P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.177502 16389 raft_consensus.cc:686] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:15.177630 13960 raft_consensus.cc:3037] T c310dd5906a24299b6f62201b30b86db P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.177747 16389 consensus_queue.cc:237] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:15.180593 13960 raft_consensus.cc:504] T c310dd5906a24299b6f62201b30b86db P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:15.181876 14218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c310dd5906a24299b6f62201b30b86db" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:15.182044 14218 raft_consensus.cc:3037] T c310dd5906a24299b6f62201b30b86db P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.182888 14219 raft_consensus.cc:1261] T 045fa6c60d334659aab79f7ba4999661 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 5 FOLLOWER]: Refusing update from remote peer abb3df711d3a46809274b5cfbd4ce4e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240503 08:15:15.183518 16210 consensus_queue.cc:1035] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:15:15.184815 14218 raft_consensus.cc:2396] T c310dd5906a24299b6f62201b30b86db P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 296f3a209cf5447aaff2b8ea96a3bc4f for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 08:15:15.185642 13960 leader_election.cc:288] T c310dd5906a24299b6f62201b30b86db P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 5 election: Requested vote from peers abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:15.185479 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c310dd5906a24299b6f62201b30b86db" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:15.186759 14090 raft_consensus.cc:3032] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:15:15.186753 13957 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f1e94dd86fc74c549979d30f0f705da7"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:15.186903 14090 raft_consensus.cc:729] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:15.186923 13957 raft_consensus.cc:482] T f1e94dd86fc74c549979d30f0f705da7 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.187013 13957 raft_consensus.cc:3037] T f1e94dd86fc74c549979d30f0f705da7 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.187297 14090 consensus_queue.cc:260] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:15.187608 14090 raft_consensus.cc:3037] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.189500 13957 raft_consensus.cc:504] T f1e94dd86fc74c549979d30f0f705da7 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:15.190598 14090 raft_consensus.cc:2396] T c310dd5906a24299b6f62201b30b86db P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 296f3a209cf5447aaff2b8ea96a3bc4f for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 08:15:15.190676 14218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e94dd86fc74c549979d30f0f705da7" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:15.190846 14218 raft_consensus.cc:3037] T f1e94dd86fc74c549979d30f0f705da7 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.191087 13896 leader_election.cc:302] T c310dd5906a24299b6f62201b30b86db P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f; no voters: 5761f1165461459f9aaa8cd5d8e9b0b9, abb3df711d3a46809274b5cfbd4ce4e0
I20240503 08:15:15.191498 16396 raft_consensus.cc:2726] T c310dd5906a24299b6f62201b30b86db P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240503 08:15:15.192865 14218 raft_consensus.cc:2454] T f1e94dd86fc74c549979d30f0f705da7 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 5.
I20240503 08:15:15.193251 13895 leader_election.cc:302] T f1e94dd86fc74c549979d30f0f705da7 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5761f1165461459f9aaa8cd5d8e9b0b9; no voters:
I20240503 08:15:15.193396 13957 leader_election.cc:288] T f1e94dd86fc74c549979d30f0f705da7 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 5 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:15.193519 16396 raft_consensus.cc:2781] T f1e94dd86fc74c549979d30f0f705da7 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:15:15.193661 16396 raft_consensus.cc:686] T f1e94dd86fc74c549979d30f0f705da7 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:15.193636 13828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e94dd86fc74c549979d30f0f705da7" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:15.193903 16396 consensus_queue.cc:237] T f1e94dd86fc74c549979d30f0f705da7 P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:15.194110 13828 raft_consensus.cc:3037] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.196444 13828 raft_consensus.cc:2396] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 296f3a209cf5447aaff2b8ea96a3bc4f for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 08:15:15.196606 13960 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "150f0d2d17bc4a59a8d82982790b4594"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:15.196792 13960 raft_consensus.cc:482] T 150f0d2d17bc4a59a8d82982790b4594 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.196893 13960 raft_consensus.cc:3037] T 150f0d2d17bc4a59a8d82982790b4594 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.197458 13689 catalog_manager.cc:5484] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 5 to 6, leader changed from abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67) to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 6 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } }
I20240503 08:15:15.198968 13960 raft_consensus.cc:504] T 150f0d2d17bc4a59a8d82982790b4594 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:15.199925 14218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150f0d2d17bc4a59a8d82982790b4594" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:15.200098 14218 raft_consensus.cc:3037] T 150f0d2d17bc4a59a8d82982790b4594 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.202832 14218 raft_consensus.cc:2454] T 150f0d2d17bc4a59a8d82982790b4594 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 5.
I20240503 08:15:15.203207 13895 leader_election.cc:302] T 150f0d2d17bc4a59a8d82982790b4594 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5761f1165461459f9aaa8cd5d8e9b0b9; no voters:
I20240503 08:15:15.203507 13960 leader_election.cc:288] T 150f0d2d17bc4a59a8d82982790b4594 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 5 election: Requested vote from peers abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:15.203505 16389 raft_consensus.cc:2781] T 150f0d2d17bc4a59a8d82982790b4594 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:15:15.203729 13828 raft_consensus.cc:1261] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Refusing update from remote peer abb3df711d3a46809274b5cfbd4ce4e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240503 08:15:15.203860 16389 raft_consensus.cc:686] T 150f0d2d17bc4a59a8d82982790b4594 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:15.204095 16389 consensus_queue.cc:237] T 150f0d2d17bc4a59a8d82982790b4594 P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:15.204183 16210 consensus_queue.cc:1035] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:15:15.204324 16210 consensus_queue.cc:1225] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Peer 5a198ad76e4f462c8be0859184aeb2fc log is divergent from this leader: its last log entry 3.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240503 08:15:15.204728 13960 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d2ba977460694815a38d415a14e1cd5f"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:15.205053 13828 pending_rounds.cc:76] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc: Aborting all ops after (but not including) 3
I20240503 08:15:15.205163 13828 pending_rounds.cc:97] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20240503 08:15:15.205202 13960 raft_consensus.cc:482] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.205241 13828 raft_consensus.cc:2866] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:15:15.205332 13960 raft_consensus.cc:3037] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.206025 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150f0d2d17bc4a59a8d82982790b4594" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:15.206511 14090 raft_consensus.cc:3032] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:15:15.206770 14090 raft_consensus.cc:729] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:15.207151 14090 consensus_queue.cc:260] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:15.207845 14090 raft_consensus.cc:3037] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.207878 13960 raft_consensus.cc:504] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:15.208709 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ba977460694815a38d415a14e1cd5f" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:15.208932 13830 raft_consensus.cc:3037] T d2ba977460694815a38d415a14e1cd5f P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.211458 13830 raft_consensus.cc:2454] T d2ba977460694815a38d415a14e1cd5f P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 5.
I20240503 08:15:15.211875 13894 leader_election.cc:302] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:15.212766 16389 raft_consensus.cc:2781] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:15:15.212934 16389 raft_consensus.cc:686] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:15.213182 16389 consensus_queue.cc:237] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:15.213650 13960 leader_election.cc:288] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 5 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:15.212673 14087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ba977460694815a38d415a14e1cd5f" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:15.214053 14087 raft_consensus.cc:3032] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:15:15.214141 14087 raft_consensus.cc:729] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:15.214346 14087 consensus_queue.cc:260] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:15.214490 13960 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "666be8071a904e6cbff984fa293d00b5"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:15.214603 14087 raft_consensus.cc:3037] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.214644 13960 raft_consensus.cc:482] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.214726 13960 raft_consensus.cc:3037] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.217103 14087 raft_consensus.cc:2396] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 296f3a209cf5447aaff2b8ea96a3bc4f for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:15:15.217614 13960 raft_consensus.cc:504] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.218524 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "666be8071a904e6cbff984fa293d00b5" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:15.218678 13830 raft_consensus.cc:3037] T 666be8071a904e6cbff984fa293d00b5 P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.211532 14090 raft_consensus.cc:2396] T 150f0d2d17bc4a59a8d82982790b4594 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 296f3a209cf5447aaff2b8ea96a3bc4f for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:15:15.220911 13830 raft_consensus.cc:2454] T 666be8071a904e6cbff984fa293d00b5 P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 5.
I20240503 08:15:15.221284 13894 leader_election.cc:302] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:15.221339 13960 leader_election.cc:288] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 5 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:15.221104 13689 catalog_manager.cc:5484] T f1e94dd86fc74c549979d30f0f705da7 P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 3 to 5, leader changed from 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65) to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 5 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } }
I20240503 08:15:15.221565 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "666be8071a904e6cbff984fa293d00b5" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:15.221704 16396 raft_consensus.cc:2781] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:15:15.221791 14090 raft_consensus.cc:3032] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:15:15.221856 16396 raft_consensus.cc:686] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:15.221879 14090 raft_consensus.cc:729] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:15.222087 16396 consensus_queue.cc:237] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.222124 14090 consensus_queue.cc:260] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.222368 14090 raft_consensus.cc:3037] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.222746 13960 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "667abbd8fb6c49489127c12977d7a323"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:15.222970 13960 raft_consensus.cc:482] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.223143 13960 raft_consensus.cc:3037] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.224774 14090 raft_consensus.cc:2396] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 296f3a209cf5447aaff2b8ea96a3bc4f for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:15:15.225265 13960 raft_consensus.cc:504] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:15.225793 13960 leader_election.cc:288] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 6 election: Requested vote from peers abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:15.226220 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "667abbd8fb6c49489127c12977d7a323" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:15.226384 14219 raft_consensus.cc:3037] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.226900 13960 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "94cfd3b764b148bb8b42e5a5cb7883b3"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:15.227056 13960 raft_consensus.cc:482] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.227134 13960 raft_consensus.cc:3037] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.228935 14219 raft_consensus.cc:2454] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 6.
I20240503 08:15:15.229429 13960 raft_consensus.cc:504] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.229686 13895 leader_election.cc:302] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5761f1165461459f9aaa8cd5d8e9b0b9; no voters:
I20240503 08:15:15.230072 16396 raft_consensus.cc:2781] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 FOLLOWER]: Leader election won for term 6
I20240503 08:15:15.230254 16396 raft_consensus.cc:686] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:15.230276 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94cfd3b764b148bb8b42e5a5cb7883b3" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:15.230459 13830 raft_consensus.cc:3037] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.230608 16396 consensus_queue.cc:237] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:15.230563 13689 catalog_manager.cc:5484] T 150f0d2d17bc4a59a8d82982790b4594 P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 4 to 5, leader changed from abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67) to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 5 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } }
I20240503 08:15:15.231159 13689 catalog_manager.cc:5484] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 4 to 5, leader changed from abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67) to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 5 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:15.231665 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "667abbd8fb6c49489127c12977d7a323" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:15.231879 14090 raft_consensus.cc:3032] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:15:15.231748 13689 catalog_manager.cc:5484] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 4 to 5, leader changed from abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67) to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 5 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } }
I20240503 08:15:15.232038 14090 raft_consensus.cc:729] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:15.232313 14090 consensus_queue.cc:260] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:15.232605 14090 raft_consensus.cc:3037] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.232849 13830 raft_consensus.cc:2454] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 6.
I20240503 08:15:15.233273 13894 leader_election.cc:302] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:15.233587 16397 raft_consensus.cc:2781] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 FOLLOWER]: Leader election won for term 6
I20240503 08:15:15.233713 16397 raft_consensus.cc:686] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:15.233999 16397 consensus_queue.cc:237] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.235447 14090 raft_consensus.cc:2396] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 296f3a209cf5447aaff2b8ea96a3bc4f for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:15:15.237988 13960 leader_election.cc:288] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 6 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:15.239559 13960 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "377cd840a48f47238bdaa5ad2257e1f4"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:15.239732 13960 raft_consensus.cc:482] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.239845 13960 raft_consensus.cc:3037] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.241132 13689 catalog_manager.cc:5484] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 5 to 6, leader changed from abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67) to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 6 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } }
I20240503 08:15:15.241458 13689 catalog_manager.cc:5484] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 5 to 6, leader changed from abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67) to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 6 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:15.241554 13960 raft_consensus.cc:504] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:15.242112 13960 leader_election.cc:288] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 5 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:15.242727 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377cd840a48f47238bdaa5ad2257e1f4" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:15.242729 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94cfd3b764b148bb8b42e5a5cb7883b3" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:15.242918 14090 raft_consensus.cc:3032] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:15:15.242923 14219 raft_consensus.cc:3037] T 377cd840a48f47238bdaa5ad2257e1f4 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.242990 14090 raft_consensus.cc:729] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:15.242972 13960 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d7ebe785e08640cb866350eae5797e9f"
dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
from {username='slave'} at 127.0.0.1:59280
I20240503 08:15:15.243132 13960 raft_consensus.cc:482] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.243217 13960 raft_consensus.cc:3037] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:15.243239 14090 consensus_queue.cc:260] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.243522 14090 raft_consensus.cc:3037] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.245517 14219 raft_consensus.cc:2454] T 377cd840a48f47238bdaa5ad2257e1f4 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 5.
I20240503 08:15:15.245687 13960 raft_consensus.cc:504] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:15.246003 14090 raft_consensus.cc:2396] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 296f3a209cf5447aaff2b8ea96a3bc4f for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:15:15.246405 13895 leader_election.cc:302] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5761f1165461459f9aaa8cd5d8e9b0b9; no voters:
I20240503 08:15:15.246482 13828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377cd840a48f47238bdaa5ad2257e1f4" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:15.246594 13960 leader_election.cc:288] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 4 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:15.246665 13828 raft_consensus.cc:3037] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.246733 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7ebe785e08640cb866350eae5797e9f" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:15.246904 13830 raft_consensus.cc:3037] T d7ebe785e08640cb866350eae5797e9f P 5a198ad76e4f462c8be0859184aeb2fc [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:15.246963 14218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7ebe785e08640cb866350eae5797e9f" candidate_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:15.247148 14218 raft_consensus.cc:3032] T d7ebe785e08640cb866350eae5797e9f P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:15:15.247229 14218 raft_consensus.cc:729] T d7ebe785e08640cb866350eae5797e9f P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Running, Role: LEADER
I20240503 08:15:15.247252 16396 raft_consensus.cc:2781] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:15:15.247474 14218 consensus_queue.cc:260] T d7ebe785e08640cb866350eae5797e9f P 5761f1165461459f9aaa8cd5d8e9b0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:15.247658 16396 raft_consensus.cc:686] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:15.247828 14218 raft_consensus.cc:3037] T d7ebe785e08640cb866350eae5797e9f P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:15.247895 16396 consensus_queue.cc:237] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:15.249573 13830 raft_consensus.cc:2454] T d7ebe785e08640cb866350eae5797e9f P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 4.
I20240503 08:15:15.250032 13894 leader_election.cc:302] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:15.250339 16397 raft_consensus.cc:2781] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:15:15.250407 14218 raft_consensus.cc:2454] T d7ebe785e08640cb866350eae5797e9f P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296f3a209cf5447aaff2b8ea96a3bc4f in term 4.
I20240503 08:15:15.250483 16397 raft_consensus.cc:686] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 LEADER]: Becoming Leader. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:15.250500 13828 raft_consensus.cc:2396] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 296f3a209cf5447aaff2b8ea96a3bc4f for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 08:15:15.250686 16397 consensus_queue.cc:237] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:15.257615 13689 catalog_manager.cc:5484] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 3 to 5, leader changed from 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65) to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 5 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } }
I20240503 08:15:15.261086 13830 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "045fa6c60d334659aab79f7ba4999661"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:15.261303 13830 raft_consensus.cc:482] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.261418 13830 raft_consensus.cc:3037] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.263731 13830 raft_consensus.cc:504] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.264304 13830 leader_election.cc:288] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 6 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:15.264675 13828 raft_consensus.cc:1261] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Refusing update from remote peer abb3df711d3a46809274b5cfbd4ce4e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240503 08:15:15.264907 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "045fa6c60d334659aab79f7ba4999661" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:15.265100 14090 raft_consensus.cc:3032] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:15:15.265182 14090 raft_consensus.cc:729] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:15.265237 14218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "045fa6c60d334659aab79f7ba4999661" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:15.265475 14218 raft_consensus.cc:3037] T 045fa6c60d334659aab79f7ba4999661 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.265456 14090 consensus_queue.cc:260] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.265785 13828 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c7a958fc685e41fb86108dfcb9c6dc8b"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:15.265836 14090 raft_consensus.cc:3037] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.265941 13828 raft_consensus.cc:482] T c7a958fc685e41fb86108dfcb9c6dc8b P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.266042 13828 raft_consensus.cc:3037] T c7a958fc685e41fb86108dfcb9c6dc8b P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.267565 14218 raft_consensus.cc:2454] T 045fa6c60d334659aab79f7ba4999661 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 6.
I20240503 08:15:15.268316 16210 consensus_queue.cc:1035] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:15:15.268496 14090 raft_consensus.cc:2454] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 6.
I20240503 08:15:15.268483 13828 raft_consensus.cc:504] T c7a958fc685e41fb86108dfcb9c6dc8b P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.268501 16210 consensus_queue.cc:1225] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Peer 5a198ad76e4f462c8be0859184aeb2fc log is divergent from this leader: its last log entry 3.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240503 08:15:15.268879 13766 leader_election.cc:302] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:15.268743 13689 catalog_manager.cc:5484] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f reported cstate change: term changed from 3 to 4, leader changed from 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68) to 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66). New cstate: current_term: 4 leader_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } }
I20240503 08:15:15.269094 13828 leader_election.cc:288] T c7a958fc685e41fb86108dfcb9c6dc8b P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 5 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:15.269264 16292 raft_consensus.cc:2781] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [term 6 FOLLOWER]: Leader election won for term 6
I20240503 08:15:15.269418 16292 raft_consensus.cc:686] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [term 6 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:15.269652 16292 consensus_queue.cc:237] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.270112 14218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a958fc685e41fb86108dfcb9c6dc8b" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:15.270299 14218 raft_consensus.cc:3037] T c7a958fc685e41fb86108dfcb9c6dc8b P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.271251 14087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a958fc685e41fb86108dfcb9c6dc8b" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:15.271431 14087 raft_consensus.cc:3037] T c7a958fc685e41fb86108dfcb9c6dc8b P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.272434 14218 raft_consensus.cc:2396] T c7a958fc685e41fb86108dfcb9c6dc8b P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 08:15:15.272250 13686 catalog_manager.cc:5484] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 5 to 6, leader changed from abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67) to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 6 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:15.273975 14087 raft_consensus.cc:2454] T c7a958fc685e41fb86108dfcb9c6dc8b P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 5.
I20240503 08:15:15.274294 13827 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "51b32125655f4ef78dc49ba7174775da"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:15.274469 13827 raft_consensus.cc:482] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.274477 13766 leader_election.cc:302] T c7a958fc685e41fb86108dfcb9c6dc8b P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc, abb3df711d3a46809274b5cfbd4ce4e0; no voters: 5761f1165461459f9aaa8cd5d8e9b0b9
I20240503 08:15:15.274600 13827 raft_consensus.cc:3037] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.274781 16215 raft_consensus.cc:2781] T c7a958fc685e41fb86108dfcb9c6dc8b P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:15:15.274960 16215 raft_consensus.cc:686] T c7a958fc685e41fb86108dfcb9c6dc8b P 5a198ad76e4f462c8be0859184aeb2fc [term 5 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:15.275285 16215 consensus_queue.cc:237] T c7a958fc685e41fb86108dfcb9c6dc8b P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.277137 13827 raft_consensus.cc:504] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:15.277786 13827 leader_election.cc:288] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 6 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:15:15.278410 14218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51b32125655f4ef78dc49ba7174775da" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:15.278518 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51b32125655f4ef78dc49ba7174775da" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:15.278744 13960 raft_consensus.cc:3032] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:15:15.278771 14218 raft_consensus.cc:3037] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.278853 13960 raft_consensus.cc:729] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:15.278862 14219 raft_consensus.cc:1261] T de418f9b6bb04763b88ccc3bbe6cd68f P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 5 FOLLOWER]: Refusing update from remote peer abb3df711d3a46809274b5cfbd4ce4e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240503 08:15:15.279101 13960 consensus_queue.cc:260] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:15.279397 13960 raft_consensus.cc:3037] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.279450 16210 consensus_queue.cc:1035] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:15:15.282222 14218 raft_consensus.cc:2454] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 6.
I20240503 08:15:15.282691 13960 raft_consensus.cc:2396] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:15:15.282673 13765 leader_election.cc:302] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:15.283015 16292 raft_consensus.cc:2781] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc [term 6 FOLLOWER]: Leader election won for term 6
I20240503 08:15:15.283669 13827 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dd3d6353358d4b02af928cd9f9410aca"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:15.283864 13827 raft_consensus.cc:482] T dd3d6353358d4b02af928cd9f9410aca P 5a198ad76e4f462c8be0859184aeb2fc [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.283967 13827 raft_consensus.cc:3037] T dd3d6353358d4b02af928cd9f9410aca P 5a198ad76e4f462c8be0859184aeb2fc [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:15:15.284850 16292 raft_consensus.cc:686] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc [term 6 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:15.285137 16292 consensus_queue.cc:237] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:15.286252 13830 pending_rounds.cc:76] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc: Aborting all ops after (but not including) 3
I20240503 08:15:15.286341 13830 pending_rounds.cc:97] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20240503 08:15:15.286443 13830 raft_consensus.cc:2866] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:15:15.286753 13827 raft_consensus.cc:504] T dd3d6353358d4b02af928cd9f9410aca P 5a198ad76e4f462c8be0859184aeb2fc [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.289544 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd3d6353358d4b02af928cd9f9410aca" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:15.289742 13960 raft_consensus.cc:3032] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 LEADER]: Stepping down as leader of term 6
I20240503 08:15:15.289857 13960 raft_consensus.cc:729] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:15.290113 13960 consensus_queue.cc:260] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.290395 13960 raft_consensus.cc:3037] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:15:15.295501 14218 raft_consensus.cc:1261] T 4b680592b1774f09ba79789afa55a0f1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 5 FOLLOWER]: Refusing update from remote peer abb3df711d3a46809274b5cfbd4ce4e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240503 08:15:15.295943 13689 catalog_manager.cc:5484] T c7a958fc685e41fb86108dfcb9c6dc8b P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 3 to 5, leader changed from 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68) to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 5 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:15.296104 16362 consensus_queue.cc:1035] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:15:15.301306 13689 catalog_manager.cc:5484] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 5 to 6, leader changed from 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66) to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 6 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } }
I20240503 08:15:15.302469 13827 leader_election.cc:288] T dd3d6353358d4b02af928cd9f9410aca P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 7 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:15.303061 14087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd3d6353358d4b02af928cd9f9410aca" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:15.303269 14087 raft_consensus.cc:3037] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:15:15.306253 13830 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8bb1779c7ac64d599cde0665505b3dba"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:15.306321 14087 raft_consensus.cc:2396] T dd3d6353358d4b02af928cd9f9410aca P abb3df711d3a46809274b5cfbd4ce4e0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:15.306423 13830 raft_consensus.cc:482] T 8bb1779c7ac64d599cde0665505b3dba P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.306532 13830 raft_consensus.cc:3037] T 8bb1779c7ac64d599cde0665505b3dba P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.306922 13960 raft_consensus.cc:2396] T dd3d6353358d4b02af928cd9f9410aca P 296f3a209cf5447aaff2b8ea96a3bc4f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:15.307317 13827 raft_consensus.cc:1261] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Refusing update from remote peer abb3df711d3a46809274b5cfbd4ce4e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240503 08:15:15.307785 13763 leader_election.cc:302] T dd3d6353358d4b02af928cd9f9410aca P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc; no voters: 296f3a209cf5447aaff2b8ea96a3bc4f, abb3df711d3a46809274b5cfbd4ce4e0
I20240503 08:15:15.308280 16215 raft_consensus.cc:2726] T dd3d6353358d4b02af928cd9f9410aca P 5a198ad76e4f462c8be0859184aeb2fc [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240503 08:15:15.308701 13830 raft_consensus.cc:504] T 8bb1779c7ac64d599cde0665505b3dba P 5a198ad76e4f462c8be0859184aeb2fc [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.309209 13830 leader_election.cc:288] T 8bb1779c7ac64d599cde0665505b3dba P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 6 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:15.309644 16362 consensus_queue.cc:1035] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:15:15.309851 16362 consensus_queue.cc:1225] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Peer 5a198ad76e4f462c8be0859184aeb2fc log is divergent from this leader: its last log entry 3.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240503 08:15:15.309908 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb1779c7ac64d599cde0665505b3dba" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:15.310138 13960 raft_consensus.cc:3032] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:15:15.310231 13960 raft_consensus.cc:729] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:15.310467 13830 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "79014e600505478883fc90159784ce6c"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:15.310524 14087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb1779c7ac64d599cde0665505b3dba" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:15.310636 13830 raft_consensus.cc:482] T 79014e600505478883fc90159784ce6c P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.310703 14087 raft_consensus.cc:3037] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.310755 13830 raft_consensus.cc:3037] T 79014e600505478883fc90159784ce6c P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.312371 13830 raft_consensus.cc:504] T 79014e600505478883fc90159784ce6c P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:15.313263 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79014e600505478883fc90159784ce6c" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:15.313453 14090 raft_consensus.cc:3037] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.313483 14087 raft_consensus.cc:2396] T 8bb1779c7ac64d599cde0665505b3dba P abb3df711d3a46809274b5cfbd4ce4e0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:15:15.314051 13960 consensus_queue.cc:260] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.314361 13960 raft_consensus.cc:3037] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.316008 14090 raft_consensus.cc:2454] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 5.
I20240503 08:15:15.316432 13766 leader_election.cc:302] T 79014e600505478883fc90159784ce6c P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:15.316715 16215 raft_consensus.cc:2781] T 79014e600505478883fc90159784ce6c P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:15:15.316867 16215 raft_consensus.cc:686] T 79014e600505478883fc90159784ce6c P 5a198ad76e4f462c8be0859184aeb2fc [term 5 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:15.317108 16215 consensus_queue.cc:237] T 79014e600505478883fc90159784ce6c P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:15.317858 14218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79014e600505478883fc90159784ce6c" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:15.318106 14218 raft_consensus.cc:3037] T 79014e600505478883fc90159784ce6c P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.321250 14218 raft_consensus.cc:2396] T 79014e600505478883fc90159784ce6c P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 08:15:15.321398 13830 leader_election.cc:288] T 79014e600505478883fc90159784ce6c P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 5 election: Requested vote from peers abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:15.310683 13827 pending_rounds.cc:76] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc: Aborting all ops after (but not including) 3
I20240503 08:15:15.321971 13827 pending_rounds.cc:97] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20240503 08:15:15.323200 13827 raft_consensus.cc:2866] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:15:15.322954 13830 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a478c28e5d4646fdb956f5b65e5c6c94"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:15.323508 13830 raft_consensus.cc:482] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.323648 13830 raft_consensus.cc:3037] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.324816 13689 catalog_manager.cc:5484] T 79014e600505478883fc90159784ce6c P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 3 to 5, leader changed from 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68) to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 5 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } }
I20240503 08:15:15.326596 13960 raft_consensus.cc:2396] T 8bb1779c7ac64d599cde0665505b3dba P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:15:15.327023 13763 leader_election.cc:302] T 8bb1779c7ac64d599cde0665505b3dba P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc; no voters: 296f3a209cf5447aaff2b8ea96a3bc4f, abb3df711d3a46809274b5cfbd4ce4e0
I20240503 08:15:15.327179 13830 raft_consensus.cc:504] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:15.327405 16215 raft_consensus.cc:2726] T 8bb1779c7ac64d599cde0665505b3dba P 5a198ad76e4f462c8be0859184aeb2fc [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240503 08:15:15.328800 14218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a478c28e5d4646fdb956f5b65e5c6c94" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:15.329030 14218 raft_consensus.cc:3037] T a478c28e5d4646fdb956f5b65e5c6c94 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.332471 14218 raft_consensus.cc:2454] T a478c28e5d4646fdb956f5b65e5c6c94 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 6.
I20240503 08:15:15.332901 13830 leader_election.cc:288] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 6 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:15:15.333364 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a478c28e5d4646fdb956f5b65e5c6c94" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:15.333647 13765 leader_election.cc:302] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:15.334192 16215 raft_consensus.cc:2781] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [term 6 FOLLOWER]: Leader election won for term 6
I20240503 08:15:15.333716 13960 raft_consensus.cc:3032] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:15:15.334334 16215 raft_consensus.cc:686] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [term 6 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:15.334401 13960 raft_consensus.cc:729] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:15.334576 16215 consensus_queue.cc:237] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:15.334704 13960 consensus_queue.cc:260] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:15.335343 13960 raft_consensus.cc:3037] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.339769 13830 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "769c4490062b4206951af2f4b9a6da05"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:15.340173 13830 raft_consensus.cc:482] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.340467 13830 raft_consensus.cc:3037] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.340706 13960 raft_consensus.cc:2396] T a478c28e5d4646fdb956f5b65e5c6c94 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:15:15.344820 13689 catalog_manager.cc:5484] T a478c28e5d4646fdb956f5b65e5c6c94 P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 5 to 6, leader changed from 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66) to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 6 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } }
I20240503 08:15:15.345506 13830 raft_consensus.cc:504] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:15.346989 14218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "769c4490062b4206951af2f4b9a6da05" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:15.347244 14218 raft_consensus.cc:3037] T 769c4490062b4206951af2f4b9a6da05 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.348323 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "769c4490062b4206951af2f4b9a6da05" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:15.348493 13960 raft_consensus.cc:3032] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:15:15.348588 13960 raft_consensus.cc:729] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:15.348840 13960 consensus_queue.cc:260] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:15.349108 13960 raft_consensus.cc:3037] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.350836 14218 raft_consensus.cc:2454] T 769c4490062b4206951af2f4b9a6da05 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 5.
I20240503 08:15:15.351229 13765 leader_election.cc:302] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:15.351501 16215 raft_consensus.cc:2781] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:15:15.351634 16215 raft_consensus.cc:686] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc [term 5 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:15.352077 16215 consensus_queue.cc:237] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:15.353737 13960 raft_consensus.cc:2396] T 769c4490062b4206951af2f4b9a6da05 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:15:15.354437 13830 leader_election.cc:288] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 5 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:15:15.354279 13689 catalog_manager.cc:5484] T 769c4490062b4206951af2f4b9a6da05 P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 4 to 5, leader changed from 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66) to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 5 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } }
I20240503 08:15:15.355978 13830 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "de418f9b6bb04763b88ccc3bbe6cd68f"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:15.356567 13830 raft_consensus.cc:482] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.356684 13830 raft_consensus.cc:3037] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.358618 13830 raft_consensus.cc:504] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.359375 13830 leader_election.cc:288] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 6 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:15.359491 14087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de418f9b6bb04763b88ccc3bbe6cd68f" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:15.359691 14087 raft_consensus.cc:3032] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:15:15.359783 14087 raft_consensus.cc:729] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:15.359944 14218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de418f9b6bb04763b88ccc3bbe6cd68f" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:15.360100 14218 raft_consensus.cc:3037] T de418f9b6bb04763b88ccc3bbe6cd68f P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.360018 14087 consensus_queue.cc:260] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.360333 14087 raft_consensus.cc:3037] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.360369 13827 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d7ebe785e08640cb866350eae5797e9f"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:15.360657 13827 raft_consensus.cc:482] T d7ebe785e08640cb866350eae5797e9f P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.360810 13827 raft_consensus.cc:3037] T d7ebe785e08640cb866350eae5797e9f P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.362833 14087 raft_consensus.cc:2454] T de418f9b6bb04763b88ccc3bbe6cd68f P abb3df711d3a46809274b5cfbd4ce4e0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 6.
I20240503 08:15:15.362951 14218 raft_consensus.cc:2454] T de418f9b6bb04763b88ccc3bbe6cd68f P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 6.
I20240503 08:15:15.363257 13766 leader_election.cc:302] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:15.363564 16215 raft_consensus.cc:2781] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [term 6 FOLLOWER]: Leader election won for term 6
I20240503 08:15:15.363700 16215 raft_consensus.cc:686] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [term 6 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:15.363525 13827 raft_consensus.cc:504] T d7ebe785e08640cb866350eae5797e9f P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:15.363936 16215 consensus_queue.cc:237] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.364665 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7ebe785e08640cb866350eae5797e9f" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:15.364837 13960 raft_consensus.cc:3032] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:15:15.364921 13960 raft_consensus.cc:729] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:15.365168 13960 consensus_queue.cc:260] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:15.365460 13960 raft_consensus.cc:3037] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.366348 14218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7ebe785e08640cb866350eae5797e9f" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:15.366587 14218 raft_consensus.cc:3037] T d7ebe785e08640cb866350eae5797e9f P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.366995 13827 leader_election.cc:288] T d7ebe785e08640cb866350eae5797e9f P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 5 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:15:15.366962 13686 catalog_manager.cc:5484] T de418f9b6bb04763b88ccc3bbe6cd68f P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 5 to 6, leader changed from abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67) to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 6 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:15.367779 13960 raft_consensus.cc:2396] T d7ebe785e08640cb866350eae5797e9f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:15:15.367758 13830 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c244226a1c40488f949735420bfacd0b"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:15.367893 13830 raft_consensus.cc:482] T c244226a1c40488f949735420bfacd0b P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.367998 13830 raft_consensus.cc:3037] T c244226a1c40488f949735420bfacd0b P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.368973 14218 raft_consensus.cc:2454] T d7ebe785e08640cb866350eae5797e9f P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 5.
I20240503 08:15:15.369318 13765 leader_election.cc:302] T d7ebe785e08640cb866350eae5797e9f P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, 5a198ad76e4f462c8be0859184aeb2fc; no voters: 296f3a209cf5447aaff2b8ea96a3bc4f
I20240503 08:15:15.369699 16215 raft_consensus.cc:2781] T d7ebe785e08640cb866350eae5797e9f P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:15:15.369864 16215 raft_consensus.cc:686] T d7ebe785e08640cb866350eae5797e9f P 5a198ad76e4f462c8be0859184aeb2fc [term 5 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:15.370105 16215 consensus_queue.cc:237] T d7ebe785e08640cb866350eae5797e9f P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:15.370487 13830 raft_consensus.cc:504] T c244226a1c40488f949735420bfacd0b P 5a198ad76e4f462c8be0859184aeb2fc [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.371881 14087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c244226a1c40488f949735420bfacd0b" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:15.372095 14087 raft_consensus.cc:3037] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.372648 13686 catalog_manager.cc:5484] T d7ebe785e08640cb866350eae5797e9f P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 4 to 5, leader changed from 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66) to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 5 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } }
I20240503 08:15:15.373610 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c244226a1c40488f949735420bfacd0b" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:15.373785 13960 raft_consensus.cc:3032] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:15:15.373873 13960 raft_consensus.cc:729] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:15.374075 13960 consensus_queue.cc:260] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.374666 13960 raft_consensus.cc:3037] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.374989 14087 raft_consensus.cc:2396] T c244226a1c40488f949735420bfacd0b P abb3df711d3a46809274b5cfbd4ce4e0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:15:15.375839 13830 leader_election.cc:288] T c244226a1c40488f949735420bfacd0b P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 6 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:15.376724 13830 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "32e141d0103848c1a2d473429cffcca9"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:15.376916 13830 raft_consensus.cc:482] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.377054 13830 raft_consensus.cc:3037] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:15:15.377928 13960 raft_consensus.cc:2396] T c244226a1c40488f949735420bfacd0b P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:15:15.378297 13763 leader_election.cc:302] T c244226a1c40488f949735420bfacd0b P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc; no voters: 296f3a209cf5447aaff2b8ea96a3bc4f, abb3df711d3a46809274b5cfbd4ce4e0
I20240503 08:15:15.378650 16215 raft_consensus.cc:2726] T c244226a1c40488f949735420bfacd0b P 5a198ad76e4f462c8be0859184aeb2fc [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240503 08:15:15.380487 13830 raft_consensus.cc:504] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:15.381078 13830 leader_election.cc:288] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 7 election: Requested vote from peers abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:15:15.381561 14087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32e141d0103848c1a2d473429cffcca9" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:15.381737 14087 raft_consensus.cc:3037] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:15:15.381872 13827 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cdf9885a1e7c4b8dbb160e370ca0f60c"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:15.382021 13827 raft_consensus.cc:482] T cdf9885a1e7c4b8dbb160e370ca0f60c P 5a198ad76e4f462c8be0859184aeb2fc [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.382104 13827 raft_consensus.cc:3037] T cdf9885a1e7c4b8dbb160e370ca0f60c P 5a198ad76e4f462c8be0859184aeb2fc [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:15:15.384513 14087 raft_consensus.cc:2396] T 32e141d0103848c1a2d473429cffcca9 P abb3df711d3a46809274b5cfbd4ce4e0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:15.384442 13827 raft_consensus.cc:504] T cdf9885a1e7c4b8dbb160e370ca0f60c P 5a198ad76e4f462c8be0859184aeb2fc [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:15.385361 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdf9885a1e7c4b8dbb160e370ca0f60c" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:15.385383 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32e141d0103848c1a2d473429cffcca9" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:15.385542 14090 raft_consensus.cc:3037] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:15:15.385592 13960 raft_consensus.cc:3032] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 LEADER]: Stepping down as leader of term 6
I20240503 08:15:15.385782 13960 raft_consensus.cc:729] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:15.386088 13960 consensus_queue.cc:260] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:15.386425 13960 raft_consensus.cc:3037] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:15:15.388188 14090 raft_consensus.cc:2396] T cdf9885a1e7c4b8dbb160e370ca0f60c P abb3df711d3a46809274b5cfbd4ce4e0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:15.388489 13957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdf9885a1e7c4b8dbb160e370ca0f60c" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:15.388664 13957 raft_consensus.cc:3032] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 LEADER]: Stepping down as leader of term 6
I20240503 08:15:15.388767 13957 raft_consensus.cc:729] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:15.389098 13960 raft_consensus.cc:2396] T 32e141d0103848c1a2d473429cffcca9 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:15.388990 13957 consensus_queue.cc:260] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:15.389596 13763 leader_election.cc:302] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc; no voters: 296f3a209cf5447aaff2b8ea96a3bc4f, abb3df711d3a46809274b5cfbd4ce4e0
I20240503 08:15:15.389653 13957 raft_consensus.cc:3037] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:15:15.389966 16215 raft_consensus.cc:2726] T 32e141d0103848c1a2d473429cffcca9 P 5a198ad76e4f462c8be0859184aeb2fc [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240503 08:15:15.391515 13957 raft_consensus.cc:2396] T cdf9885a1e7c4b8dbb160e370ca0f60c P 296f3a209cf5447aaff2b8ea96a3bc4f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:15.391721 13827 leader_election.cc:288] T cdf9885a1e7c4b8dbb160e370ca0f60c P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 7 election: Requested vote from peers abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:15:15.391933 13763 leader_election.cc:302] T cdf9885a1e7c4b8dbb160e370ca0f60c P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc; no voters: 296f3a209cf5447aaff2b8ea96a3bc4f, abb3df711d3a46809274b5cfbd4ce4e0
I20240503 08:15:15.392261 16215 raft_consensus.cc:2726] T cdf9885a1e7c4b8dbb160e370ca0f60c P 5a198ad76e4f462c8be0859184aeb2fc [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240503 08:15:15.392477 13828 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4b680592b1774f09ba79789afa55a0f1"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:15.392622 13828 raft_consensus.cc:482] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.392709 13828 raft_consensus.cc:3037] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.394836 13828 raft_consensus.cc:504] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:15.395362 13828 leader_election.cc:288] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 6 election: Requested vote from peers abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:15.396040 14218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b680592b1774f09ba79789afa55a0f1" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:15.396221 14218 raft_consensus.cc:3037] T 4b680592b1774f09ba79789afa55a0f1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.398258 14218 raft_consensus.cc:2454] T 4b680592b1774f09ba79789afa55a0f1 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 6.
I20240503 08:15:15.398612 13765 leader_election.cc:302] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:15.398849 16215 raft_consensus.cc:2781] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [term 6 FOLLOWER]: Leader election won for term 6
I20240503 08:15:15.398969 16215 raft_consensus.cc:686] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [term 6 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:15.399209 16215 consensus_queue.cc:237] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:15.399617 13828 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f1e94dd86fc74c549979d30f0f705da7"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:15.399772 13828 raft_consensus.cc:482] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.399884 13828 raft_consensus.cc:3037] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.400862 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b680592b1774f09ba79789afa55a0f1" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:15.401031 14090 raft_consensus.cc:3032] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:15:15.401103 14090 raft_consensus.cc:729] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:15.401365 14090 consensus_queue.cc:260] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:15.401636 14090 raft_consensus.cc:3037] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.401700 13686 catalog_manager.cc:5484] T 4b680592b1774f09ba79789afa55a0f1 P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 5 to 6, leader changed from abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67) to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 6 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } }
I20240503 08:15:15.404515 14090 raft_consensus.cc:2454] T 4b680592b1774f09ba79789afa55a0f1 P abb3df711d3a46809274b5cfbd4ce4e0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 6.
I20240503 08:15:15.407527 13828 raft_consensus.cc:504] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:15.408556 14218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e94dd86fc74c549979d30f0f705da7" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:15.408774 14218 raft_consensus.cc:3037] T f1e94dd86fc74c549979d30f0f705da7 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.409956 13957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e94dd86fc74c549979d30f0f705da7" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:15.410179 13957 raft_consensus.cc:3032] T f1e94dd86fc74c549979d30f0f705da7 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:15:15.410298 13957 raft_consensus.cc:729] T f1e94dd86fc74c549979d30f0f705da7 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:15.411185 13830 raft_consensus.cc:1261] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Refusing update from remote peer abb3df711d3a46809274b5cfbd4ce4e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240503 08:15:15.411481 14218 raft_consensus.cc:2454] T f1e94dd86fc74c549979d30f0f705da7 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 6.
I20240503 08:15:15.411796 16361 consensus_queue.cc:1035] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:15:15.411870 13765 leader_election.cc:302] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:15.412128 16292 raft_consensus.cc:2781] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [term 6 FOLLOWER]: Leader election won for term 6
I20240503 08:15:15.412256 16292 raft_consensus.cc:686] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [term 6 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:15.412500 16292 consensus_queue.cc:237] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:15.410669 13957 consensus_queue.cc:260] T f1e94dd86fc74c549979d30f0f705da7 P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:15.415004 14219 raft_consensus.cc:1261] T 06e7243793e9467daeed79f3b1f5be5a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 5 FOLLOWER]: Refusing update from remote peer abb3df711d3a46809274b5cfbd4ce4e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240503 08:15:15.415747 16362 consensus_queue.cc:1035] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:15:15.416736 13957 raft_consensus.cc:3037] T f1e94dd86fc74c549979d30f0f705da7 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.420125 13957 raft_consensus.cc:2396] T f1e94dd86fc74c549979d30f0f705da7 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:15:15.420593 13828 leader_election.cc:288] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 6 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:15.422715 13828 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "06e7243793e9467daeed79f3b1f5be5a"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:15.422892 13828 raft_consensus.cc:482] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.422986 13828 raft_consensus.cc:3037] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.425570 13828 raft_consensus.cc:504] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.429349 13681 catalog_manager.cc:5484] T f1e94dd86fc74c549979d30f0f705da7 P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 5 to 6, leader changed from 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66) to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 6 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } }
I20240503 08:15:15.433336 13828 leader_election.cc:288] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 6 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:15.433604 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e7243793e9467daeed79f3b1f5be5a" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:15.433797 14090 raft_consensus.cc:3032] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:15:15.433892 14090 raft_consensus.cc:729] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:15.433974 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e7243793e9467daeed79f3b1f5be5a" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:15.434146 14219 raft_consensus.cc:3037] T 06e7243793e9467daeed79f3b1f5be5a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.434123 14090 consensus_queue.cc:260] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.434442 14090 raft_consensus.cc:3037] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.436291 14219 raft_consensus.cc:2454] T 06e7243793e9467daeed79f3b1f5be5a P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 6.
I20240503 08:15:15.436595 14090 raft_consensus.cc:2454] T 06e7243793e9467daeed79f3b1f5be5a P abb3df711d3a46809274b5cfbd4ce4e0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 6.
I20240503 08:15:15.436622 13765 leader_election.cc:302] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:15.436867 16292 raft_consensus.cc:2781] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [term 6 FOLLOWER]: Leader election won for term 6
I20240503 08:15:15.437032 16292 raft_consensus.cc:686] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [term 6 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:15.437252 16292 consensus_queue.cc:237] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.438880 13828 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "377cd840a48f47238bdaa5ad2257e1f4"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:15.439044 13828 raft_consensus.cc:482] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.439155 13828 raft_consensus.cc:3037] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.439582 13681 catalog_manager.cc:5484] T 06e7243793e9467daeed79f3b1f5be5a P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 5 to 6, leader changed from abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67) to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 6 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:15.441802 13828 raft_consensus.cc:504] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:15.443949 13828 leader_election.cc:288] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 6 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713)
I20240503 08:15:15.444491 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377cd840a48f47238bdaa5ad2257e1f4" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:15.444677 14219 raft_consensus.cc:3037] T 377cd840a48f47238bdaa5ad2257e1f4 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.444952 13830 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d2ba977460694815a38d415a14e1cd5f"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:15.445132 13830 raft_consensus.cc:482] T d2ba977460694815a38d415a14e1cd5f P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.445219 13830 raft_consensus.cc:3037] T d2ba977460694815a38d415a14e1cd5f P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.447160 14219 raft_consensus.cc:2454] T 377cd840a48f47238bdaa5ad2257e1f4 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 6.
I20240503 08:15:15.447824 13830 raft_consensus.cc:504] T d2ba977460694815a38d415a14e1cd5f P 5a198ad76e4f462c8be0859184aeb2fc [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:15.448887 13957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377cd840a48f47238bdaa5ad2257e1f4" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:15.448941 13765 leader_election.cc:302] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:15.449082 13957 raft_consensus.cc:3032] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:15:15.449169 13957 raft_consensus.cc:729] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:15.449183 16292 raft_consensus.cc:2781] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [term 6 FOLLOWER]: Leader election won for term 6
I20240503 08:15:15.449332 16292 raft_consensus.cc:686] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [term 6 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:15.449437 13957 consensus_queue.cc:260] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:15.449558 16292 consensus_queue.cc:237] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:15.449715 13957 raft_consensus.cc:3037] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.452448 13957 raft_consensus.cc:2396] T 377cd840a48f47238bdaa5ad2257e1f4 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:15:15.454079 13827 raft_consensus.cc:1261] T 9cae16af5a034cc29b8bb410a758c332 P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Refusing update from remote peer abb3df711d3a46809274b5cfbd4ce4e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240503 08:15:15.454152 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ba977460694815a38d415a14e1cd5f" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:15.454352 14090 raft_consensus.cc:3037] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.455590 16361 consensus_queue.cc:1035] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:15:15.455796 16361 consensus_queue.cc:1225] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Peer 5a198ad76e4f462c8be0859184aeb2fc log is divergent from this leader: its last log entry 2.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240503 08:15:15.457041 13830 leader_election.cc:288] T d2ba977460694815a38d415a14e1cd5f P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 6 election: Requested vote from peers abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:15:15.457926 13827 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9cae16af5a034cc29b8bb410a758c332"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:15.458117 13827 raft_consensus.cc:482] T 9cae16af5a034cc29b8bb410a758c332 P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.458213 13827 raft_consensus.cc:3037] T 9cae16af5a034cc29b8bb410a758c332 P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.460918 13827 raft_consensus.cc:504] T 9cae16af5a034cc29b8bb410a758c332 P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:15.461150 13957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ba977460694815a38d415a14e1cd5f" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:15.461310 13829 raft_consensus.cc:1226] T 9cae16af5a034cc29b8bb410a758c332 P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Rejecting Update request from peer abb3df711d3a46809274b5cfbd4ce4e0 for earlier term 4. Current term is 5. Ops: [3.4-4.5]
I20240503 08:15:15.461318 13957 raft_consensus.cc:3032] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:15:15.461416 13957 raft_consensus.cc:729] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:15.461685 13957 consensus_queue.cc:260] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:15.462308 13957 raft_consensus.cc:3037] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.462620 16361 consensus_queue.cc:1046] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:15:15.462842 16361 consensus_queue.cc:1225] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [LEADER]: Peer 5a198ad76e4f462c8be0859184aeb2fc log is divergent from this leader: its last log entry 2.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240503 08:15:15.462967 16361 raft_consensus.cc:3032] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:15:15.462904 14087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cae16af5a034cc29b8bb410a758c332" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:15.463048 16361 raft_consensus.cc:729] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:15.463162 13827 leader_election.cc:288] T 9cae16af5a034cc29b8bb410a758c332 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 5 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:15.463299 16361 consensus_queue.cc:260] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:15.463656 16361 raft_consensus.cc:3037] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.464094 13830 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "666be8071a904e6cbff984fa293d00b5"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:15.464248 13830 raft_consensus.cc:482] T 666be8071a904e6cbff984fa293d00b5 P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.464350 13830 raft_consensus.cc:3037] T 666be8071a904e6cbff984fa293d00b5 P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.464797 13957 raft_consensus.cc:2396] T d2ba977460694815a38d415a14e1cd5f P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:15:15.467139 13830 raft_consensus.cc:504] T 666be8071a904e6cbff984fa293d00b5 P 5a198ad76e4f462c8be0859184aeb2fc [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.468101 14090 raft_consensus.cc:2396] T d2ba977460694815a38d415a14e1cd5f P abb3df711d3a46809274b5cfbd4ce4e0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:15:15.468518 13766 leader_election.cc:302] T d2ba977460694815a38d415a14e1cd5f P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc; no voters: 296f3a209cf5447aaff2b8ea96a3bc4f, abb3df711d3a46809274b5cfbd4ce4e0
I20240503 08:15:15.468911 16215 raft_consensus.cc:2726] T d2ba977460694815a38d415a14e1cd5f P 5a198ad76e4f462c8be0859184aeb2fc [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240503 08:15:15.470116 13957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "666be8071a904e6cbff984fa293d00b5" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:15.470356 13957 raft_consensus.cc:3032] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:15:15.470347 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "666be8071a904e6cbff984fa293d00b5" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:15.470544 14090 raft_consensus.cc:3037] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.470539 13957 raft_consensus.cc:729] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:15.470899 13957 consensus_queue.cc:260] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.470939 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cae16af5a034cc29b8bb410a758c332" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:15.471113 14219 raft_consensus.cc:3037] T 9cae16af5a034cc29b8bb410a758c332 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.471266 13957 raft_consensus.cc:3037] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.473273 14090 raft_consensus.cc:2396] T 666be8071a904e6cbff984fa293d00b5 P abb3df711d3a46809274b5cfbd4ce4e0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:15:15.473723 13830 leader_election.cc:288] T 666be8071a904e6cbff984fa293d00b5 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 6 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:15.473991 13957 raft_consensus.cc:2396] T 666be8071a904e6cbff984fa293d00b5 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:15:15.474385 14087 raft_consensus.cc:2396] T 9cae16af5a034cc29b8bb410a758c332 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 08:15:15.474436 13763 leader_election.cc:302] T 666be8071a904e6cbff984fa293d00b5 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc; no voters: 296f3a209cf5447aaff2b8ea96a3bc4f, abb3df711d3a46809274b5cfbd4ce4e0
I20240503 08:15:15.474694 16215 raft_consensus.cc:2726] T 666be8071a904e6cbff984fa293d00b5 P 5a198ad76e4f462c8be0859184aeb2fc [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240503 08:15:15.475198 13828 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "94cfd3b764b148bb8b42e5a5cb7883b3"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:15.475387 13828 raft_consensus.cc:482] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.475483 13828 raft_consensus.cc:3037] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:15:15.477295 14219 raft_consensus.cc:2396] T 9cae16af5a034cc29b8bb410a758c332 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 08:15:15.478219 13828 raft_consensus.cc:504] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.479157 13957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94cfd3b764b148bb8b42e5a5cb7883b3" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:15.479318 13957 raft_consensus.cc:3032] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 LEADER]: Stepping down as leader of term 6
I20240503 08:15:15.479398 13957 raft_consensus.cc:729] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:15.479641 13957 consensus_queue.cc:260] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.479959 13957 raft_consensus.cc:3037] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:15:15.482909 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94cfd3b764b148bb8b42e5a5cb7883b3" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:15.483080 14090 raft_consensus.cc:3037] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:15:15.485659 14090 raft_consensus.cc:2396] T 94cfd3b764b148bb8b42e5a5cb7883b3 P abb3df711d3a46809274b5cfbd4ce4e0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:15.486056 13828 leader_election.cc:288] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 7 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:15.486826 13830 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "84204cbcda32446ab708717e9633af69"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:15.486984 13830 raft_consensus.cc:482] T 84204cbcda32446ab708717e9633af69 P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.487084 13830 raft_consensus.cc:3037] T 84204cbcda32446ab708717e9633af69 P 5a198ad76e4f462c8be0859184aeb2fc [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.489634 13830 raft_consensus.cc:504] T 84204cbcda32446ab708717e9633af69 P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:15.490721 13830 leader_election.cc:288] T 84204cbcda32446ab708717e9633af69 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 5 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:15:15.490886 13960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84204cbcda32446ab708717e9633af69" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:15.491047 13960 raft_consensus.cc:3032] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:15:15.491132 13960 raft_consensus.cc:729] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:15.491364 13960 consensus_queue.cc:260] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:15.491466 13828 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e82cb19fece24181b7607d9447ba1eb2"
dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
from {username='slave'} at 127.0.0.1:49838
I20240503 08:15:15.491631 13828 raft_consensus.cc:482] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.491647 13960 raft_consensus.cc:3037] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.491730 13828 raft_consensus.cc:3037] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.491987 13765 leader_election.cc:302] T 9cae16af5a034cc29b8bb410a758c332 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc; no voters: 5761f1165461459f9aaa8cd5d8e9b0b9, abb3df711d3a46809274b5cfbd4ce4e0
I20240503 08:15:15.492491 16215 raft_consensus.cc:2726] T 9cae16af5a034cc29b8bb410a758c332 P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240503 08:15:15.493301 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84204cbcda32446ab708717e9633af69" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:15.493541 14219 raft_consensus.cc:3037] T 84204cbcda32446ab708717e9633af69 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:15:15.494340 13960 raft_consensus.cc:2396] T 84204cbcda32446ab708717e9633af69 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:15:15.494383 13828 raft_consensus.cc:504] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.495457 14218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82cb19fece24181b7607d9447ba1eb2" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
I20240503 08:15:15.495659 14218 raft_consensus.cc:3037] T e82cb19fece24181b7607d9447ba1eb2 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.496465 13957 raft_consensus.cc:2396] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:15.496655 14219 raft_consensus.cc:2454] T 84204cbcda32446ab708717e9633af69 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 5.
I20240503 08:15:15.496836 13763 leader_election.cc:302] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a198ad76e4f462c8be0859184aeb2fc; no voters: 296f3a209cf5447aaff2b8ea96a3bc4f, abb3df711d3a46809274b5cfbd4ce4e0
I20240503 08:15:15.497123 16215 raft_consensus.cc:2726] T 94cfd3b764b148bb8b42e5a5cb7883b3 P 5a198ad76e4f462c8be0859184aeb2fc [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240503 08:15:15.497357 13765 leader_election.cc:302] T 84204cbcda32446ab708717e9633af69 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, 5a198ad76e4f462c8be0859184aeb2fc; no voters: 296f3a209cf5447aaff2b8ea96a3bc4f
I20240503 08:15:15.498175 14218 raft_consensus.cc:2454] T e82cb19fece24181b7607d9447ba1eb2 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a198ad76e4f462c8be0859184aeb2fc in term 6.
I20240503 08:15:15.498626 13765 leader_election.cc:302] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, 5a198ad76e4f462c8be0859184aeb2fc; no voters:
I20240503 08:15:15.498871 16215 raft_consensus.cc:2781] T 84204cbcda32446ab708717e9633af69 P 5a198ad76e4f462c8be0859184aeb2fc [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:15:15.498930 16422 raft_consensus.cc:2781] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [term 6 FOLLOWER]: Leader election won for term 6
I20240503 08:15:15.502936 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82cb19fece24181b7607d9447ba1eb2" candidate_uuid: "5a198ad76e4f462c8be0859184aeb2fc" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:15.503441 14090 raft_consensus.cc:3032] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:15:15.503551 14090 raft_consensus.cc:729] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: abb3df711d3a46809274b5cfbd4ce4e0, State: Running, Role: LEADER
I20240503 08:15:15.503808 14090 consensus_queue.cc:260] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.504094 14090 raft_consensus.cc:3037] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.507249 13828 leader_election.cc:288] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [CANDIDATE]: Term 6 election: Requested vote from peers 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68:33713), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:15.507404 16215 raft_consensus.cc:686] T 84204cbcda32446ab708717e9633af69 P 5a198ad76e4f462c8be0859184aeb2fc [term 5 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:15.508273 16215 consensus_queue.cc:237] T 84204cbcda32446ab708717e9633af69 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } }
I20240503 08:15:15.508301 13689 catalog_manager.cc:5484] T 377cd840a48f47238bdaa5ad2257e1f4 P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 5 to 6, leader changed from 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66) to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 6 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } }
I20240503 08:15:15.510223 14090 raft_consensus.cc:2396] T e82cb19fece24181b7607d9447ba1eb2 P abb3df711d3a46809274b5cfbd4ce4e0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a198ad76e4f462c8be0859184aeb2fc for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240503 08:15:15.514714 16422 raft_consensus.cc:686] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [term 6 LEADER]: Becoming Leader. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:15.515183 14218 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "045fa6c60d334659aab79f7ba4999661"
dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
from {username='slave'} at 127.0.0.1:58176
I20240503 08:15:15.515168 16422 consensus_queue.cc:237] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.515379 14218 raft_consensus.cc:482] T 045fa6c60d334659aab79f7ba4999661 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.515484 14218 raft_consensus.cc:3037] T 045fa6c60d334659aab79f7ba4999661 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:15:15.517550 14218 raft_consensus.cc:504] T 045fa6c60d334659aab79f7ba4999661 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.518206 14218 leader_election.cc:288] T 045fa6c60d334659aab79f7ba4999661 P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 7 election: Requested vote from peers 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925), abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647)
I20240503 08:15:15.518875 13828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "045fa6c60d334659aab79f7ba4999661" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:15.519107 13828 raft_consensus.cc:3032] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [term 6 LEADER]: Stepping down as leader of term 6
I20240503 08:15:15.519222 13828 raft_consensus.cc:729] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:15.519881 13828 consensus_queue.cc:260] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.520229 13828 raft_consensus.cc:3037] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:15:15.520490 14218 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "667abbd8fb6c49489127c12977d7a323"
dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
from {username='slave'} at 127.0.0.1:58176
I20240503 08:15:15.520658 14218 raft_consensus.cc:482] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.520788 14218 raft_consensus.cc:3037] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:15:15.521432 13689 catalog_manager.cc:5484] T 84204cbcda32446ab708717e9633af69 P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 4 to 5, leader changed from 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66) to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 5 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } }
I20240503 08:15:15.521984 13689 catalog_manager.cc:5484] T e82cb19fece24181b7607d9447ba1eb2 P 5a198ad76e4f462c8be0859184aeb2fc reported cstate change: term changed from 5 to 6, leader changed from abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67) to 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65). New cstate: current_term: 6 leader_uuid: "5a198ad76e4f462c8be0859184aeb2fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:15.523761 13828 raft_consensus.cc:2396] T 045fa6c60d334659aab79f7ba4999661 P 5a198ad76e4f462c8be0859184aeb2fc [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 5761f1165461459f9aaa8cd5d8e9b0b9 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240503 08:15:15.524741 14218 raft_consensus.cc:504] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:15.525847 14087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "667abbd8fb6c49489127c12977d7a323" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:15.525887 13957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "667abbd8fb6c49489127c12977d7a323" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:15.526042 14087 raft_consensus.cc:3037] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:15:15.526051 13957 raft_consensus.cc:3032] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 LEADER]: Stepping down as leader of term 6
I20240503 08:15:15.526135 13957 raft_consensus.cc:729] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 296f3a209cf5447aaff2b8ea96a3bc4f, State: Running, Role: LEADER
I20240503 08:15:15.526350 13957 consensus_queue.cc:260] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } }
I20240503 08:15:15.526610 13957 raft_consensus.cc:3037] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:15:15.528661 14087 raft_consensus.cc:2396] T 667abbd8fb6c49489127c12977d7a323 P abb3df711d3a46809274b5cfbd4ce4e0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 5761f1165461459f9aaa8cd5d8e9b0b9 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:15.529062 13957 raft_consensus.cc:2396] T 667abbd8fb6c49489127c12977d7a323 P 296f3a209cf5447aaff2b8ea96a3bc4f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 5761f1165461459f9aaa8cd5d8e9b0b9 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:15.529834 14153 leader_election.cc:302] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9; no voters: 296f3a209cf5447aaff2b8ea96a3bc4f, abb3df711d3a46809274b5cfbd4ce4e0
I20240503 08:15:15.530138 16315 raft_consensus.cc:2726] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240503 08:15:15.530480 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "045fa6c60d334659aab79f7ba4999661" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:15.530629 14090 raft_consensus.cc:3037] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:15:15.531790 14218 leader_election.cc:288] T 667abbd8fb6c49489127c12977d7a323 P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 7 election: Requested vote from peers abb3df711d3a46809274b5cfbd4ce4e0 (127.20.230.67:34647), 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781)
I20240503 08:15:15.533071 14218 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "51b32125655f4ef78dc49ba7174775da"
dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
from {username='slave'} at 127.0.0.1:58176
I20240503 08:15:15.533300 14218 raft_consensus.cc:482] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.533432 14218 raft_consensus.cc:3037] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:15:15.533710 14090 raft_consensus.cc:2454] T 045fa6c60d334659aab79f7ba4999661 P abb3df711d3a46809274b5cfbd4ce4e0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5761f1165461459f9aaa8cd5d8e9b0b9 in term 7.
I20240503 08:15:15.534143 14156 leader_election.cc:302] T 045fa6c60d334659aab79f7ba4999661 P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, abb3df711d3a46809274b5cfbd4ce4e0; no voters: 5a198ad76e4f462c8be0859184aeb2fc
I20240503 08:15:15.534423 16315 raft_consensus.cc:2781] T 045fa6c60d334659aab79f7ba4999661 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 7 FOLLOWER]: Leader election won for term 7
I20240503 08:15:15.534610 16315 raft_consensus.cc:686] T 045fa6c60d334659aab79f7ba4999661 P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 7 LEADER]: Becoming Leader. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Running, Role: LEADER
I20240503 08:15:15.534854 16315 consensus_queue.cc:237] T 045fa6c60d334659aab79f7ba4999661 P 5761f1165461459f9aaa8cd5d8e9b0b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } }
I20240503 08:15:15.536571 14218 raft_consensus.cc:504] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:15.537343 13681 catalog_manager.cc:5484] T 045fa6c60d334659aab79f7ba4999661 P 5761f1165461459f9aaa8cd5d8e9b0b9 reported cstate change: term changed from 6 to 7, leader changed from 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65) to 5761f1165461459f9aaa8cd5d8e9b0b9 (127.20.230.68). New cstate: current_term: 7 leader_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } }
I20240503 08:15:15.537570 13957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51b32125655f4ef78dc49ba7174775da" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f"
I20240503 08:15:15.537801 13957 raft_consensus.cc:3037] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:15:15.538308 14218 leader_election.cc:288] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 7 election: Requested vote from peers 296f3a209cf5447aaff2b8ea96a3bc4f (127.20.230.66:44781), 5a198ad76e4f462c8be0859184aeb2fc (127.20.230.65:44925)
I20240503 08:15:15.538841 13828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51b32125655f4ef78dc49ba7174775da" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:15.539196 13828 raft_consensus.cc:3032] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc [term 6 LEADER]: Stepping down as leader of term 6
I20240503 08:15:15.539279 13828 raft_consensus.cc:729] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 5a198ad76e4f462c8be0859184aeb2fc, State: Running, Role: LEADER
I20240503 08:15:15.539553 13828 consensus_queue.cc:260] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "296f3a209cf5447aaff2b8ea96a3bc4f" member_type: VOTER last_known_addr { host: "127.20.230.66" port: 44781 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:15.539817 13828 raft_consensus.cc:3037] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:15:15.541745 13957 raft_consensus.cc:2396] T 51b32125655f4ef78dc49ba7174775da P 296f3a209cf5447aaff2b8ea96a3bc4f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 5761f1165461459f9aaa8cd5d8e9b0b9 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:15.542080 13828 raft_consensus.cc:2396] T 51b32125655f4ef78dc49ba7174775da P 5a198ad76e4f462c8be0859184aeb2fc [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 5761f1165461459f9aaa8cd5d8e9b0b9 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:15:15.542448 14154 leader_election.cc:302] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9; no voters: 296f3a209cf5447aaff2b8ea96a3bc4f, 5a198ad76e4f462c8be0859184aeb2fc
I20240503 08:15:15.542572 14218 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "79014e600505478883fc90159784ce6c"
dest_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9"
from {username='slave'} at 127.0.0.1:58176
I20240503 08:15:15.542765 14218 raft_consensus.cc:482] T 79014e600505478883fc90159784ce6c P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:15.542856 16316 raft_consensus.cc:2726] T 51b32125655f4ef78dc49ba7174775da P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240503 08:15:15.542901 14218 raft_consensus.cc:3037] T 79014e600505478883fc90159784ce6c P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.545647 14218 raft_consensus.cc:504] T 79014e600505478883fc90159784ce6c P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65" port: 44925 } }
I20240503 08:15:15.546639 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79014e600505478883fc90159784ce6c" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "abb3df711d3a46809274b5cfbd4ce4e0"
I20240503 08:15:15.546849 14090 raft_consensus.cc:3037] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:15:15.549666 14090 raft_consensus.cc:2454] T 79014e600505478883fc90159784ce6c P abb3df711d3a46809274b5cfbd4ce4e0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5761f1165461459f9aaa8cd5d8e9b0b9 in term 6.
I20240503 08:15:15.550134 14156 leader_election.cc:302] T 79014e600505478883fc90159784ce6c P 5761f1165461459f9aaa8cd5d8e9b0b9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5761f1165461459f9aaa8cd5d8e9b0b9, abb3df711d3a46809274b5cfbd4ce4e0; no voters:
I20240503 08:15:15.550410 16316 raft_consensus.cc:2781] T 79014e600505478883fc90159784ce6c P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 08:15:15.550531 16316 raft_consensus.cc:686] T 79014e600505478883fc90159784ce6c P 5761f1165461459f9aaa8cd5d8e9b0b9 [term 6 LEADER]: Becoming Leader. State: Replica: 5761f1165461459f9aaa8cd5d8e9b0b9, State: Running, Role: LEADER
I20240503 08:15:15.550583 13828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79014e600505478883fc90159784ce6c" candidate_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5a198ad76e4f462c8be0859184aeb2fc"
I20240503 08:15:15.550771 13828 raft_consensus.cc:3032] T 79014e600505478883fc90159784ce6c P 5a198ad76e4f462c8be0859184aeb2fc [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:15:15.550761 16316 consensus_queue.cc:237] T 79014e600505478883fc90159784ce6c P 5761f1165461459f9aaa8cd5d8e9b0b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb3df711d3a46809274b5cfbd4ce4e0" member_type: VOTER last_known_addr { host: "127.20.230.67" port: 34647 } } peers { permanent_uuid: "5761f1165461459f9aaa8cd5d8e9b0b9" member_type: VOTER last_known_addr { host: "127.20.230.68" port: 33713 } } peers { permanent_uuid: "5a198ad76e4f462c8be0859184aeb2fc" member_type: VOTER last_known_addr { host: "127.20.230.65"